Update to FTS5

Our old stale copy of the FTS3/4 module is now deleted, replaced by
a shinier FTS5 embedded module. If at configure time we detect that
SQLite doesn't offer the FTS5 module, we will load our own, just as
we used to do with FTS4.

FTS5 brings a few differences in the ways it's meant to be extended,
the tokenizer has been updated to cope with the differences. Also,
FTS5 offers no offsets() builtin function, nor matchinfo() which we
used to implement ranking. It offers though ways to implement
additional functions, and builtin rank support which can be tweaked
to achieve the same functional results than we did.

Other than that, the ways to interact with the FTS virtual table
are roughly similar to those in FTS4, insertions and deletions have
been updated to do things the FTS5 way.

Since it's not worth to bump the database format (data is reproducted
from the journal, so we drop some embedded data such as
nie:plainTextContent), the nco:hobby property has been modified to
no longer be fulltext indexed, AFAIK there's no users ever setting/
accessing that, and the FTS properties change will trigger the
regeneration of the FTS view and virtual tables, resulting in a
seamless update to FTS5.

However, we don't leave completely unscathed from the fts3_tokenizer()
change. Since the older FTS3/4 tokenizer is not registered, we can't
just drop the older FTS table. So it is left dangling and never
accessed again, in favor of the newer fts5 table. This is obviously
not a problem when creating the database from scratch.

In the way, a few bugs were found. per-property weights in ranking
were being given in a scrambled way (although stable across database
generations). And deletion of FTS properties (or entire rows) could
result in the tokens not being fully removed from the FTS table,
resulting in confused searches. These are now fixed.

Impact to users of tracker should be none. All the FTS Sparql-to-SQL
translation has been updated to just use FTS5 syntax and tables.

1 files changed, 8 insertions, 8 deletions

diff --git a/configure.ac b/configure.ac
index 29d68d72e..2f071f52d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -820,19 +820,19 @@ fi
 if test "x$have_tracker_fts" = "xyes"; then
    AC_DEFINE(HAVE_TRACKER_FTS, [1], [Define to 1 if tracker FTS is compiled])
 
-   AX_SQLITE_BUILTIN_FTS4
-   if test "x$ax_cv_sqlite_builtin_fts4" = "xyes" ; then
-      have_builtin_fts4="yes"
-      AC_DEFINE(HAVE_BUILTIN_FTS, [], [Defined if Sqlite has FTS4 compiled in])
+   AX_SQLITE_BUILTIN_FTS5
+   if test "x$ax_cv_sqlite_builtin_fts5" = "xyes" ; then
+      have_builtin_fts5="yes"
+      AC_DEFINE(HAVE_BUILTIN_FTS, [], [Defined if Sqlite has FTS5 compiled in])
    else
-      have_builtin_fts4="no"
+      have_builtin_fts5="no"
    fi
 else
    AC_DEFINE(HAVE_TRACKER_FTS, [0], [Define to 0 if tracker FTS is not compiled])
-   have_builtin_fts4="disabled"
+   have_builtin_fts5="disabled"
 fi
 
-AM_CONDITIONAL(HAVE_BUILTIN_FTS, test "$have_builtin_fts4" = "yes")
+AM_CONDITIONAL(HAVE_BUILTIN_FTS, test "$have_builtin_fts5" = "yes")
 AM_CONDITIONAL(HAVE_TRACKER_FTS, test "$have_tracker_fts" = "yes")
 
 ####################################################################
@@ -2790,7 +2790,7 @@ Feature Support:
 	Unicode support library:                $with_unicode_support
 
 	Build with Journal support:             $have_tracker_journal
-	Build with SQLite FTS support:          $have_tracker_fts (built-in FTS: $have_builtin_fts4)
+	Build with SQLite FTS support:          $have_tracker_fts (built-in FTS: $have_builtin_fts5)
 	Build with Stemming support:            $have_libstemmer
 
 	Cache media art                         $have_libmediaart (libmediaart)