FlexSearch v0.8

[ts-thomas]

Let's discuss the new version. Also I have some important new features I would like to show you.

FlexSearch Playground

Use a playground to interactively test different settings and compare them.

Feature: Storage Interface including Postgres, SQLite, MongoDB, Clickhouse + IndexedDB

The whole export/import will be improved. A new storage interface lets you use different storage technologies. Those will be provided as builtin:

1. Filesystem (server-side)
2. PostgreSQL (server-side)
3. SQLite (server-side)
4. MongoDB (server-side)
5. Clickhouse (server-side)
6. Local Storage (client-side)
7. Session Storage (client-side)
8. IndexedDB (client-side)

The storage interface also will provide enrich the results from database objects.

It's probably better to exclude those libraries from package.json. Alternatively each of the storage could be installed by the developer by just npm install .... Let me know what you think.

How we can add several storage adapter on server-side which are easy to install and use?

I would like to exclude all those DBMS dependencies. We could make a dedicated npm package for each of the storages. But it needs some additional configuration anyway (on server-side).

Simple (Non-Flexible)

Instead we can let the developer pass an authorized db client instance to the storage class provided by this library, then theoretically it just needs to specify the table name additionally for the destination. This sounds like a simple solution. But it might not work with all kinds of interface variants of the same storage engine. This solution needs to make a support for each of them.

Flexible (Non-Simple)

A different approach would be let the developer implement some kind of abstract class, where some very basic methods needs to be implement by giving a full working template for each storage DBMS to start with. Here it needs more configuration but also support all interface variants of the same storage engines.

We could provide both. What did you think?

Feature: Resolver

This feature will solve a couple of limitations of the current version. It is providing:

• a well customizable boost of indexes/fields on search
• an un-resolved intermediate result which fully contains the relevance ranking
• apply and chain logical expressions to different indexes
• add custom output formatter, e.g. for highlighting results

The resolve feature will add as opt-in, so older version does not need a migration. For this reason the automatic resolve needs either to be disabled or a full resolver should be passed here:

const index = new Index({
    resolve: false
});

The new version also supports document indexes by just using the class Index and passing a document:

const index = new Index({
    resolve: false,
    document: { /* descriptor */ }
});

When you make a search you will get a non-resolved intermeditate result you can use to pass into post-processing:

const rawindex = index.search("foobar");

You can also switch to raw results when not configured before:

const rawindex = index.search("foobar", { resolve: false });

FlexSearch will providing some methods you can use to resolve those results by applying custom logic.

The most simple one is the default resolve, which always runs under the hood when not specified:

const result = FlexSearch.resolve(rawindex);

Boost

const result = FlexSearch.resolve(rawindex, {
    index: "field-1",
    boost: 2
},{
    index: "field-2",
    boost: 0.5
});

Chain arguments or use an array.

Logical Expression

const result = FlexSearch.resolve(rawindex, {
    or: [{
        and: ["field-1", "field-2"]
    },{
        and: ["field-2", "field-3"]
    }]
});

You can nest those expressions: and, or, xor and not

Custom Functions

Custom functions could be used to determine score through a custom function:

function score(index){
    let score = 1;
    // apply some logic and determine score
    // return a numeric score:
    return score;
}

const result = FlexSearch.resolve(rawindex, {
    index: "field-1",
    score: score
},{
    index: "field-2",
    score: score
});

This could be used in combination with logical expressions:

const result = FlexSearch.resolve(rawindex, {
    or: [{
        and: [{
            index: "field-1",
            boost: 2
        },{
            index: "field-2",
            score: score
        }]
    },{
        and: [{
            index: "field-2",
            score: score
        },{
            index: "field-3",
            boost: 0.5
        }]
    }]
});

This could be also done directly by .search(query, options):

const result = index.search("foobar", { 
    resolve: { /* a full resolver */ }
});

Feature: Virtual Fields/Tags

Actually you can just derive fields from data. To add dynamic contents it needs to apply this to the data. Virtual fields are indexes which rely on custom functions (which have access to the data). Also virtual fields could be used as tags.

Assume we have this data:

const data = [{
    id: 1,
    title: "Title",
    content: "Some content",
    date: "2023-12-08",
    location : {
        postal: "...",
        city: "...",
        street: "...",
        number: "..."
    }
}];

A document could be initialized as follows by applying a virtual field "location" and a virtual tag "status":

const index = new Document({
    id: "id",
    index: ["title", "content", {
        name: "location",
        // define virtual field
        data: function(data){
            let str;
            // apply some logic to construct value or string
            str = data.street + " " + 
                  data.number + ", " + 
                  data.postal + " " + 
                  data.city ;
            // return a string
            return str;
        }
    }],
    tag: ["cat", "location", {
        name: "status",
        // define virtual tag
        data: function(data){
            let tag;
            // apply some logic to construct value or string
            tag = Date.now() < Date.parse(data.date) ? "planned" : "expired";
            // return a string or number
            return tag;
        }
    }]
});

Virtual fields can override existing data fields.

Also the virtual field "location" was used here as a tag, just to demonstrate. But you can't access virtual fields on the data inside a virtual definition. Virtual fields will also not export to the data, so you might need to refresh them.

Partial Updates

The new version is also providing making partial updates to specific indexes instead of re-indexing the whole data payload at all.

So .update(<id>, data, fields) is getting a new parameter to limit partial updates on specific fields without changing other fields:

index.update({
    id: 1,
    date: "2024-01-06",
    location : {
        postal: "...",
        city: "...",
        street: "...",
        number: "..."
    }
},{
   index: "location",
   tag: "status"
});

Asume you have indexed an event and just the location and date has changed. Then you can use partial data to update indexes without loosing contents from other fields when not included.

Result Highlighting

const result = FlexSearch.resolve(rawindex, {
    // needs to be defined on root level:
    highlight: true
    // combine with other resolver, also nested ...
});

This will output something like:

[{
    id: 1,
    doc: { /* document */ }
    highlight: "This wasn't match, but <b>this was a match</b>"
}]

You can also specify a regular expression which should be used to use custom wrappers for matches:

const result = FlexSearch.resolve(rawindex, {
    highlight: "[$1]"
});

// result: "This wasn't match, but [this was a match]"

Or:

const result = FlexSearch.resolve(rawindex, {
    highlight: "<span class="highlighted">$1</span >"
});

// result: "This wasn't match, but <span class="highlighted">this was a match</span >"

[xoxys]

#99 This FR exists for years and has many upvotes. Would really like to see this feature.

[ts-thomas]

@xoxys I've added a similar feature by some customiziation in a project. This is something which could be done by post-processing. Looks like we have a further candidate for a "resolver" 👍

[cutiful]

Is there a specific reason to use Web SQL over IndexedDB? Web SQL is deprecated in Blink, removed from WebKit and it was never implemented in Gecko.

[ts-thomas]

@cutiful IndexedDB of course :)

[cutiful]

Cool :)

[pehkay]

When will this be released? :D

[ts-thomas]

Preview version is probably available end of week :)

[flip111]

When you search documents for several fields you get a result like this

[{
    field: "title",
    result: [0, 1, 2]
},{
    field: "content",
    result: [3, 4, 5]
}]

Would be useful to get the score from each item. When you don't to separate the search results into two different lists in your UI then the results lists needs to be combined. Combining the list while keeping the scoring/ranking intact is also useful.

[ts-thomas]

Yes, the intermediate results includes the rank (accordingly to the resolution)

[Properko]

Regarding export/import improvements, I for one forked this project solely to remove the JSON.stringify call here, because I have indexes bigger than 512MB (node/chromium JSON size limit).
So 👍 for export/import improvements, especially an interface that I can write a custom filesystem serializer against that would let me split the indexes into multiple parts.

[ts-thomas]

I plan to provide more features for the backend integration, e.g. #418

[sandstrom]

Some feedback:

• For any new release, don't think only about what to add, but also what to remove. Even good software usually has some functionality that isn't used much, or maybe is no longer needed. Maybe there are things that could be removed in this release too?
• For highlight, instead of using HTML tags, I'd go for the new Highlight API (https://developer.mozilla.org/en-US/docs/Web/API/CSS_Custom_Highlight_API).

[ts-thomas]

Thanks for the highlight api 👍
I plan to remove some very rare features (don't worry, nothing like tags will be removed). This was happen in every major release.

[ezra-en]

How would the Highlight API work with other platforms like React Native? Highlighting is already decently complicated on RN, and I'd really like to see this working in a way that makes sense across the board, and if the API is a part of that, I'd love to see it!

[ts-thomas]

Actually I dived into the core functionality of the index and it looks like there is some unlocked capability available by switching over to Map and Set. I need to do some bigger load tests before I can savely change the data types. It isn't really needed, though it would be nice to get some noticable improvements for the upcoming version.

[NetOpWibby]

Would be neat if there was EdgeDB support.

https://www.edgedb.com/
https://github.com/edgedb/edgedb
https://github.com/edgedb/edgedb-deno
https://github.com/edgedb/edgedb-js

[aizotov]

Will the new release include sorting? It was mentioned here but I haven't seen it in the recent documentation.
#13 (comment)

[terpimost]

To have a Playground would be amazing!
I didn't quite get how resolver has access to the Index/Document

About the highlight.
I see that proposal returns formatted string. I guess this string is a some content field + formatting tag. By default I suggest to use <mark> for the highlight.
Initially I thought that flexsearch would return the id of the doc, path to a matching field (in case it is a doc), initial position and the length of the match

[terpimost]

I'm open to help on a playground.
Do you plan to do that just in browser and host it on github pages?

[svalaskevicius]

would be great if it could support sync export/import, to be used on server side index preparation for static websites (I've added a flexsearch plugin for vuepress but failed to use the 0.7 version due to inability to export synchronously)

[terpimost]

Why is this a problem? I'm doing the same thing and just await it

[svalaskevicius]

hi. I suppose I'm not quite sure how to just await it :)

for context - https://github.com/vuepress/ecosystem/pull/91/files#diff-3382763f21e8324f8736b2c1366b14761d619717684afbf7033238be87ade1abR57

if you're also working on such plugin maybe you could help with this?

[svalaskevicius]

actually. it is returning a promise, so scratch that, I do see how to just await it.. 🤦

(for some reason I thought that the api was returning a plain string instead of the promise)

[mnove]

Any plans to support fuzzy search as discussed here #118 ?

[nickchomey]

I just discovered FlexSearch and was initially very impressed, but my next thought was about the lack of automatic persistence of the index to IndexedDB. I see that there is export/import, but it the current per-key implementation is surely far from ideal. So, I'm very glad to see that this will be the focus of the next version.

I wonder if any of the following ideas might be possible?

• Is there any way we can simply store the entire instantiated FlexSearch object in indexeddb (as blob or arraybuffer?) and then simply retrieve that and re-instantiate it?
• Or given that it makes little sense to store all the methods etc..., can we simply extract the data/index properties of the FlexSearch object and store that as-is in IDB and then restore it to the newly instantiated FlexSearch object?

I stumbled upon this convo from the devs of RxDB and SurrealDB, where the latter seems to say that they store the indexes as ArrayBuffer

https://news.ycombinator.com/item?id=32554752

Similar ideas are here as well:
https://web.dev/articles/indexeddb-best-practices#not_everything_can_be_stored_in_indexeddb_on_all_platforms
This seems particularly helpful
https://medium.com/samsung-internet-dev/being-fast-and-light-using-binary-data-to-optimise-libraries-on-the-client-and-the-server-5709f06ef105

[nickchomey]

I have no idea if this is just LLM hallucination, but here's what Claude 3 Opus just gave me when I kept pressing not to use JSON for anything. Note that I had prompted about storing the entire instantiated FlexSearch object. But I assume you could do something similar on just the data/index property of the object.

Whatever the case, surely this is a decent enough starting point?

---

Here's an approach using ArrayBuffer and the structured clone algorithm to store and restore the FlexSearch object without using JSON:

Saving the FlexSearch object to IndexedDB using ArrayBuffer:

function saveFlexSearchObject(flexSearchObject, dbName, storeName) {
  const request = indexedDB.open(dbName);

  request.onupgradeneeded = function(event) {
    const db = event.target.result;
    db.createObjectStore(storeName);
  };

  request.onsuccess = function(event) {
    const db = event.target.result;
    const transaction = db.transaction(storeName, 'readwrite');
    const store = transaction.objectStore(storeName);

    const arrayBuffer = new ArrayBuffer(1024 * 1024); // Adjust the size as needed
    const dataView = new DataView(arrayBuffer);
    const serializedObject = structuredClone(flexSearchObject);
    const blob = new Blob([serializedObject]);

    const reader = new FileReader();
    reader.onload = function() {
      const uint8Array = new Uint8Array(reader.result);
      dataView.set(uint8Array);
      store.put(arrayBuffer, 'flexSearchObject');
    };

    reader.readAsArrayBuffer(blob);
  };
}

Restoring the FlexSearch object from IndexedDB using ArrayBuffer:

function restoreFlexSearchObject(dbName, storeName) {
  return new Promise((resolve, reject) => {
    const request = indexedDB.open(dbName);

    request.onsuccess = function(event) {
      const db = event.target.result;
      const transaction = db.transaction(storeName, 'readonly');
      const store = transaction.objectStore(storeName);

      const getRequest = store.get('flexSearchObject');

      getRequest.onsuccess = function(event) {
        const arrayBuffer = event.target.result;
        const dataView = new DataView(arrayBuffer);
        const uint8Array = new Uint8Array(dataView.buffer);
        const blob = new Blob([uint8Array]);

        const reader = new FileReader();
        reader.onload = function() {
          const flexSearchObject = structuredClone(reader.result);
          resolve(flexSearchObject);
        };

        reader.readAsArrayBuffer(blob);
      };

      getRequest.onerror = function(event) {
        reject(event.target.error);
      };
    };
  });
}

In this approach, we use the structured clone algorithm (structuredClone()) to create a deep copy of the FlexSearch object. The structured clone algorithm is capable of serializing complex objects, including typed arrays and certain built-in types.

Here's how it works:

1. In the saveFlexSearchObject function, we create an ArrayBuffer with a fixed size (adjust the size as needed based on your FlexSearch object's size).
2. We create a DataView of the ArrayBuffer to allow setting the serialized data.
3. We use structuredClone() to create a deep copy of the FlexSearch object.
4. We create a Blob from the serialized object and read it as an ArrayBuffer using a FileReader.
5. Once the FileReader's onload event is triggered, we set the Uint8Array data to the DataView and store the ArrayBuffer in IndexedDB.
6. In the restoreFlexSearchObject function, we retrieve the ArrayBuffer from IndexedDB.
7. We create a DataView and a Uint8Array from the retrieved ArrayBuffer.
8. We create a Blob from the Uint8Array and read it as an ArrayBuffer using a FileReader.
9. Once the FileReader's onload event is triggered, we use structuredClone() to deserialize the ArrayBuffer back into the FlexSearch object.

This approach relies on the structured clone algorithm to serialize and deserialize the FlexSearch object

[nickchomey]

I also recall stumbling on similar info here:

https://docs.partykit.io/guides/handling-binary-messages/
https://docs.partykit.io/guides/persisting-state-into-storage/