Suggestions for temporal distance function

[priamai]

Hi there,
Probably related to: #279

I am implementing an algorithm called ST( spatio temporal) DBSCAN which is very simple it basically requires 2 distances one for the spatial field (can be your vector dense or sparse field) and one for the temporal field (integer such as linux epoch).

Now in ES there is provision to boost queries by the pivot query on the date field: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-distance-feature-query.html
However:

• I don't believe it can be combined with non bool queries such as the other sparse field
• I have to reverse the relevance score to retrieve the original distance
• I am unable to query elements below a certain distance, I will have to incrementally take the top documents until I get to the required condition.

So I am just wondering if it makes sense to add a simple integer field in elastiknn which will allow to calculate a simple distance abs(t1-t2) between integers.

This will allow me to then retrieve points close to each other in space and time, I will still have the problem to unable to specify a specific range but at least is an improvement.

Let me know your thoughts.

[alexklibisz]

From the docs you linked:

To use the distance_feature query, your index must include a date, date_nanos or geo_point field.

Maybe there's a way to add support for the elastiknn_vector type? Someone would have to dig into the Elasticsearch internals to see how they implement this currently. I doubt I'll have time for this anytime soon. If someone else wants to look into it and make a PR, there are some developer docs here: https://github.com/alexklibisz/elastiknn/blob/master/developer-guide.md

I have to reverse the relevance score to retrieve the original distance

There's no way around this - the score has to be positive, and some of the common distance functions can return negative values (e.g., cosine is in [-1, 1]).

I am unable to query elements below a certain distance, I will have to incrementally take the top documents until I get to the required condition.

Yeah. That would require something like #279

[alexklibisz]

As an aside, Elastiknn is really not intended for implementing iterative clustering algos. Any clustering algo that has Elasticsearch in the loop will probably be painfully slow. Even just the JSON serialization and network time are a big drag on performance. Elastiknn was pretty strictly intended for the use-case of just searching vectors that have already been computed by some other means.

[priamai]

Agreed!