[Feature Request] Optimizer for sortable TAG field

[keithchew]

I have been testing the new sortby Optimizer and so far have been very happy with the results, achieving performance gains up to 100x in some use cases.

Currently the optimizer only applies to numeric fields in the sortby, I would like some information or thoughts on whether the optimizer can be extended to cover tag fields.

Perhaps initially just some feedback on whether this is feasible or not, and if yes, I would be happy to jump deeper into the code and do some experiments.

[adrianoamaral]

hey @keithchew thanks for reaching out! Glad that you're playing with the Sort optimisations. I would love to hear your inputs for the TAG implementation.

[keithchew]

Hi @adrianoamaral

Currently, I only have the dataset which I plan to use for the test, ie creating 2mil sets with both numeric and tag numbers:

hset test:1 fnum "1" ftag "0000001"
hset test:2 fnum "2" ftag "0000002"
...
hset test:2000000 fnum "2000000" ftag "2000000"

then, using the profile to compare:

ft.profile test:idx search query "*" sortby fnum desc WITHOUTCOUNT limit 0 5
ft.profile test:idx search query "*" sortby ftag desc WITHOUTCOUNT limit 0 5

The first profile will be optimized as it has a numeric field. The second will not be optimized as it is not numeric, but hopefully with an enhancement to the Optimizer, it can be improved to be comparable to the first.

[github-actions]

This issue is stale because it has been open for 60 days with no activity.