Version 2.4.6

This is a maintenance release for RediSearch 2.4

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bugfixes:

• #2716 Removed assert that could cause crashes with replica-of (MOD-3008, MOD-3012)
• #2734 ON_TIMEOUT RETURN policy fix: return results obtained until timeout rather than discarding them
• #2714 Memory leak on non-TLS setup in Coordinator

Version 2.4.5

This is a maintenance release for RediSearch 2.4

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bug fixes:

• #2702: INKEYS combined with Vector Similarity caused server unresponsiveness (MOD-2952)
• #2705: Incorrect results when deleting a document that was skipped at index time
• #2698: Synonyms in Chinese

Improvements:

• #2694: Performance: In a TEXT field, skip term iterator if term does not appear in requested field

2.4 GA (v2.4.3)

This is the General Availability release of RediSearch 2.4

Headlines:

RediSearch 2.4 introduces a new capability, Vector Similarity Search (VSS), which allows indexing and querying vector data stored (as BLOBs) in Redis hashes.

It also introduces a new query syntax to address query parser inconsistencies found in previous versions of RediSearch. Users can now choose between Dialect version 1 (to keep existing query parser behavior) or Dialect version 2 (to switch to the updated behavior).

All VSS queries or any query using the PARAMS option must use Dialect version 2

What's new in 2.4?

• FT.CREATE is extended to support the creation of 2 popular types of vector indexes:

  • FLAT Index - This type of index is used when the recall is more important than the speed of query execution. A query vector will be compared to all vectors in a flat index. The search results will return the exact Top K Nearest Neighbors to the query vector
  • Hierarchical Navigable Small World (HNSW) Index. A modified implementation of the library written by the author of the influential academic paper. An HNSW vector index is used when the speed of query execution is preferred over recall. The results returned are Approximate Nearest Neighbors (ANNs).
    You can try out different HNSW index parameters (M, EFCONSTRUCTION, EFRUNTIME) to improve the “recall vs speed” balance

• Use FT.SEARCH to retrieve the Top K hashes with the most similar vectors to a given query vector

• Hybrid queries - Discover the power of hybrid queries in FT.SEARCH
  Use hybrid queries to retrieve Redis hashes matching a combination of vector and non-vector search criteria. Non-vector search criteria can include expressions combining NUMERIC, TEXT, TAG, GEO fields.
  Hybrid queries are commonly used in modern e-commerce search applications featuring “visual” similarity + metadata similarity.
  For example, find products that are visually similar to a given image within a price range and/or geo-location.
  All in one query.

• Use FT.CONFIG to set DEFAULT_DIALECT at the module level. By default, DEFAULT_DIALECT is set to 1

• Override DIALECT. It is possible to override the module-level dialect for a specific command run. You can specify dialect when executing any of the following commands: FT.SEARCH, FT.AGGREGATE, FT.EXPLAIN, FT.EXPLAINCLI,FT.SPELLCHECK
  If you do not specify dialect when running any of these commands, the default module-level dialect value will be used

Details

Features and improvements:

• #2671 Add Dialect support
• #2645 Ignore NULL values in JSON documents on indexing. (prior behaviour would ignore the entire document)

Performance enhancements (since 2.4-RC1):

• #2647 normalize vector once for ad-hoc flat search
• #2638 optimized hybrid query when no scores are required
• #2653 updating specific field load optimization rule
• #2670 Use REDISMODULE_EVENT_SHUTDOWN to clear resources

Security and privacy (since 2.4-RC1):

• #2584 Added support for tls-key-file-pass capability (MOD-2086)

Bug fixes (since 2.4-RC1):

• #2651 Client freeze on docs expire during query
• #2641 Memory leak in Coordinator
• #2654 VecSim hybrid query - return empty iterator for invalid intersection child iterator

Introducing DIALECT

RediSearch 2.4.3 introduces a new query syntax to address query parser inconsistencies found in previous versions of RediSearch. Users can now choose between:

• Dialect version 1 (to keep query dialect as in RediSearch 2.2) or
• Dialect version 2 (to use updated dialect).

Existing RediSearch 2.2 users will not have to modify their queries as the default dialect is 1
HOWEVER, and over time, we encourage all RediSearch users to gradually update their queries to use dialect version 2

Background

We found that under certain conditions, some query parsing rules did not behave as originally intended
Particularly, some queries containing the operators below could return unexpected results

1. AND
2. quotes, ~, -, and % (exact, optional, negation, fuzzy)
3. OR

To minimize the impact on existing RediSearch users not affected, it was decided that a DIALECT setting could be introduced to allow:

• Existing queries to run without any modification (DIALECT 1)
• New queries to benefit from the updated query parsing behavior (DIALECT 2)

Examples of queries impacted

Your existing queries may behave differently under different DIALECT versions, if they fall into any of the following categories:

1. Your query has a field modifier followed by multiple words.

Consider this simple query @name:James Brown.

Here the field modifier @name is followed by 2 words James and Brown

In DIALECT 1, this query would be interpreted as find James Brown in the @name field.
In DIALECT 2, this query will be interpreted as find James in the @name field AND Brown in ANY text field. In other words, it would be interpreted as (@name:James) Brown
In DIALECT 2, You could achieve the default behavior as in dialect 1 by updating your query to @name:(James Brown)

2. Your query uses quotes, ~, -, % (exact, optional, negation, fuzzy)

Consider a simple query with negation -hello world.

In DIALECT 1, this query is interpreted as "find values in any field that does not contain hello AND does not contain world". The equivalent of -(hello world) or -hello -world.
In DIALECT 2, this query is interpreted as -hello AND world (only hello is negated)
In DIALECT 2, You could achieve the default behavior as in dialect 1 by updating your query to -(hello world)

Another example illustrating the differences in parser behavior
hello world | "goodbye" moon.

In DIALECT 1, this query is interpreted as searching for (hello world | "goodbye") moon.
In DIALECT 2, this query is interpreted as searching for either hello world OR "goodbye" moon

Notes:

• This is the first GA version of 2.4. The version inside Redis will be 2.4.3 in semantic versioning. Since the version of a module in Redis is numeric, we could not add a GA flag.
• Minimal Redis version: 6.0.0

Version 2.2.10

This is a maintenance release for RediSearch 2.2

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Improvements:

• Mac M1 support
• #2645 Ignore NULL values in JSON documents on indexing. (prior behaviour would ignore the entire document)
• #2623 Improved Multi Sortby error message for FT.SEARCH

Bug fixes:

• #2641 Memory leak in Coordinator
• #2651 Client freeze on docs expire during query
• #2670 Memory leak in RediSearch found in Active-Active (MOD-2518)

2.4-RC1 (v2.4.2)

This is the first Release Candidate of RediSearch 2.4

Headlines:

RediSearch 2.4 introduces a new capability, Vector Similarity Search (VSS), which allows indexing and querying vector data stored (as BLOBs) in Redis hashes.

This release ships an important bug fix addressing operator precedence rules in the RediSearch query syntax. Please see the "Breaking behaviour" section below as you may need to update existing FT.SEARCH queries to assure same behaviour when upgrading.

What's new in 2.4

• FT.CREATE is extended to allow creation of 2 types vector indexes:

  • FLAT Index - This type of index is used when recall is more important than speed of query execution. With a flat index, a query vector will be compared to all vectors in the index and the search results will content the exact Top K Nearest Neighbors.
  • Hierarchical Navigable Small World (HNSW) Index. A custom implementation of this influential academic paper. An HNSW vector index is used when speed of query execution is preferred over recall. The results returned are Approximate Nearest Neighbors (ANNs).
    You can try out different HNSW index parameters to improve the “recall vs speed” balance for your dataset

• Use FT.SEARCH to retrieve the Top K hashes with most similar vectors to a given query vector

• Discover the power of hybrid queries in FT.SEARCH .
  Use hybrid queries to retrieve redis hashes matching a combination of vector and non-vector search criteria data. Non-vector search criteria can include NUMERIC, TEXT, TAG, GEO fields.
  Hybrid queries are commonly used in modern e-commerce search applications featuring “visual” similarity + metadata similarity.
  For example, find products that are visually similar to a given image but also within a price range and available in a geo-location.
  All in one query.

Details

Features (since 2.4-M01):

• #2590 Vecsim coordinator allows executing KNN and hybrid queries on a RediSearch cluster.

Improvements (since 2.4-M01):

• #2619 Added and improved hybrid query execution optimisations. An on-demand flat index scan when a small subset of vectors satisfy the query filter. This PR also improves an existing hybrid query execution when a large number of vectors that satisfy query filter.
• #2583 Release index specific information off the main thread (performance enhancement)

Bug fixes (since 2.4-M01):

• #2641 Memory leak in the coordinator (MOD-2716)
• #2623 Multi SORTBY error message for FT.SEARCH
• #2596 Operator precedence rules in the RediSearch query syntax. This PR also changes modifier's behaviour to ensure consistency with other logical operators. Please read section "Breaking Behaviour".

Breaking behaviour

Part of this release is aligning the query syntax operator precedence. There were certain situations where the following rules were not respected in the query syntax of RediSearch:

Operator precedence (in order of appearance):  
1. AND
2. quotes, ~, -, and % (exact, optional, negation, fuzzy)
3. OR

Parentheses can override precedence

(AND in RediSearch is a space and OR is a |)

Your existing queries may be impacted if they fall into any of the following categories:

1. Your query has a field modifier followed by multiple words.

Consequence 1: If a field modifier precedes multiple words or expressions, it applies only to the adjacent expression

Consider this simple query @name:James Brown. Here the field modifier @name is followed by 2 words James and Brown
Previously, the query would be interpreted as find James Brown in the @name field.

From this version on, this query will be interpreted as find James in the @name field AND Brown in ANY text field. In other words, it would be interpreted as (@name:James) Brown

In order to keep the previous behaviour, you should update your query to @name:(James Brown)

2. Your query uses quotes, ~, -, % (exact, optional, negation, fuzzy)

Consequence 2: The operators: quotes, `~`, `-`, and `%` (exact, optional, negation, and fuzzy) are not affecting AND'ing and OR'ing of expressions

Consider a simple query with negation -hello world. Previously, the query would be interpreted as find values in any field that do not contain hello AND do not contain world. The equivalent of -(hello world) or -hello -world.

From now on, the original query would be interpreted as -hello AND world (only hello is negated)

In order to keep the previous behaviour, you should update your query to -(hello world)

Another examples is this query hello world | "goodbye" moon. Previously, this query was interpreted as searching for (hello world | "goodbye") moon. This query is now interpreted as searching for either hello world OR "goodbye" moon

Notes:

• The version inside Redis will be 2.4.2 in semantic versioning. Since the version of a module in Redis is numeric, we could not add a Release Candidate flag.
• Minimal Redis version: 6.0

Version 2.2.9

This is a maintenance release for RediSearch 2.2

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Improvements:

• #2605 Added support for tls-key-file-pass capability (MOD-2086)
• #2583 Release index specific information off the main thread (performance enhancement)

Bug fixes:

• #2436 When indexing JSON documents filters cause no documents to be indexed (MOD-2214)
• #2507 QUANTILE aggregation function outputting wrong values (MOD-2432)
• #2521 contains() with an empty string argument leaves Redis hanging at CPU 100% indefinitely (MOD-2428)
• #2560 Free prefix and cursor efficiently for cases with many indices (MOD-2080)
• #2541 Numeric types for FT.INFO on coordinator
• #2553 Fix union high iterator
• #2404 Update coordination strategy of FlatSearchCommandHandler

2.4-M01 (v2.4.0)

This is the first Milestone of RediSearch 2.4

Highlights

This milestone release introduces a new Vector Similarity Search (VSS) capability, which allows indexing and querying vector data stored (as BLOBs) in Redis hashes.

Details

• FT.CREATE is extended to allow creation of 2 types vector indexes:
  • FLAT Index This type of index is used when recall is more important than speed of query execution. With a flat index, a query vector will be compared to all vectors in the index and the search results will content the Top K Nearest Neighbors
  • Hierarchical Navigable Small World (HNSW) Index. A custom implementation of this influential academic paper. An HNSW vector index is used when query execution speed is more important than recall. While it returns approximate nearest neighbors, you can try out different HNSW index parameters to improve the “recall vs speed” balance for your dataset
• Use FT.SEARCH to query vector data. FT.SEARCH allows you to retrieve the Top K hashes with most similar vectors to a given query vector
• Discover the power of hybrid queries in FT.SEARCH . Use hybrid queries to retrieve redis hashes matching a combination of vector and non-vector data (NUMERIC, TEXT, TAG, GEO). Hybrid queries are commonly used in modern e-commerce search applications featuring “visual” similarity + metadata similarity. For example, find products that are visually similar to a given image but also within a price range and available in a geo-location. All in one query.

To learn more:

• Check out this blog post Building Intelligent Apps with New Vector Similarity Search
• Try out this hands on demo showcasing semantic similarity on Financial news headlines

Notes:

• The version inside Redis will be 2.4.0 in semantic versioning. Since the version of a module in Redis is numeric, we could not add a Milestone flag.
• Minimal Redis version: 6.2

2.2.7

This is a maintenance release for RediSearch 2.2

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bug fixes:

• #2466 FT.PROFILE for FT.AGGREGATE on clustered databases
• #2473 FT.PROFILE with no result processor present
• #2490 Case sensitivity issue in searches of TAG field on JSON

Improvements:

• #2469 Add API for TAG children query nodes

2.2.6

This is a maintenance release for RediSearch 2.2.

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details:

Bug fixes:

• #2362 Crash on empty field name
• #2407 Inconsistency with FT.CREATE and ON: return error for spec w/o fields parameters
• #2392 Remove limit on Geo field percision
• #2440 Fix in NOT(-) iterator
• #2414 Prevent server freeze when FT.SEARCH timeout while sorting
• #2386 Memory leak in numeric field index

Improvements:

• #2424, #2425 Performance improvements for numeric indices
• #2408 API: Added RediSearch_IndexInfo
• #2448 API: Added setLang and setScore

2.0.15

This is a maintenance release for RediSearch 2.0.

Update urgency: MODERATE - Program an upgrade of the server, but it's not urgent.

Details:

Bug fixes:

• #2388 Garbage Collection (GC) for empty ranges in numeric index
• #2409 Introduction of FORK_GC_CLEAN_NUMERIC_EMPTY_NODES module argument to enable #2388 (off by default)
• #325 Used Redis allocator in hiredis (RSCoordinator)
• #2362 Crash on empty field name