ROADMAP 2024

[writinwaters]

v0.2.0

• Supports sparse vector index. [Feature Request]: Support sparse vector #1174
• Supports tensor data type. [Feature Request]: Supports tensor data type #1179
• Supports cosine similarity. [Feature Request]: Supports cosine similarity #1176
• Supports configurable reciprocal rank fusion operator. [Feature Request]: Supports configurable reciprocal rank fusion operator #1177
• Multiple recall supports more than two ways. [Feature Request]: Multiple recall supports more than two ways. #1178
• HTTP API: Supports GET/SET variables. [Feature Request]: HTTP API: Supports GET/SET variables. #1180
• Embedded infinity. [Feature Request]: Embedded infinity #1181
• Export data into CSV and JSONL file type. [Feature Request]: Export data into CSV and JSONL file type #1175
• Unify the background computation task running into task executor. [Feature Request]: Unify the background computation task running into task executor. #1182
• Infinity database backup and restore tools. [Feature Request]: Infinity database backup and restore tools. #1183
• Integrates later interaction models, such as Colbert.

v0.1.0

• Building HNSW index in parallel. [Feature Request]: Add parallel construction of knn index. #341
• Supports aggregate operation. [Feature Request]: Supports aggregate operation. #357
• Supports order by (sort) operation. [Feature Request]: Support Order By #339
• Supports limit operation. [Feature Request]: Support Limit #362
• Supports order by + limit as top operation. [Feature Request]: Support order by + limit as top operation #408
• Secondary index on structured data type. [Feature Request]: Secondary index #360
• New full text search. [Feature Request]: New full text index #358
• Minmax of column data. Minmax of column data. #448
• Bloomfilter of structured data column. [Feature Request]: Add bloomfilter to segment/block column. #467
• Refactor ColumnVector: Reduce serialization times as much as possible. [Feature Request]: Refactor ColumnVector #449
• Supports new data type: date. [Feature Request]: Support DATE data type #371
• Supports new data type: bool. [Feature Request]: Support BOOL data type #394
• Refactor meta data: Provides a clear interface to access meta data, instead of traversing meta data tree. [Refactor]: Refactor catalog #368
• Refactor error handling: Provides normalized error code and error message. [Feature Request]: Unified error message and error code #439
• Segment GC and segment compaction. [Feature Request]: Segment compaction #466
• Refactor WAL with physical log, instead of logical log. [Feature Request]: WAL physical log #431
• Asynchronous index building: Data become query-able once imported / inserted.
• Storage clean up: Deprecated index/segment/catalog ... files need to be clean up to save the disk space. [Feature Request]: Clean up #635
• Incremental checkpoint. [Feature Request]: Incremental checkpoint #438
• New python API to show database system value. [Feature Request]: Show Variable #495
• New python API to explain the query plan. [Feature Request]: Add Explain api for infinity_sdk #496
• HTTP API [Feature Request]: HTTP API support #779

[yuzhichang]

CI improvement: post logs of infinity when CI failure, use Ubuntu 20.04 as base of dev image.
Fuzz test of infinity.

[cjkbjhb]

Secordary index on structured data type.
--->
Secondary index on structured data types.

Here is a mis-spelling error.

[JinHai-CN]

• Secondary

Fixed and thank you.

[yuzhichang]

compatibility testing

image	tag	refer
centos	7 8	https://hub.docker.com/_/centos/
ubuntu	20.04 22.04 24.04	https://hub.docker.com/_/ubuntu https://releases.ubuntu.com/
debian	8 9 10 11 12	https://hub.docker.com/_/debian https://www.debian.org/releases/
opensuse/leap	15.0 15.1 15.2 15.3 15.4 15.5	https://hub.docker.com/r/opensuse/leap
openeuler/openeuler	20.03   22.03	https://hub.docker.com/r/openeuler/openeuler
openanolis/anolisos	8.6 23	https://hub.docker.com/r/openanolis/anolisos
openkylin/openkylin	1.0	https://hub.docker.com/r/openkylin/openkylin

[Kelvinyu1117]

I would like to contribute to this project, which issue would be a good start?

[JinHai-CN]

@Kelvinyu1117
We do have a couple of issues that might work for contributors new to this project.

1. Add minmax information to blocks/segments in the current datastore. This information is primarily used for data filtering. (Minmax of column data. #448)
2. Implement a bloomfilter for the blocks/segments to enhance point queries. ([Feature Request]: Add bloomfilter to segment/block column. #467)
3. Currently, query results are stored in memory in a columnar format. However, the client expects the results in Apache Arrow format. At the moment, the format conversion is executed on the Python client, but this worsens the performance, so we plan to convert the results to Apache Arrow format on the server side before sending them to the client.
4. There are several optimizer rules to implement, such as constant folding and simplification of arithmetic expressions, which are not yet on the roadmap. Feel free to work on them if interested.
5. We have additional complicated tasks not listed here. For instance, the current executor operates with one thread per CPU. We're considering using coroutine to enhance efficiency, but we don't have a solid solution yet. If you have experience in this area, you are very welcome to propose your solution.
6. We understand you're interested in contributing C++ code. However, if that's not the case, there's also unimplemented Python code, such as test cases and the Python SDK API.

[abdullah-alnahas]

Your work is exceptional! I would like to propose that, considering the current landscape, incorporating binary quantization and ColBERT-like ranking would be crucial for any vector database.
Apologies for commenting on the road map issue instead of creating a separate feature request.

[JinHai-CN]

Your work is exceptional! I would like to propose that, considering the current landscape, incorporating binary quantization and ColBERT-like ranking would be crucial for any vector database. Apologies for commenting on the road map issue instead of creating a separate feature request.

Nice, we will put this request into v0.2.0 release.

[niebayes]

@JinHai-CN Hi, I have experience in developing a database using Arrow. Is the issue that converting query results to Arrow format still active? I'd like to take it.

[JinHai-CN]

@niebayes #1198, issue is created and we can discuss the requirement in that issue.