Meilisearch
Experimental feature: Hybrid Search and Vector Store #677
Replies: 28 comments 75 replies
-
|
hi, an excellent feature, can we use filters with vector search ? |
Beta Was this translation helpful? Give feedback.
-
|
Hello 👋 Yes! Just use the filters as usual in your vector search request |
Beta Was this translation helpful? Give feedback.
-
|
Hi, if I use OpenAI to create the vector to store in DB but when I search I use Hugingface vector to search is it will work fine or show the right result or not? |
Beta Was this translation helpful? Give feedback.
-
|
Hey @taehun007 👋 No, sorry. You must use the same source of vectors. So, either generate the vectors from Hugging Face or OpenAI for both endpoints. |
Beta Was this translation helpful? Give feedback.
-
|
so can we use openAI embedding to create vector and store it in meilisearch? |
Beta Was this translation helpful? Give feedback.
-
|
yes, if you are comfortable to try you can try rust-bert too, helps getting more instantsearch like feel, not sure about the accuracy |
Beta Was this translation helpful? Give feedback.
-
|
When I tested the meilisearch:v1.3.0-rc.3 vector search, I got results, but didn't see _semanticSimilarity. Strange |
Beta Was this translation helpful? Give feedback.
-
|
Hey @noangel2014 👋 It's because it has been renamed |
Beta Was this translation helpful? Give feedback.
-
|
I've been experimenting with Vector Search with the release in 1.3, but I am having two issues, which I am not sure how to address
I know it's experimental, but this seems to be very drastic issue, so maybe I am not doing it right? Update: |
Beta Was this translation helpful? Give feedback.
-
|
Here is my Rails configuration for this specific index: meilisearch enqueue: :reindex_search do
attribute :id, :title, :alt_titles, :content_rating, :content_type
attribute :available_chapters_count, :users_count
attribute :has_active_sources do
manga_sources.any? { |s| s.deprecated_at.nil? && !s.pending }
end
attribute :manga_source_site_ids do
manga_sources.pluck(:manga_source_site_id)
end
attribute :created_at_unix do
created_at.to_i
end
attribute :classifications do
series_classifications.as_json.map { |c| c.slice('category', 'name') }
end
attribute :_vectors do
EmbeddingRetriever.calculate_vector(description)&.first || []
end |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much @doutatsu for this clear report of the tasks. I now understand the issue: we must accept In the mean time, could you try to not set at all the |
Beta Was this translation helpful? Give feedback.
-
|
For your information, I just fixed the issue in this PR, and it should be merged and released in v1.3.2. You'll now be able to send |
Beta Was this translation helpful? Give feedback.
-
|
Was about to say that setting nothing does work. As well as sending only documents with vectors also works. Look forward to giving a go when the new version comes out, to see if it's all fixed on my side as well. Thanks for a quick fix! |
Beta Was this translation helpful? Give feedback.
-
|
I was also having this issue and found this helpful. I found something new that I also thought was interesting as well. I'm testing this on 629 results and only one has a vector value. its a 384 dimension vector and its coming from a Postgres database where the row data type is a vector from the pg vector extension. It is returned as a string and not as an array from the database from the client I am using. When I try to index all 629 documents with either that one's vector value, or null, it will index 540 documents. If I parse the string so that it sends up an array when its indexed (via the Meili JS client) it'll index all 629 results, one with a vector and the rest with null values for _vectors.
|
Beta Was this translation helpful? Give feedback.
-
|
Hybrid search is not yet in the experimental right? I assume we are looking into it before the 1.4 release. I could set up and use the current vector search right away, I also had a session at "Drupal Camp Pune" and did a small introduction to vectors and hybrid search added it behind a small demo for the users to interact with. (We are trying to use it behind a QnA bot) Let me know how can I help with this hybrid search thing. Even though I am not a rust expert I can give it a try :) |
Beta Was this translation helpful? Give feedback.
-
|
Is it possible to have varying lengths of vectors across documents in an index? documents = [
{ 'id': 1, 'Text': 'I like to eat broccoli and bananas.', '_vectors': embedding_service(["I like to eat broccoli", "I like to eat bananas", "I like to eat broccoli and bananas."]) },
{ 'id': 2, 'Text': 'The Packers won the 2011 NFL Super Bowl with Aaron Rodgers', '_vectors': embedding_service(["The Packers won in 2011", "The Packers won the 2011 NFL Super Bowl with Aaron Rodgers"]) },
]would fail with this message: What I'm looking for is having multiple vectors belong to a document and upon vector search, compute dot product with all vectors for all documents |
Beta Was this translation helpful? Give feedback.
-
|
To add some more detail:
|
Beta Was this translation helpful? Give feedback.
-
|
Just checked the example under "Send Vectorized Documents" in #621 (comment) and it fails as well. Similar approach to what I posted above |
Beta Was this translation helpful? Give feedback.
-
|
Hey @sanjay920. Yes, you should be able to send multiple (any number) vectors per document to an index.
Here as you can see meilisearch reports that your vector didn’t have the same size. |
Beta Was this translation helpful? Give feedback.
-
|
I have some questions regarding this feature:
[1] I am new to the whole vector search so please correct me if I mention something wrong. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @nlgtuankiet , apologies for the late reply!
It is technically possible to change the parameters at index creation time. However we don't have any guidelines to offer as we use instant-distance for HNSH. It might be worth asking in their repo :)
Yes, it should work with both of them! |
Beta Was this translation helpful? Give feedback.
-
Hi everyone, its september 30 today, how is the feature going?, what other potential date can be expect it to be fully stable relatively? Thanks for considering this feature 👍 |
Beta Was this translation helpful? Give feedback.
-
|
I absolutely love this software, I am self-hosting it and it's just amazing how fast it is and how well it 'just works' compared to TypeSense but you can still customize quiet nicely. To clarify, is v1.6 where vectors will be a stable feature? I used it experimentally with 60-70k records for product deals and it was highly relevant, it was great and didn't have any issues indexing thankfully but I am still edgy because it's alpha before I deploy to live site. Again, great work on this all around. |
Beta Was this translation helpful? Give feedback.
-
+1!! How did you create the embeddings for your product deals? |
Beta Was this translation helpful? Give feedback.
-
|
I used OpenAI API for embeddings, was looking at some options on huggingface though too but it was inexpensive enough to do it |
Beta Was this translation helpful? Give feedback.
-
Do you mind sharing what input you used specifically for the product embedding and also what input you used for account/search side embedding? We've been experimenting as well but haven't found a good combo of model and input to have expected outputs. |
Beta Was this translation helpful? Give feedback.
-
|
I trashed my code for it/overwrote my notebook I was using for it, I was just looking to share with ya, only have Typesense testing code left. I believe I just used title and possibly description or a category but don't remember as it was a few months ago. Sorry can't help more. |
Beta Was this translation helpful? Give feedback.
-
|
hi, i just setup a collection with 16 Million 3 dimension vectors basically all possible RGB values normalized , as i inserted them, first 300k records got added fine after that, its painfully slow , the tasks still running after 12 hrs can anyone suggest any limit to what amount of vector we can add ? looking for a solution of 40M records with 1024 dimensions. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @geminigeek 👋 We are working hard on making it possible with Meilisearch v1.6, which will be officially released on January 15, and the first release candidates will be available on December 18. This version will feature the new Arroy library, which will support many more vectors with high dimensionality than Meilisearch can support now. You can read more about it in this article and the soon-to-be-released others. |
Beta Was this translation helpful? Give feedback.
-
|
Great read! Has @irevoire released his blog on incremental indexing? |
Beta Was this translation helpful? Give feedback.
-
|
Hey @clxyder ! |
Beta Was this translation helpful? Give feedback.
-
|
Vector Search Is it possible to search for images by image? |
Beta Was this translation helpful? Give feedback.
-
|
Hello, to perform a search for images by image, you need the following in the upcoming v1.6.0:
curl \
-X PATCH 'http://localhost:7700/experimental-features/' \
-H 'Content-Type: application/json' -H 'Authorization: Bearer foo' \
--data-binary '{ "vectorStore": true }'
curl \
-X PATCH 'http://localhost:7700/indexes/your_index/settings/embedders' \
-H 'Content-Type: application/json' --data-binary \
'{ "image": { "source": "userProvided", "dimensions": 512 } }'
curl \
-X POST 'http://localhost:7700/indexes/your_index/documents' \
-H 'Content-Type: application/json' \
--data-binary '{"id": 0, "title": "Other fields of your documents as normal", "_vectors": {"image": [0.00397, 0.553, ..., 0.0] } }'
curl \
-X POST 'http://localhost:7700/indexes/your_index/search' \
-H 'Content-Type: application/json' \
--data-binary '{"vector": [0.3967, 0.333, ..., 0.1], "hybrid": {"semanticRatio": 1.0, "embedder": "image"} }'As Meilisearch does not provide embedding generation for images at the moment, you will have to provide the vectors corresponding to the images in your document and the image in your query yourself, like demonstrated above. |
Beta Was this translation helpful? Give feedback.
-
|
As far as I understand hugging face can only calculate text vectors, openai is not very clear yet. Does Meilisearch have plans to develop its own vector calculation engine? It is best to support vector calculations of text, images, audio and video |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Avey777 , we're not planning to develop our own vector embedding engine. Our goal is to provide the best search experience regardless of your chosen model. You can provide vectors generated by your preferred embedder to get image search like @dureuill mentioned above. |
Beta Was this translation helpful? Give feedback.
-
|
Hey folks 👋 v1.6 has been released! 🦊 Hybrid search and auto-embedding features are now available for your use ✨ Check out our documentation to learn how to use them. We're looking forward to your feedback! |
Beta Was this translation helpful? Give feedback.
-
is this stable and ready for prod? 😃 |
Beta Was this translation helpful? Give feedback.
-
|
error causing example https://www.meilisearch.com/docs/learn/experimental/vector_search#generate-auto-embeddings-with-openai -- source for OpenAI should be openAi |
Beta Was this translation helpful? Give feedback.
-
|
Thank for the feedback, just opened meilisearch/documentation#2687 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for awesome search tool! |
Beta Was this translation helpful? Give feedback.
-
|
hi @macraig, is it possible to hybrid search with user-provided embeddings? I read through the document but seem like they are mutual exclusive? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @dureuill thank you very much for the guide 😍 could you please put this in the documentation, I have no idea how to do it 🙌 |
Beta Was this translation helpful? Give feedback.
-
|
Hello, it is given as a usage example in the page describing the feature on notion We can ask @guimachiavelli if he would be OK to add this use case, but I think the idea was to be minimalistic on the documentation for this feature since it is experimental, and the use case is advanced already. For future reference you can contribute to our documentation by following our contributing documentation 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @dureuill I tried the hybrid search {
"q": "...",
"offset": 0,
"limit": 20,
"attributesToRetrieve": [...],
"attributesToHighlight": [
"description"
],
"highlightPreTag": "<em>",
"highlightPostTag": "</em>",
"hybrid": {
"semanticRatio": 0.5,
"embedder": "openai-text-embedding-ada-002"
},
"vector": [...]
}response: {
"hits": [
{
"description": "...",
"id": "...",
"groupId": "...",
"_formatted": {
"description": "...",
"id": "...",
"groupId": "..."
}
},
{
"description": "...",
"id": "...",
"groupId": "...",
"_formatted": {
"description": "...",
"id": "...",
"groupId": "..."
},
"_semanticScore": 0.9361339
}
],
"query": "...",
"vector": [...],
"processingTimeMs": 24,
"limit": 20,
"offset": 0,
"estimatedTotalHits": 2
}in the response, hits contain 2 results, 1 for normal search (without the |
Beta Was this translation helpful? Give feedback.
-
|
Hello @nlgtuankiet I reproduce the issue, I opened a new issue about it, I think I have a fix. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, |
Beta Was this translation helpful? Give feedback.
-
|
I've wanted to add some more recent and well-performing embedders, like E5 models or even mpnet model, which is the 3rd most popular Sentence Similarity model on Hugging Face. But as I encountered errors and asked in support, I was told only BERT models for autoembedding with HuggingFace are supported at the moment, which makes it impractical for me to use built-in Meilisearch capabilities for this and I have to use a separate service, where I can use better models. Here's a good link discussing differences and why many mainstream BERT embedders are not as good as new alternatives: https://blog.metarank.ai/from-zero-to-semantic-search-embedding-model-592e16d94b61 So it would be great if support for more models could be added. |
Beta Was this translation helpful? Give feedback.
-
|
Hey, even if you can't use the hybrid search directly, can't you use the vector store embedded in meilisearch to get better performances? |
Beta Was this translation helpful? Give feedback.
-
|
@irevoire I meant that it doesn't let me index anything, due to the model I am trying to use. Unless I am misunderstanding what you meant |
Beta Was this translation helpful? Give feedback.
-
|
Anyone knows how to monitor the state of the auto-embedding? I configured the embedder using the embedders: {
default: {
source: 'huggingFace',
model: 'BAAI/bge-base-en-v1.5',
documentTemplate:
"Title: {{doc.title}}. {% if doc.subheading %}Subheading: {{doc.subheading}}. {% endif %}Content: {{doc.content}}."
}I have 100k documents in the index. And it is running for a while already in my local machine, so I would like to debug how long the full auto-embedding would take. Is there an endpoint that could give me information of the current embedding? |
Beta Was this translation helpful? Give feedback.
-
|
Hello 👋, We don't have this at this point. You could restart the indexing after putting the Looking at your embedders: {
default: {
source: 'huggingFace',
model: 'BAAI/bge-base-en-v1.5',
documentTemplate:
"Title: {{doc.title}}. {% if doc.subheading %}Subheading: {{doc.subheading}}. {% endif %}Content: {{doc.content|truncatewords: 50}}."
}As a point of reference, to embed 30k documents with a rendered prompt of about 50 words per document I needed about 30 mins on a mac M1. The rest of the indexing process was negligible (about 3s) |
Beta Was this translation helpful? Give feedback.
-
|
Good to have a reference like that. That amounts to around 0.0003 s/character in M1. My M2 should take maybe 0.00025 s/character. That should have taken around 45 minutes for my dataset, and it was running for 12h, so there is definitely something off. Did you run meilisearch on Docker or natively @dureuill? |
Beta Was this translation helpful? Give feedback.
-
|
Hello. I'm running natively. I'm not certain the time is going to be linear. |
Beta Was this translation helpful? Give feedback.
-
|
Since 25/01 OpenAI added two new embedding model:
They also added an optional field that can limit the number of dimensions for both new models. As of right now only
see: OpenAI blog post |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Gosti , the new OpenAI models and the optional dimension parameter will be available in our upcoming v1.7 release, which will be out on March 11th. |
Beta Was this translation helpful? Give feedback.
-
|
So you have an |
Beta Was this translation helpful? Give feedback.
-
|
Hi @skerit , this is something we want to support in the near future. Would you mind sharing a bit more about your use case and your expectations? It will help us shape the solution. |
Beta Was this translation helpful? Give feedback.
-
|
Next to being able to select "OpenAI", I would also expect the ability to select an "OpenAI-compatible provider". So my usecase would be: instead of having to use OpenAI, I'd be able to directly use any other compatible API. Like Mistral. Or even a self-hosted one (like LocalAI) |
Beta Was this translation helpful? Give feedback.
-
|
Hey @skerit 👋 We're considering a general REST embedder, would that serve your need? |
Beta Was this translation helpful? Give feedback.
-
|
We also landed ollama support in main this morning: https://meilisearch.notion.site/v1-8-AI-search-API-usage-135552d6e85a4a52bc7109be82aeca42#830277b016ea44b2ba2517ca067ebfc5 |
Beta Was this translation helpful? Give feedback.
-
|
Hi everyone 👋 v1.7 has been released! 🐇 Check out our documentation for more info. As always, we're looking forward to your feedback! |
Beta Was this translation helpful? Give feedback.
-
|
Can the localized deployment version use vector search functionality? |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure I understand what you mean by "localized deployment version". If you mean an open-source Meilisearch version that you deploy on your own (self-hosted), then the answer to your question is yes. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @abdirahmn1 , we're actively iterating over our solution at the moment. We'll remove the Experimental flag once we're confident the API is stable, but it's a few months away (with no concrete date for now). |
Beta Was this translation helpful? Give feedback.
-
|
I'm considering displaying documents similar in content to a given document using vector search. All documents already have vectors set with embedding (Text-embedding-3-small). Is there a way to retrieve the _vector value of a starting document to search for similar documents? Or, is there a method to obtain documents similar to a given one based on the document id? Currently, I use the content of the starting document as a query and recalculate the vector value according to the embedding settings before conducting a vector search. However, recalculating vectors seems wasteful both in terms of speed and cost. Could you please suggest a better approach? |
Beta Was this translation helpful? Give feedback.
-
|
Hey, it's my first time with Meilisearch, so I'm not sure whether it's just a beauty of experimental features, or I'm just doing something wrong. Every time I run hybrid search with What is also very weird, lots of those items have a ranking score higher than 1, which makes no sense because it's against what is said there and this simply doesn't match, this content should get a score around zero as it has no common parts with the search query. What is also very weird, lots of those items have a ranking score higher than I'm not doing any crazy stuff, just one searchable field with a string, one embedder based on OpenAIs Obviously normal search works just fine, returns only relevant results. |
Beta Was this translation helpful? Give feedback.
-
|
I've got an answer on the Discord - it's in the works, there will be a |
Beta Was this translation helpful? Give feedback.
-
|
Hello 👋
Oh I probably haven't been clear in the chat. The We might release a prototype so that y'all can try it before v1.9, though. |
Beta Was this translation helpful? Give feedback.
-
|
Most people use vector database to build RAG systems, but its not clear how to do it with Meilisearch The documentation states: However, for one document, I need multiple vectors to find the most accurate part of the document. For example, if my documents are research papers, there are several fields: title, dates, authors... and there is a field called content. According to your documentation, the only way to vectorize this document is to create an embedder with a document template like "{{ title }} written by {{authors}} on {{date}}, content: {{content: truncate(20)}}". But I would like to be able to vectorize the entire document to return the most relevant parts during a search. So when users search, they can see which part of the document refers to their search. How can I do this? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @underthesand , the document templates are optional. If you don't specify one, Meilisearch will use the default one below, which vectorizes the entire document by listing all the fields by their name and value.
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your answer @macraig . The fact of vectorizing the entire document (for an article, for example) is only effective if it is done in a segmented way, and during a search, we can retrieve the most relevant parts of the document in relation to the query. I may have misunderstood, but from reading the documentation, it seems that Meilisearch vectorizes the entire document into a single vector, which does not allow the most relevant segments to be retrieved and which gives a rather crude representation of the document. I work with other vector databases; the standard pipeline is: Simple model, one field: Complex model, multiple field embeddings: Meilisearch model, one field, no splitting: Example : Article with content and tags. I want to search wich part of the content are similar to the query or i want to only query the tags to know the sementic proximity with the query... This is not a criticism; your model has the advantage of being simple and efficient but I find it rather unsuitable for many use cases, notably RAGs. I wonder if it is your wish/priority to go further. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your detailed reply! We're definitely interested in going further. We're keeping the feature as experimental (opt-in) for now so we can iterate based on our users' needs without introducing breaking changes to the engine. Comments like yours help us know where we're lacking and how we can improve the feature before making it GA. Feel free to continue contributing to the discussions with any other feedback you have! |
Beta Was this translation helpful? Give feedback.
-
|
Hey @underthesand 👋 |
Beta Was this translation helpful? Give feedback.
-
|
Hi genereal question on how the sdks should work: (obviously) Can adjust the PR accordingly ^^ |
Beta Was this translation helpful? Give feedback.
-
|
Hello,
I'm not sure what you mean by "adding env variables to the SDKs as the default", could you please elaborate?
please note that OpenAI's API key can be set either through the dedicated thank you for your PR ❤️ |
Beta Was this translation helpful? Give feedback.
-
|
No worries, that is the least that I can do for using it in a product we build on.
I mean: Should the sdk react the same way as the server would to said environment variables (i.e. a overriding of
Ah thanks. I think I completely overread them. |
Beta Was this translation helpful? Give feedback.
-
I don't think so
I think so, otherwise it might be surprising to users of the SDK. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks 👍🏻 |
Beta Was this translation helpful? Give feedback.
-
|
Hi everyone 👋 v1.8 has been released! 🪼 We've added a new REST embedder source and a distribution shift setting to fine-tune your hybrid ranking scores. There are also some breaking changes in the search response to take into account, so make sure to check the release notes before updating. As usual, let us know what you think! |
Beta Was this translation helpful? Give feedback.
-
|
The features are fantastic, and the responses are incredibly quick! I do have a question that's been puzzling me for a few days now: How can I find documents related to a specific document? I tried using a hybrid search, but it just lists all the documents by decreasing relevancy. Since the relevancy varies widely across different documents, I can't use a simple cutoff. I'm currently using v1.7.1. Is there any other way to achieve the results I need? |
Beta Was this translation helpful? Give feedback.
-
|
Hello and thanks for the kind words ❤️
We have a plan to add a feature to get a document by similar documents. It would let you input a document ID, and return documents similar to the target document. Would that satisfy your use case? I'm not certain I understand why you would need a relevancy cutoff for this use case, feel free to elaborate 🙂 We hope to release it in v1.9 (RC.0 in June, release the first of July). We'll keep you updated :-) |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that is exactly my use case. Thanks for informing. |
Beta Was this translation helpful? Give feedback.
-
|
Are there any updates on the stabilisation of this feature? Really want to start using it in production, but waiting till it's not an experimental feature |
Beta Was this translation helpful? Give feedback.
-
|
Hi @doutatsu , we're still iterating on the feature and improving some performance aspects of it before we consider it stable. To give you some context, for v1.9 (which will be out July 1st) we're planning to move all vector storage to the vector store and avoid regenerating embeddings when you import a dump. We don't have a fixed stabilization date, but it won't be sooner than v1.10 (out on August 26th). In the meantime, please let us know if you have any further feedback on the feature. |
Beta Was this translation helpful? Give feedback.
-
|
No worries, just wanted to understand the timeline a bit better. This gives me some clarity, thanks |
Beta Was this translation helpful? Give feedback.
-
Regarding the storage part, an excellent solution would be binary quantization of vectors. It maintains 95% to 99% of the retrieval performance with significantly less storage because the vectors are binary instead of floating-point numbers. Regarding the speed, I think the necessary algorithms and data structures are already available. There are many established and emerging players in the vector search field that claim impressive speeds. For example, a new library claims "0.1 milliseconds query latency on million-scale vector datasets." Therefore, achieving this should be feasible. All that being said, considering the current vector search landscape, natively implementing Colbert ranking is indispensable for a top-choice, go-to database solution. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @abdullah-alnahas 👋 Thank you for taking the time to look at this feature. We already plan to work on binary quantization on our internal vector store. It would be awesome to reduce the storage size by around 32x 🚀 Regarding the speed, we are pretty happy with it, but we still want to measure and compare it with other solutions. For now, Meilisearch/arroy is very pleasant to work with, and the quality and speed of this library are very good! I will probably need more information about Colbert's ranking. Do you have any links or something? |
Beta Was this translation helpful? Give feedback.
-
|
Hey @Kerollmops! 👋 I'm thrilled about the prospect of getting BQ upstreamed to MS! Here are some resources that you might find useful about ColBERT and its late interaction algorithm
In my opinion, the key innovation of ColBERT is not just the embedding model they train/finetune, but rather the late interaction algorithm. The algorithm calculates the relevance scores of documents to a given query by first embedding each token of the query and each token of each document. It then computes the dot product between the query embeddings and document embeddings, applies max-pooling to the dot product matrix to extract the most relevant information, and sums up the resulting scores to obtain a final relevance score for each document. Finally, the algorithm sorts the documents based on their relevance scores and returns the top-k documents. |
Beta Was this translation helpful? Give feedback.
-
|
Is there a known memory leak issue on version 1.8.1 with added vector store? As can be seen, prior to adding vector info to the index (26k documents with 768 vector space) I've had stable RAM consumption just below 1GB. But after adding the vector store, even when not in use, there seems to be a memory leak 
|
Beta Was this translation helpful? Give feedback.
-
|
hey @doutatsu, thanks for the report, I can think of multiple things going on here:
Hope we'll get to the bottom of this, have a nice week ☀️ |
Beta Was this translation helpful? Give feedback.
-
Here is my rails configuration - I simply added meilisearch enqueue: :reindex_search do
attribute :id, :title, :alt_titles, :content_rating, :content_type, :description
attribute :available_chapters_count, :users_count
attribute :has_active_sources do
manga_sources.any? { |s| s.deprecated_at.nil? && !s.pending }
end
attribute :manga_source_site_ids do
manga_sources.pluck(:manga_source_site_id)
end
attribute :created_at_unix do
created_at.to_i
end
attribute :score do
score.to_f
end
attribute :classifications do
series_classifications.as_json.map { |c| c.slice('category', 'name') }
end
add_attribute :_vectors do
{
'ai/google/models/text-embedding-004' => embedding.present? ? embedding : nil
}
end
searchable_attributes %i[title alt_titles]
attributes_to_highlight %i[title alt_titles]
sortable_attributes %i[created_at_unix available_chapters_count users_count score]
filterable_attributes %i[
id created_at_unix content_rating content_type manga_source_site_ids
has_active_sources classifications score
]
pagination max_total_hits: 50_000
end |
Beta Was this translation helpful? Give feedback.
-
|
Your configuration looks rather good, just also add a list of That list should not contain the That way, search response won't contain the |
Beta Was this translation helpful? Give feedback.
-
|
That seems to have done the trick @dureuill, thanks for the help! |
Beta Was this translation helpful? Give feedback.
-
|
Glad that helped 😊 |
Beta Was this translation helpful? Give feedback.
and others
-
Meilisearch v1.3 (released July 31) introduces a Vector Store feature:
_vectorsfield.vectorquery field. The_semanticScorefield is added to the resulting documents. It represents a dot product of the distance between the nearest vector and thevectorfrom the search query.Meilisearch v1.6 (released January 15th) improves on the Vector Store feature:
Meilisearch v1.7 (released March 11th) improves on the Vector Store feature:
Keywords: Semantic Search, Vector Search, Embeddings Search, Hybrid Search.
Experimental feature abstract
Creating one or multiple
embeddersfor an index triggers a new step in the indexing process where embeddings are generated for each indexed document.Passing
"hybrid": {}to a query from the/indexes/{:indexUid}/searchor/multi-searchperforms both a keyword search and a vector search. If no"vector"was provided, it is generated from the"q"field.How to use the feature?
Please refer to the public API page
What is an experimental feature
By enabling this feature via the
/experimental-featuresroute, you opt into the following:embedderssetting can change in a breaking way between two minor versions of Meilisearch.You can use this feature in production but be prepared to update your code from one version to the next.
Why is this feature not stable yet?
Storing the vectors is currently very expensive, and retrieving them is too. We hope to make progress on that. We are unsure of the API surface we want to expose, even if the current one seems correct.
🗣️ You are welcome to give feedback about the score details or ask any question on its usage; we are eager to collect feedback on the feature
When will the feature potentially be stable?
[Updated] Due to the large increase in API surface and us missing previous estimates, we cannot provide an estimate for the time being
To fully disable the feature, you need to delete the
embedderssetting from any index using the feature by callingDELETE http://localhost:7700/indexes/<index_using_the_feature>/settings/embeddersbefore callingPATCH 'http://localhost:7700/experimental-features/'withvectorStoreset tofalse.experimental-featuresrouteBeta Was this translation helpful? Give feedback.
All reactions