-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"sentence-transformers/all-MiniLM-L6-v2" - incorrect embeddings and rather slow speedup. #22
Comments
Thanks for flagging this. Will investigate. First guess would be that these are two different models possibly since FlagEmbedding is different from Sentence Transformers. |
please try v0.1.1 |
On my system the code above still vails with v0.1.1 Have you tried the above code? @NirantK For models, i use "sentence-transformers/all-MiniLM-L6-v2" on both sides. |
sentence-transformers=2.22 sentence = ["This is a test sentence."]
arrays are not almost equal to 1 decimals
Mismatched elements: 2 / 384 (0.521%)
Max absolute difference: 0.81547204
Max relative difference: 2334.82220783
x: array([ 1.4e-02, -1.9e-02, 6.3e-03, 3.0e-02, 1.8e-02, -1.5e-02,
-8.6e-03, 1.3e-02, 1.1e-02, -4.0e-03, -6.7e-04, 7.2e-03,
5.4e-03, 1.2e-02, 1.5e-03, -4.8e-03, 1.8e-02, -1.6e-02,...
y: array([ 8.4e-02, 5.8e-02, 4.5e-03, 1.1e-01, 7.1e-03, -1.8e-02,
-1.7e-02, -1.5e-02, 4.0e-02, 3.3e-02, 1.0e-01, -4.7e-02,
6.9e-03, 4.1e-02, 1.9e-02, -4.1e-02, 2.4e-02, -5.7e-02,... |
FYI for "BAAI/bge-base-en" i get a cosine_sim of |
Hey! I've not done a thorough analysis, but I've also had some really quirky results with the |
Hey, I can confirm that the sentence-transforms quantization isn't perfect. The cosine similarity is lower than we'd like. Thanks for flagging this. |
I am curious if this is still present? I want to use |
I wrote a small unit test. Your models seem to have a couple of issues:
The text was updated successfully, but these errors were encountered: