New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

does it support Korea and Japanese? #35

Open

sz2three opened this issue Jun 14, 2023 · 1 comment

sz2three commented Jun 14, 2023

it supports Chinese, while does it also work for Korea and Japanese?

Owner

Muennighoff commented Jun 14, 2023

You can check Section 4.4 of the MTEB paper (https://arxiv.org/pdf/2210.07316.pdf) where https://huggingface.co/bigscience/sgpt-bloom-7b1-msmarco is benchmarked on many languages incl. Korean & Japanese against other models. As it hasn't extensively seen them in pre-training it performs rather poorly on them.

You may want to use a different model for those languages (check e.g. this leaderboard to see what's best: https://huggingface.co/spaces/mteb/leaderboard for those languages).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment