Testing a model from HuggingFace

[mina-taraghi]

Hello,

I would like to run the benchmark for a few models hosted publicly on HuggingFace and I was wondering if that is possible? I saw that modelgauge has a plugin for Huggingface but I am not sure how it works in modelbench.

I would really appreciate any help. Thank you very much in advance!

[dhosterman]

Hello! Thanks for your interest in ModelBench!

ModelBench uses ModelGauge's plugins, so you can write a very simple SUT plugin that should get you going with a HuggingFace model. What you should be able to do is:

1. pip install modelbench modelgauge-huggingface into your virtual environment.
2. Create a very simple plugin project using these instructions.
3. In your plugin file, you can do something like the following:

from modelgauge.secret_values import InjectSecret
from modelgauge.sut_registry import SUTS
from modelgauge.suts.huggingface_client import HuggingFaceToken, HuggingFaceSUT

SUTS.register(HuggingFaceSUT, "my-model", "my-model", InjectSecret(HuggingFaceToken))
SUTS.register(HuggingFaceSUT, "my-other-model", "my-other-model", InjectSecret(HuggingFaceToken))

4. Install your new plugin into your virtual environment using pip install -e. You'll need to make your plugin an installable package. I recommend just creating a simple pyproject.toml file in its root with name and version identifiers.
5. Run the benchmark with your new SUT(s) using something like modelbench benchmark -s openai-gpt -s my-other-model -m 10

Please let me know if that helps!

[mina-taraghi]

Thank you so much for your answer. Unfortunately I seem to not be able to make it work. Eventhough I tried to do all the steps and my package is installed successfully, when I run modelbench it doesn't recognize the SUT (Error: Invalid value for sut). I tried modelgauge list and it recognizes 22 SUTs (the 21 default and the "gpt2 HuggingFaceSUT" which I presume is the one registered at the end of the "huggingface_client.py" file).
I then tried copying the plugin into the directory modelguage/suts and it worked!
I wonder if I am doing something wrong in making my package in the modelgauge.suts namespace.
Can I ask another question? Would it be possible to run the benchmark somewhere other than together.ai (e.g. locally, amazon AWS, etc.)?

[dhosterman]

I'm glad copying the plugin into the directory worked! That's a great alternative that I should have thought to suggest to you.

As for your next question, we'd love to get together with you and get a better idea of your use cases and how we can best support them. Would you be open to meeting with the team for a bit and talking over some of what you'd like to accomplish and how we might be able to help? If so, please send me an email at daniel@mlcommons.org and I'd be happy to schedule a time to chat.