HuggingFaceHub fails with "Error raised by inference API: Cannot override task for LLM models" #772
Comments
|
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗 |
|
Met the same problem. And I think I'm already get access to this model through "https://huggingface.co/bigcode/starcoder" |
|
I have the same problem with different models as well (including mistralai/Mistral-7B-Instruct-v0.2 and meta-llama/Meta-Llama-3-8B) |
1 similar comment
|
I have the same problem with different models as well (including mistralai/Mistral-7B-Instruct-v0.2 and meta-llama/Meta-Llama-3-8B) |
|
I can solve it by updating the libraries:langchain and huggingface-hub |
|
according to https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2/discussions/115, changing huggingface_hub.interface_api file on 152 line i replaced api_url as as self.api_url = f"{INFERENCE_ENDPOINT}/models/{repo_id}" But it seems that the length of answer is limitted. I don't know why :( |
|
Thank you all for reporting this issue! We just merged a PR to fix this. It will be included in the next release, tentatively scheduled for early next week. Note: this PR will not be backported to 1.x as JupyterLab 3 reached end-of-maintenance yesterday. See #761. |
Description
Hugging Face Hub Integration does not work. I can access HF Inference API from curl, but jupyter-ai chat fails with the
ValueError: Error raised by inference API: Cannot override task for LLM models(see Context section for the full traceback). I've tried different open models: mistralai/Mistral-7B-Instruct-v0.2, bigcode/starcoder2-3b. None of them works.I guess, the reason is that you set
taskargument explicitly, while langchain does not set it. Moreover, Langchain uses InferenceClient instead of deprecated InferenceApi.Anyway, the best solution, in my opinion, is to replace langchain's HuggingFaceHub with HuggingFaceEndpoint, because the first is deprecated in langchain.
Reproduce
Expected behavior
Any answer from the model served on HF Inference API
Context
Traceback From Chat
Traceback (most recent call last): File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/jupyter_ai/chat_handlers/base.py", line 125, in on_message await self.process_message(message) File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/jupyter_ai/chat_handlers/default.py", line 61, in process_message response = await self.llm_chain.apredict(input=message.body, stop=["\nHuman:"]) File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain/chains/llm.py", line 333, in apredict return (await self.acall(kwargs, callbacks=callbacks))[self.output_key] File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain_core/_api/deprecation.py", line 157, in awarning_emitting_wrapper return await wrapped(*args, **kwargs) File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain/chains/base.py", line 428, in acall return await self.ainvoke( File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain/chains/base.py", line 212, in ainvoke raise e File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain/chains/base.py", line 203, in ainvoke await self._acall(inputs, run_manager=run_manager) File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain/chains/llm.py", line 298, in _acall response = await self.agenerate([inputs], run_manager=run_manager) File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain/chains/llm.py", line 165, in agenerate return await self.llm.agenerate_prompt( File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 643, in agenerate_prompt return await self.agenerate( File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 1018, in agenerate output = await self._agenerate_helper( File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 882, in _agenerate_helper raise e File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 866, in _agenerate_helper await self._agenerate( File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 1338, in _agenerate else await self._acall(prompt, stop=stop, **kwargs) File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/jupyter_ai_magics/providers.py", line 684, in _acall return await self._call_in_executor(*args, **kwargs) File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/jupyter_ai_magics/providers.py", line 322, in _call_in_executor return await loop.run_in_executor(executor, _call_with_args) File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/home/user/.pyenv/versions/3.10.5/lib/python3.10/site-packages/jupyter_ai_magics/providers.py", line 647, in _call raise ValueError(f"Error raised by inference API: {response['error']}") ValueError: Error raised by inference API: Cannot override task for LLM modelsTroubleshoot Output
Command Line Output
The text was updated successfully, but these errors were encountered: