How to use deepseek

[huanhuan6666]

When I modified the configuration file to use the model="deepseek-coder-33b-instruct" and ran the code for model inference, it failed with the following error:

alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ...

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

ERROR:root:Error during OpenAI inference

It seems that litellm does not support deepseek, and I would like to know how to resolve this issue.

[huanhuan6666]

More error info:

Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details LiteLLM-Failure Call
ERROR:root:Error during OpenAI inference
WARNING:root:Failed to generate prediction with deepseek-coder-33b-instruct: Traceback (most recent call last):
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
    yield
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_transports/default.py", line 371, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 215, in handle_async_request
    raise UnsupportedProtocol(
httpcore.UnsupportedProtocol: Request URL is missing an 'http://' or 'https://' protocol.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/litellm/llms/huggingface_restapi.py", line 569, in acompletion
    response = await client.post(
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py", line 1877, in post
    return await self.request(
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py", line 1559, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py", line 1646, in send
    response = await self._send_handling_auth(
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py", line 1674, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py", line 1711, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py", line 1748, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_transports/default.py", line 370, in handle_async_request
    with map_httpcore_exceptions():
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.UnsupportedProtocol: Request URL is missing an 'http://' or 'https://' protocol.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/litellm/main.py", line 273, in acompletion
    response = await init_response
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/litellm/llms/huggingface_restapi.py", line 600, in acompletion
    raise HuggingfaceError(status_code=500, message=f"{str(e)}")
litellm.llms.huggingface_restapi.HuggingfaceError: Request URL is missing an 'http://' or 'https://' protocol.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/hzhang/AlphaCodium/alpha_codium/llm/ai_invoker.py", line 15, in send_inference
    return await f(model)
  File "/data/hzhang/AlphaCodium/alpha_codium/gen/coding_competitor.py", line 49, in _run
    response, finish_reason = await self.ai_handler.chat_completion(
  File "/data/hzhang/AlphaCodium/alpha_codium/llm/ai_handler.py", line 92, in chat_completion
    response = await acompletion(
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/litellm/utils.py", line 2868, in wrapper_async
    raise e
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/litellm/utils.py", line 2717, in wrapper_async
    result = await original_function(*args, **kwargs)
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/litellm/main.py", line 286, in acompletion
    raise exception_type(
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/litellm/utils.py", line 7462, in exception_type
    raise e
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/litellm/utils.py", line 6919, in exception_type
    raise APIError(
litellm.exceptions.APIError: HuggingfaceException - Request URL is missing an 'http://' or 'https://' protocol.

ERROR:root:'run_self_reflect' stage, counter_retry 0, Error: HuggingfaceException - Request URL is missing an 'http://' or 'https://' protocol.

[EdgedSquirrels]

Hi,

I also encountered the same problem in Python3.9. Since the error message is about request URL, I tried to comment out api_base in alpha_codium/llm/ai_handler.py.

response = await acompletion(
    model="huggingface/deepseek-ai/deepseek-coder-33b-instruct",
    messages=[
        {"role": "system", "content": system},
        {"role": "user", "content": user},
    ],
    # api_base=get_settings().get("config.model"),
    temperature=temperature,
    repetition_penalty=frequency_penalty+1, # the scale of TGI is different from OpenAI
    force_timeout=get_settings().config.ai_timeout,
    max_tokens=2000,
    stop=['<|EOT|>'],
)

After the modification, it seems that litellm can connect to deepseek now, but it complains that the model size is too big:

LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

ERROR:root:Error during OpenAI inference
WARNING:root:Failed to generate prediction with huggingface/deepseek-ai/deepseek-coder-33b-instruct: Traceback (most recent call last):
  File "/data1/b09902053/miniconda3/envs/alphacodium/lib/python3.9/site-packages/litellm/main.py", line 284, in acompletion
    response = await init_response
  File "/data1/b09902053/miniconda3/envs/alphacodium/lib/python3.9/site-packages/litellm/llms/huggingface_restapi.py", line 607, in acompletion
    raise e
  File "/data1/b09902053/miniconda3/envs/alphacodium/lib/python3.9/site-packages/litellm/llms/huggingface_restapi.py", line 579, in acompletion
    raise HuggingfaceError(
litellm.llms.huggingface_restapi.HuggingfaceError: The model deepseek-ai/deepseek-coder-33b-instruct is too large to be loaded automatically (66GB > 10GB). Please use Spaces (https://huggingface.co/spaces) or Inference Endpoints (https://huggingface.co/inference-endpoints).

During handling of the above exception, another exception occurred:

...

I would like to know how to resolve the issue, too.

[liuqi8827]

I met the same error : #34 (comment) @huanhuan6666