Size of Tensor A must match size of Tensor B

[rohitnanda1443]

HI,

I am trying to do RAG query on a large PDF file and get the below error:

Error: The size of tensor a (3351) must match the size of tensor b (4096) at non-singleton dimension 3.

The run script: python generate.py --base_model=mistralai/Mixtral-8x7B-Instruct-v0.1 --pre_load_embedding_model=True --score_model=None --enable_tts=False --enable_stt=False --enable_transcriptions=False --auth=auth.json --system_prompt="My name is H2O-GPT and I am an intelligent AI" --attention_sinks=True --max_new_tokens=100000 --max_max_new_tokens=100000 --top_k_docs=-1 --use_gpu_id=False --max_seq_len=4096 --sink_dict="{'num_sink_tokens': 4, 'window_length': 4096}"

[pseudotensor]

Can you provide more of the stack trace? My guess is that attention sinks in transformers is not bug free.

Separately, I recommend using Mixtral through vLLM in general. Likely it will be hard to make Mixtral run for long sequences, and it already supports 32k total input+output.

[pseudotensor]

I tried same with Mistral and didn't find any issues.

But the output and input aren't very long. The input isn't very long because you set --max_seq_len=4096, so it tries to take 29 docs but gets reduced.

[rohitnanda1443]

Noted.

I will try again with removing --max_seq_len=4096

Also where are the error log files saved in the H2O-GPT folder? (so that i can send you the stack trace) Does one get the CLI output dumped on a file using "> /home/user/dump" after the CLI startup script?

[rohitnanda1443]

I tried using Mixtral with vLLM and did the following:

1. Using this guide: https://github.com/h2oai/h2ogpt/blob/main/docs/README_InferenceServers.md Local install the Inference server

2. Ran inference server: NCCL_SHM_DISABLE=1 CUDA_VISIBLE_DEVICES=0 text-generation-launcher --model-id h2oai/h2ogpt-oig-oasst1-512-6_9b --port 8080 --sharded false --trust-remote-code --max-stop-sequences=6

3. Ran Model: python generate.py --base_model=mistralai/Mixtral-8x7B-Instruct-v0.1 --prompt_type=zephyr --max_seq_len=4096 --pre_load_embedding_model=True --score_model=None --enable_tts=False --enable_stt=False --enable_transcriptions=False --max_seq_len=4096 --auth=auth.json --inference_server="http://127.0.0.1:8080" &

[rohitnanda1443]

Issue:

Unable to connect to the inference server: After starting the inference server if I do the curl test to connect to it I get the connection refused error at the port.

The Gradio Dump:

Using Model mistralai/mixtral-8x7b-instruct-v0.1
load INSTRUCTOR_Transformer
max_seq_length 512
Starting get_model: mistralai/Mixtral-8x7B-Instruct-v0.1 http:://127.0.0.1:8080
GR Client Begin: http://http: mistralai/Mixtral-8x7B-Instruct-v0.1
GR Client Failed http://http: mistralai/Mixtral-8x7B-Instruct-v0.1: HTTPConnectionPool(host='http', port=80): Max retries exceeded with url: / (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f7357c07bb0>: Failed to resolve 'http' ([Errno -2] Name or service not known)"))
HF Client Begin: http://http: mistralai/Mixtral-8x7B-Instruct-v0.1
HF Client Failed http://http: mistralai/Mixtral-8x7B-Instruct-v0.1: Traceback (most recent call last):
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/urllib3/connectionpool.py", line 496, in _make_request
conn.request(
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/urllib3/connection.py", line 400, in request
self.endheaders()
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/urllib3/connection.py", line 238, in connect
self.sock = self._new_conn()
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/urllib3/connection.py", line 205, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPConnection object at 0x7f735dbfd000>: Failed to resolve 'http' ([Errno -2] Name or service not known)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
retries = retries.increment(
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='http', port=80): Max retries exceeded with url: / (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f735dbfd000>: Failed to resolve 'http' ([Errno -2] Name or service not known)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/h2ogpt/src/gen.py", line 2498, in get_client_from_inference_server
res = hf_client.generate('What?', max_new_tokens=1)
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/text_generation/client.py", line 275, in generate
resp = requests.post(
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/requests/api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/root/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='http', port=80): Max retries exceeded with url: / (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f735dbfd000>: Failed to resolve 'http' ([Errno -2] Name or service not known)"))

HF Client End: http://http: mistralai/Mixtral-8x7B-Instruct-v0.1 : None
Begin auto-detect HF cache text generation models
End auto-detect HF cache text generation models
Begin auto-detect llama.cpp models
End auto-detect llama.cpp models
Running on local URL: http://0.0.0.0:7863

To create a public link, set share=True in launch().
Started Gradio Server and/or GUI: server_name: localhost port: None
Use local URL: http://localhost:7863/

[pseudotensor]

If you look at the trace, you have an odd "Begin: http:://127.0.0.1:8080" with extra :. As in the docs, with vLLM one would do something like vllm:127.0.0.1:8080 or with HF client http://127.0.0.1:8080 but not extra :'s