AssertionError: Fail to convert pytorch model with 'Intel/neural-chat-7b-v3-3' WOQ

[eduand-alvarez]

from transformers import AutoTokenizer, TextStreamer
from intel_extension_for_transformers.transformers import AutoModelForCausalLM, WeightOnlyQuantConfig
model_name = "Intel/neural-chat-7b-v3-3"

# for int8, should set weight_dtype="int8"       
config = WeightOnlyQuantConfig(compute_dtype="bf16", weight_dtype="int4")
prompt = "Once upon a time, there existed a little girl,"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
inputs = tokenizer(prompt, return_tensors="pt").input_ids
streamer = TextStreamer(tokenizer)

model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=config)
outputs = model.generate(inputs, streamer=streamer, max_new_tokens=300)

results in:

cmd: ['python', PosixPath('/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py'), '--outfile', 'runtime_outs/ne_mistral_f32.bin', '--outtype', 'f32', 'Intel/neural-chat-7b-v3-3']
Loading model file Intel/neural-chat-7b-v3-3
Traceback (most recent call last):
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 1272, in
main()
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 1252, in main
model_plus = load_some_model(args.model)
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 1178, in load_some_model
models_plus.append(lazy_load_file(path))
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py", line 946, in lazy_load_file
fp = open(path, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'Intel/neural-chat-7b-v3-3'
Traceback (most recent call last):
File "/home/sdp/leaderboard/inference/serve.py", line 26, in
neural_chat_7B_v3_3_Model, neural_chat_7B_v3_3_Tokenizer = ITREXLoader('Intel/neural-chat-7b-v3-3')
File "/home/sdp/leaderboard/inference/utils/loader.py", line 16, in ITREXLoader
model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=config)
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/transformers/modeling/modeling_auto.py", line 178, in from_pretrained
model.init(
File "/home/sdp/miniconda3/envs/pbi/lib/python3.9/site-packages/intel_extension_for_transformers/llm/runtime/graph/init.py", line 122, in init
assert os.path.exists(fp32_bin), "Fail to convert pytorch model"

env:

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
accelerate 0.27.2 pypi_0 pypi
aiofiles 23.2.1 pypi_0 pypi
altair 5.2.0 pypi_0 pypi
annotated-types 0.6.0 pypi_0 pypi
anyio 4.3.0 pypi_0 pypi
attrs 23.2.0 pypi_0 pypi
ca-certificates 2023.12.12 h06a4308_0
certifi 2024.2.2 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
click 8.1.7 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
contextlib2 21.6.0 pypi_0 pypi
contourpy 1.2.0 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
deprecated 1.2.14 pypi_0 pypi
einops 0.7.0 pypi_0 pypi
exceptiongroup 1.2.0 pypi_0 pypi
fastapi 0.110.0 pypi_0 pypi
ffmpy 0.3.2 pypi_0 pypi
filelock 3.13.1 pypi_0 pypi
fonttools 4.49.0 pypi_0 pypi
fsspec 2024.2.0 pypi_0 pypi
gradio 4.20.1 pypi_0 pypi
gradio-client 0.11.0 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
httpcore 1.0.4 pypi_0 pypi
httpx 0.27.0 pypi_0 pypi
huggingface-hub 0.21.4 pypi_0 pypi
idna 3.6 pypi_0 pypi
importlib-resources 6.1.3 pypi_0 pypi
intel-extension-for-pytorch 2.1.100 pypi_0 pypi
intel-extension-for-transformers 1.2.2 pypi_0 pypi
jinja2 3.1.3 pypi_0 pypi
joblib 1.3.2 pypi_0 pypi
jsonschema 4.21.1 pypi_0 pypi
jsonschema-specifications 2023.12.1 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
matplotlib 3.8.3 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
ncurses 6.4 h6a678d5_0
networkx 3.2.1 pypi_0 pypi
neural-compressor 2.4.1 pypi_0 pypi
neural-speed 0.2 pypi_0 pypi
numpy 1.26.4 pypi_0 pypi
nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi
nvidia-cudnn-cu12 8.9.2.26 pypi_0 pypi
nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi
nvidia-curand-cu12 10.3.2.106 pypi_0 pypi
nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi
nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi
nvidia-nccl-cu12 2.18.1 pypi_0 pypi
nvidia-nvjitlink-cu12 12.4.99 pypi_0 pypi
nvidia-nvtx-cu12 12.1.105 pypi_0 pypi
opencv-python-headless 4.9.0.80 pypi_0 pypi
openssl 1.1.1w h7f8727e_0
orjson 3.9.15 pypi_0 pypi
packaging 23.2 pypi_0 pypi
pandas 2.2.1 pypi_0 pypi
pillow 10.2.0 pypi_0 pypi
pip 23.3.1 py39h06a4308_0
prettytable 3.10.0 pypi_0 pypi
psutil 5.9.8 pypi_0 pypi
py-cpuinfo 9.0.0 pypi_0 pypi
pycocotools 2.0.7 pypi_0 pypi
pydantic 2.6.3 pypi_0 pypi
pydantic-core 2.16.3 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pygments 2.17.2 pypi_0 pypi
pyparsing 3.1.2 pypi_0 pypi
python 3.9.0 hdb3f193_2
python-dateutil 2.9.0.post0 pypi_0 pypi
python-multipart 0.0.9 pypi_0 pypi
pytz 2024.1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 8.2 h5eee18b_0
referencing 0.33.0 pypi_0 pypi
regex 2023.12.25 pypi_0 pypi
requests 2.31.0 pypi_0 pypi
rich 13.7.1 pypi_0 pypi
rpds-py 0.18.0 pypi_0 pypi
ruff 0.3.1 pypi_0 pypi
safetensors 0.4.2 pypi_0 pypi
schema 0.7.5 pypi_0 pypi
scikit-learn 1.4.1.post1 pypi_0 pypi
scipy 1.12.0 pypi_0 pypi
semantic-version 2.10.0 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
setuptools 68.2.2 py39h06a4308_0
shellingham 1.5.4 pypi_0 pypi
six 1.16.0 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0
starlette 0.36.3 pypi_0 pypi
sympy 1.12 pypi_0 pypi
threadpoolctl 3.3.0 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
tokenizers 0.14.1 pypi_0 pypi
tomlkit 0.12.0 pypi_0 pypi
toolz 0.12.1 pypi_0 pypi
torch 2.1.1 pypi_0 pypi
tqdm 4.66.2 pypi_0 pypi
transformers 4.34.1 pypi_0 pypi
triton 2.1.0 pypi_0 pypi
typer 0.9.0 pypi_0 pypi
typing-extensions 4.10.0 pypi_0 pypi
tzdata 2024.1 pypi_0 pypi
urllib3 2.2.1 pypi_0 pypi
uvicorn 0.27.1 pypi_0 pypi
wcwidth 0.2.13 pypi_0 pypi
websockets 11.0.3 pypi_0 pypi
wheel 0.41.2 py39h06a4308_0
wrapt 1.16.0 pypi_0 pypi
xz 5.4.6 h5eee18b_0
zipp 3.17.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_0

Running on Baremetal dual socker 4th Gen Xeon Machine in Intel Developer Cloud

[Zhenzhong1]

@eduand-alvarez Hi, seems your ITREX version is old and incompatiable.

My ITREX version:

Please reinstall intel-extension-for-transformers and Neural Speed from the source code.

Installation methods:

git clone https://github.com/intel/neural-speed; cd neural-speed
pip install -r requirements.txt
python setup.py install

git clone https://github.com/intel/intel-extension-for-transformers; cd intel-extension-for-transformers
pip install -r requirements.txt
python setup.py install

This is my test.

[eduand-alvarez]

sure, I'll give this a try. Thank you!