qwen1half-moe-2.7B-chat采用gptq量化后infer报错

[Treedy2020]

我对qwen1half-moe-2.7B-chat使用常规lora微调后尝试了gptq的4bit量化，但在重新推理时出现了：
[rank0]: File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1709, in getattr
[rank0]: raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
[rank0]: AttributeError: 'MergedColumnParallelLinear' object has no attribute 'weight'. Did you mean: 'qweight'?

环境：Python 3.10(ubuntu22.04) Cuda 12.1

推理命令

 CUDA_VISIBLE_DEVICES=0 swift infer --model_type wen1half-moe-a2_7b-chat-int4 --ckpt_dir ./checkpoint-395-merged-gptq-int4

报错信息：

INFO 05-13 21:18:53 utils.py:660] Found nccl from library /root/.config/vllm/nccl/cu12/libnccl.so.2.18.1
INFO 05-13 21:18:53 selector.py:27] Using FlashAttention-2 backend.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/swift/cli/infer.py", line 5, in <module>
[rank0]:     infer_main()
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/swift/utils/run_utils.py", line 27, in x_main
[rank0]:     result = llm_x(args, **kwargs)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/swift/llm/infer.py", line 228, in llm_infer
[rank0]:     llm_engine, template = prepare_vllm_engine_template(args)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/swift/llm/utils/vllm_utils.py", line 375, in prepare_vllm_engine_template
[rank0]:     llm_engine = get_vllm_engine(
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/swift/llm/utils/vllm_utils.py", line 91, in get_vllm_engine
[rank0]:     llm_engine = llm_engine_cls.from_engine_args(engine_args)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 292, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 160, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 41, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 23, in _init_executor
[rank0]:     self._init_non_spec_worker()
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 69, in _init_non_spec_worker
[rank0]:     self.driver_worker.load_model()
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/worker/worker.py", line 118, in load_model
[rank0]:     self.model_runner.load_model()
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 164, in load_model
[rank0]:     self.model = get_model(
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 19, in get_model
[rank0]:     return loader.load_model(model_config=model_config,
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 222, in load_model
[rank0]:     model = _initialize_model(model_config, self.load_config,
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 88, in _initialize_model
[rank0]:     return model_class(config=model_config.hf_config,
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_moe.py", line 377, in __init__
[rank0]:     self.model = Qwen2MoeModel(config, quant_config)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_moe.py", line 341, in __init__
[rank0]:     self.layers = nn.ModuleList([
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_moe.py", line 342, in <listcomp>
[rank0]:     Qwen2MoeDecoderLayer(config, layer_idx, quant_config=quant_config)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_moe.py", line 283, in __init__
[rank0]:     self.mlp = Qwen2MoeSparseMoeBlock(config=config,
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_moe.py", line 113, in __init__
[rank0]:     self.pack_params()
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_moe.py", line 137, in pack_params
[rank0]:     w1.append(expert.gate_up_proj.weight)
[rank0]:   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1709, in __getattr__
[rank0]:     raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
[rank0]: AttributeError: 'MergedColumnParallelLinear' object has no attribute 'weight'. Did you mean: 'qweight'?

我尝试使用了pt作为推理后端，但是在模型加载后，也无法正常的进行推理：
推理命令

swift infer --model_type wen1half-moe-a2_7b-chat-int4 --ckpt_dir ./checkpoint-395-merged-gptq-int4/  --infer_backend pt

报错信息

            (up_proj): QuantLinear()
          )
          (gate): QuantLinear()
          (shared_expert_gate): QuantLinear()
        )
        (input_layernorm): Qwen2MoeRMSNorm()
        (post_attention_layernorm): Qwen2MoeRMSNorm()
      )
    )
    (norm): Qwen2MoeRMSNorm()
  )
  (lm_head): Linear(in_features=2048, out_features=151936, bias=False)
)
[INFO:swift] Qwen2MoeForCausalLM: 622.4302M Params (622.4302M Trainable [100.0000%]), 2049.3039M Buffers.
[INFO:swift] system: You are a helpful assistant.
[INFO:swift] Input `exit` or `quit` to exit the conversation.
[INFO:swift] Input `multi-line` to switch to multi-line input mode.
[INFO:swift] Input `reset-system` to reset the system and clear the history.
[INFO:swift] Input `clear` to clear the history.
<<< 你好
Exception in thread Thread-1 (generate):
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/root/miniconda3/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/generation/utils.py", line 1622, in generate
    result = self._sample(
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/generation/utils.py", line 2791, in _sample
    outputs = self(
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/qwen2_moe/modeling_qwen2_moe.py", line 1350, in forward
    outputs = self.model(
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/qwen2_moe/modeling_qwen2_moe.py", line 1219, in forward
    layer_outputs = decoder_layer(
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/qwen2_moe/modeling_qwen2_moe.py", line 929, in forward
    hidden_states = self.mlp(hidden_states)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/transformers/models/qwen2_moe/modeling_qwen2_moe.py", line 821, in forward
    router_logits = self.gate(hidden_states)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/lib/python3.10/site-packages/auto_gptq/nn_modules/qlinear/qlinear_cuda_old.py", line 348, in forward
    weight = scales * (weight - zeros)
RuntimeError: The size of tensor a (60) must match the size of tensor b (32) at non-singleton dimension 2

pip list
Package                           Version
--------------------------------- ---------------
absl-py                           2.0.0
accelerate                        0.27.2
addict                            2.4.0
aiofiles                          23.2.1
aiohttp                           3.9.5
aiosignal                         1.3.1
alibabacloud-credentials          0.3.2
alibabacloud-endpoint-util        0.0.3
alibabacloud-gateway-spi          0.0.1
alibabacloud-openapi-util         0.2.2
alibabacloud-sts20150401          1.1.4
alibabacloud-tea                  0.3.6
alibabacloud-tea-openapi          0.3.8
alibabacloud-tea-util             0.3.12
alibabacloud-tea-xml              0.0.2
alipai                            0.4.7
aliyun-python-sdk-core            2.15.1
aliyun-python-sdk-kms             2.16.2
altair                            5.3.0
annotated-types                   0.6.0
anyio                             4.2.0
appdirs                           1.4.4
argon2-cffi                       23.1.0
argon2-cffi-bindings              21.2.0
arrow                             1.3.0
asttokens                         2.4.1
async-lru                         2.0.4
async-timeout                     4.0.3
attrs                             23.2.0
auto_gptq                         0.7.1
Babel                             2.14.0
backoff                           1.11.1
beautifulsoup4                    4.12.2
bitsandbytes                      0.42.0
bleach                            6.1.0
blinker                           1.8.1
brotlipy                          0.7.0
cachetools                        5.3.2
certifi                           2022.12.7
cffi                              1.15.1
charset-normalizer                2.0.4
click                             8.1.7
cloudpickle                       3.0.0
cmake                             3.29.2
colorama                          0.4.6
coloredlogs                       15.0.1
comm                              0.2.1
conda                             22.11.1
conda-content-trust               0.1.3
conda-package-handling            1.9.0
contourpy                         1.2.0
cpm-kernels                       1.0.11
crcmod                            1.7
cryptography                      42.0.7
cycler                            0.12.1
dacite                            1.8.1
datasets                          2.18.0
debugpy                           1.8.0
decorator                         5.1.1
decord                            0.6.0
deepspeed                         0.14.0
defusedxml                        0.7.1
Deprecated                        1.2.14
diffusers                         0.25.0
dill                              0.3.8
diskcache                         5.6.3
docker-pycreds                    0.4.0
docstring_parser                  0.16
eas-prediction                    0.24
editdistance                      0.8.1
einops                            0.8.0
evaluate                          0.4.1
exceptiongroup                    1.2.0
executing                         2.0.1
fastapi                           0.110.2
fastjsonschema                    2.19.1
ffmpy                             0.3.2
filelock                          3.13.1
flash-attn                        2.5.8
fonttools                         4.47.0
fqdn                              1.5.1
frozenlist                        1.4.1
fsspec                            2023.12.2
gast                              0.5.4
gekko                             1.0.6
gitdb                             4.0.11
GitPython                         3.1.43
google-auth                       2.26.1
google-auth-oauthlib              1.2.0
gradio                            4.28.3
gradio_client                     0.16.0
grpcio                            1.60.0
h11                               0.14.0
hjson                             3.1.0
httpcore                          1.0.5
httptools                         0.6.1
httpx                             0.27.0
huggingface-hub                   0.23.0
humanfriendly                     10.0
idna                              3.4
importlib_metadata                7.1.0
importlib_resources               6.4.0
iniconfig                         2.0.0
interegular                       0.3.3
ipykernel                         6.28.0
ipython                           8.20.0
ipywidgets                        8.1.1
isoduration                       20.11.0
jedi                              0.19.1
jieba                             0.42.1
Jinja2                            3.1.2
jmespath                          0.10.0
joblib                            1.4.0
json5                             0.9.14
jsonlines                         4.0.0
jsonpointer                       2.4
jsonschema                        4.20.0
jsonschema-specifications         2023.12.1
jupyter_client                    8.6.0
jupyter_core                      5.7.1
jupyter-events                    0.9.0
jupyter-lsp                       2.2.1
jupyter_server                    2.12.2
jupyter_server_terminals          0.5.1
jupyterlab                        4.0.10
jupyterlab-language-pack-zh-CN    4.0.post6
jupyterlab_pygments               0.3.0
jupyterlab_server                 2.25.2
jupyterlab-widgets                3.0.9
kiwisolver                        1.4.5
lark                              1.1.9
llmuses                           0.3.0
llvmlite                          0.42.0
lm-format-enforcer                0.9.8
lxml                              5.2.1
Markdown                          3.5.1
markdown-it-py                    3.0.0
MarkupSafe                        2.1.3
marshmallow                       3.21.2
marshmallow-oneofschema           3.1.1
matplotlib                        3.8.2
matplotlib-inline                 0.1.6
mdurl                             0.1.2
mistune                           3.0.2
modelscope                        1.14.0
mpmath                            1.3.0
ms-swift                          2.0.4
msgpack                           1.0.8
multidict                         6.0.5
multiprocess                      0.70.16
nbclient                          0.9.0
nbconvert                         7.14.0
nbformat                          5.9.2
nest-asyncio                      1.5.8
networkx                          3.2.1
ninja                             1.11.1.1
nltk                              3.8.1
notebook_shim                     0.2.3
numba                             0.59.1
numpy                             1.26.4
nvidia-cublas-cu12                12.1.3.1
nvidia-cuda-cupti-cu12            12.1.105
nvidia-cuda-nvrtc-cu12            12.1.105
nvidia-cuda-runtime-cu12          12.1.105
nvidia-cudnn-cu12                 8.9.2.26
nvidia-cufft-cu12                 11.0.2.54
nvidia-curand-cu12                10.3.2.106
nvidia-cusolver-cu12              11.4.5.107
nvidia-cusparse-cu12              12.1.0.106
nvidia-ml-py                      12.550.52
nvidia-nccl-cu12                  2.20.5
nvidia-nvjitlink-cu12             12.4.127
nvidia-nvtx-cu12                  12.1.105
oauthlib                          3.2.2
openai                            0.28.0
optimum                           1.19.1
orjson                            3.10.1
oss2                              2.18.5
outlines                          0.0.34
overrides                         7.4.0
packaging                         23.2
pandas                            2.2.2
pandocfilters                     1.5.0
parso                             0.8.3
peft                              0.10.0
pexpect                           4.9.0
pillow                            10.2.0
pip                               22.3.1
platformdirs                      4.1.0
plotly                            5.22.0
pluggy                            1.5.0
ply                               3.11
portalocker                       2.8.2
prometheus-client                 0.19.0
prometheus-fastapi-instrumentator 7.0.0
prompt-toolkit                    3.0.43
protobuf                          3.20.3
psutil                            5.9.7
ptyprocess                        0.7.0
pure-eval                         0.2.2
py-cpuinfo                        9.0.0
pyarrow                           16.0.0
pyarrow-hotfix                    0.6
pyasn1                            0.5.1
pyasn1-modules                    0.3.0
pycosat                           0.6.4
pycparser                         2.21
pycryptodome                      3.20.0
pydantic                          2.7.1
pydantic_core                     2.18.2
pydeck                            0.9.0
pydub                             0.25.1
Pygments                          2.17.2
Pympler                           1.0.1
pynvml                            11.5.0
pyodps                            0.11.6.1
pyOpenSSL                         24.1.0
pyparsing                         3.1.1
PySocks                           1.7.1
pytest                            8.2.0
python-dateutil                   2.8.2
python-dotenv                     1.0.1
python-json-logger                2.0.7
python-multipart                  0.0.9
pytz                              2024.1
PyYAML                            6.0.1
pyzmq                             25.1.2
ray                               2.20.0
referencing                       0.32.1
regex                             2024.4.28
requests                          2.31.0
requests-oauthlib                 1.3.1
requests-toolbelt                 1.0.0
responses                         0.18.0
rfc3339-validator                 0.1.4
rfc3986-validator                 0.1.1
rich                              13.7.1
rouge                             1.0.1
rouge-chinese                     1.0.3
rouge-score                       0.1.2
rpds-py                           0.16.2
rsa                               4.9
ruamel.yaml                       0.17.21
ruamel.yaml.clib                  0.2.6
ruff                              0.4.2
sacrebleu                         2.4.2
safetensors                       0.4.3
scikit-learn                      1.4.2
scipy                             1.13.0
seaborn                           0.13.2
semantic-version                  2.10.0
Send2Trash                        1.8.2
sentencepiece                     0.2.0
sentry-sdk                        2.0.1
setproctitle                      1.3.3
setuptools                        65.5.0
shellingham                       1.5.4
shtab                             1.7.1
simple-ddl-parser                 1.1.0
simplejson                        3.19.2
six                               1.16.0
smmap                             5.0.1
sniffio                           1.3.0
sortedcontainers                  2.4.0
soupsieve                         2.5
stack-data                        0.6.3
starlette                         0.37.2
streamlit                         1.34.0
supervisor                        4.2.5
sympy                             1.12
tabulate                          0.9.0
tenacity                          8.2.3
tensorboard                       2.15.1
tensorboard-data-server           0.7.2
terminado                         0.18.0
threadpoolctl                     3.5.0
tiktoken                          0.6.0
tinycss2                          1.2.1
tokenizers                        0.19.1
toml                              0.10.2
tomli                             2.0.1
tomlkit                           0.12.0
toolz                             0.12.0
torch                             2.3.0
torchaudio                        2.1.2
torchdata                         0.7.1
torchvision                       0.16.2+cu121
tornado                           6.4
tqdm                              4.64.1
traitlets                         5.14.1
transformers                      4.40.1
transformers-stream-generator     0.0.5
triton                            2.3.0
trl                               0.8.6
typer                             0.12.3
types-python-dateutil             2.8.19.20240106
typing_extensions                 4.9.0
tyro                              0.8.3
tzdata                            2024.1
uri-template                      1.3.0
urllib3                           2.2.1
uvicorn                           0.29.0
uvloop                            0.19.0
vllm                              0.4.2
vllm-nccl-cu12                    2.18.1.0.4.0
wandb                             0.16.6
watchdog                          4.0.0
watchfiles                        0.21.0
wcwidth                           0.2.13
webcolors                         1.13
webencodings                      0.5.1
websocket-client                  1.7.0
websockets                        11.0.3
Werkzeug                          3.0.1
wheel                             0.37.1
widgetsnbextension                4.0.9
wrapt                             1.16.0
xformers                          0.0.26.post1
xxhash                            3.4.1
yapf                              0.40.2
yarl                              1.9.4
zipp                              3.18.1