Weight int4 quantization, but actually it is int16

[dongxuemin666]

Hi I used weight int4, but when I run inference, finding that weight is actually int16, is my pipeline wrong

[dongxuemin666]

image seems to be broken, please see this one

[dongxuemin666]

below is my script to do quant

python -m awq.entry --model_path $MODEL
--w_bit 4 --q_group_size 128
--run_awq --dump_awq awq/llava_w4/llava-v1.6-vicuna-7b-w4-g128.pt

python -m awq.entry --model_path $MODEL
--w_bit 4 --q_group_size 128
--load_awq awq/llava_w4/llava-v1.6-vicuna-7b-w4-g128.pt
--q_backend real --dump_quant awq/llava_w4/llava-v1.6-vicuna-7b-w4-g128-awq.pt

[dongxuemin666]

I get this, weight is fake int4, in calculation, actually is int16

[ponytaill]

I get this, weight is fake int4, in calculation, actually is int16

If it's convenient for you, could you explain it?