RuntimeError: could not create a primitive descriptor for a matmul primitive

[buvanesh-git]

🐛 Describe the bug

Runtime Error noticed while running llama2 on aarch64 Grace super server, onednn built on the native system from https://github.com/oneapi-src/oneDNN/blob/main/README.md and ACL from https://github.com/ARM-software/ComputeLibrary

tried to run with different torch version (Torch 2.0.0 Torch 2.1.0, torch 2.2.0 torch 2.2.2 and torch 2.3.0 ) but same Runtime error is repeating. here is script attached used to run, modules are from HuggingFace
[
llama2_perf.zip
](url)

Error message :
return forward_call(*args, **kwargs)
File "/home/amd/miniforge3/envs/llama_env/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: could not create a primitive descriptor for a matmul primitive

the reproducer program from https://discuss.pytorch.org/t/why-cant-i-have-out-features-1-in-a-linear-layer/191593 works with torch 2.2.2 and above. but not helping to run attached llama2_perf.zip

t = torch.tensor([
[1.0, 2.0],
[3.0, 4.0],
[5.0, 6.0],
])
lin = torch.nn.Linear(2, 1)
lin(t)

Versions

Collecting environment information...
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (aarch64)
GCC version: (conda-forge gcc 12.3.0-5) 12.3.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 21:44:20) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-105-generic-aarch64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 144
On-line CPU(s) list: 0-143
Vendor ID: ARM
Model name: Neoverse-V2
Model: 0
Thread(s) per core: 1
Core(s) per socket: 72
Socket(s): 2
Stepping: r0p0
Frequency boost: disabled
CPU max MHz: 3438.0000
CPU min MHz: 81.0000
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti
L1d cache: 9 MiB (144 instances)
L1i cache: 9 MiB (144 instances)
L2 cache: 144 MiB (144 instances)
L3 cache: 228 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-71
NUMA node1 CPU(s): 72-143
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.1
[pip3] torchvision==0.15.1
[pip3] tpp-pytorch-extension==0.0.1
[conda] numpy 1.26.4 pypi_0 pypi
[conda] torch 2.0.0 pypi_0 pypi
[conda] torchaudio 2.0.1 pypi_0 pypi
[conda] torchvision 0.15.1 pypi_0 pypi
[conda] tpp-pytorch-extension 0.0.1 pypi_0 pypi

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10

[marktansoprano]

I have the same problem on aarch64, when doing basic inference on this model https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0

site-packages/torch/nn/modules/linear.py", line 116, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: could not create a primitive descriptor for a matmul primitive

regardless of torch version, 2.0.1, 2.2.2, 2.3.0, all the same.

The above happens when using bfloat16

pipe = pipeline("text-generation", model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", torch_dtype=torch.bfloat16, device_map="auto")

when change to use float16, no error & works. But takes up enormous memory & inference takes forever

pipe = pipeline("text-generation", model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", torch_dtype=torch.float16, device_map="auto")

[WilliamTambellini]

likely an exception coming from intel oneDNN:
https://oneapi-src.github.io/oneDNN/v1/dev_guide_matmul.html

[jgong5]

cc @milpuz01 for the oneDNN integration issue related to ARM.