Intel ARC Support

[h3x4g0ns]

Thoughts? There's also a successful PyTorch extension. I know this hardware is in its infancy, but the performance to price ratio is insane. Not to mention that with added driver support in the future, the performance gains are only going to be better...

[geohot]

Just grabbed a A770 16GB off eBay. They are so cheap, an issue is the cost per PCI-E slot for the computer is higher than the card! So the card itself might have a decent TFLOPS/$, but what do you plug in into?

[naturalkind]

Good afternoon, what are your impressions of the a770? Do you use it in your activities? P.S. I am waiting for the launch of your project: "Transferring a copy of consciousness into the matrix", I am improving my skills to participate, with your activity you inspired me to study programming, thank you!

[fredlarochelle]

It does seems to run somewhat, albeit slowly, using the GPU runtime. Tried adding IPEX to the TORCH runtime, but I am running into some tests failures, need to take a deeper look. Might be as simple as adding the following after device = torch.device... in ops_torch.py, *if I find what is causing some tests to fail:

# There is definitely a cleaner way to do it
try:
  import intel_extension_for_pytorch as ipex
  device = torch.device("xpu:0" if torch.xpu.is_available() else device)
except ImportError:
  print("IPEX is not installed.")

The main benefits of the A770 are the XMX matrix operations with their 8:1 throughput for FP16/BF16 (157.28 TFLOPS theorical performance at base 2.1 GHz, mine is boosting to 2.4GHz all day without even overclocking it, soo probably closer to 180 TFLOPS) and the good memory bandwith for the price, 560 GBps (definitely better than recently released rtx40xx series cards...). The FP32 performance is okay, not amazing, not bad either.

Right now, IPEX is more optimized towards BF16 than FP16 tho...

[fredlarochelle]

I haven't looked that deep into the whole thing, but from my understanding most of it is open-source.

Starting with Linux 6.2, Arc is supported in the Linux kernel. Over that, they have their oneAPI Level Zero, than their compute runtime, over that there is SYCL, then all the rest of oneAPI (oneMKL, oneDNN, ...) and finally IPEX, ITEX, ... All of oneAPI is an open spec, but for most of it, Intel has the only implementation available (which they open source).

Can't find a doc with the instructions set tho, definitely possible to ask Intel in one of their multiple repo on github, they are really active.

[ColonelPhantom]

Here's what I know about DG2/Alchemist:

• The hardware is documented at https://www.intel.com/content/www/us/en/docs/graphics-for-linux/developer-reference/1-0/alchemist-arctic-sound-m.html
• The compute stack is largely FOSS, although some libraries (notably MKL) are proprietary to my knowledge.
• The stack is to my knowledge split between the NEO runtime https://github.com/intel/compute-runtime and the compiler https://github.com/intel/llvm
• The XMX hardware is exposed to OpenCL via two custom extensions: https://registry.khronos.org/OpenCL/extensions/intel/cl_intel_subgroup_matrix_multiply_accumulate.html and https://registry.khronos.org/OpenCL/extensions/intel/cl_intel_subgroup_split_matrix_multiply_accumulate.html which might be nicer than doing it over SYCL given that Tinygrad already supports OpenCL? In SYCL, there is the joint_matrix extension iirc.

[martin19]

Tried llama.py with the patch applied. My machine has 32gb of RAM and 128GB swap.
Out of memory occurs.

TORCH=1 python3 llama.py --model=/home/martin/projects/ml/models/LLaMA/7B --size="7B" --temperature=0 --count=50 --prompt="Hello."
using TORCH backend
using LLaMA-7B model
ram used: 13.48 GB, freqs_cis                                         : 100%|█████████████████████████████████████████████████████████████████████████████████████████| 292/292 [00:05<00:00, 55.09it/s]
loaded weights in 5302.90 ms, 13.48 GB loaded at 2.54 GB/s
...
  File "/home/martin/.local/lib/python3.10/site-packages/torch/functional.py", line 378, in einsum
    return _VF.einsum(equation, operands)  # type: ignore[attr-defined]
RuntimeError: Native API failed. Native API returns: -6 (PI_ERROR_OUT_OF_HOST_MEMORY) -6 (PI_ERROR_OUT_OF_HOST_MEMORY)

If i use the "quantize" paremeter it seems to generate a few tokens then an out of memory error occurs.

TORCH=1 python3 llama.py --model=/home/martin/projects/ml/models/LLaMA/7B --size="7B" --temperature=0 --count=50 --prompt="Hello." --quantize
using TORCH backend
using LLaMA-7B model
ram used:  6.87 GB, freqs_cis                                         : 100%|██████████████████████████████████████████████████████████████████████████████████████| 517/517 [00:00<00:00, 23786.57it/s]
loaded weights in 23.49 ms, 0.00 GB loaded at 0.00 GB/s
Hello. I'
...
  File "/home/martin/projects/tinygrad/tinygrad/tinygrad/runtime/ops_torch.py", line 53, in copyin
    def copyin(self, dest:torch.Tensor, src:memoryview): dest.copy_(torch.frombuffer(src, dtype=dest.dtype))
RuntimeError: Native API failed. Native API returns: -6 (PI_ERROR_OUT_OF_HOST_MEMORY) -6 (PI_ERROR_OUT_OF_HOST_MEMORY)

These error does not seem like specific to ARC and I assume they might appear on any torch backend? Don't understand this enough yet - i think einsum_mulacc will always run on cpu because it does not use torch now... will look further into this.

Anybody running llama using the torch backend on ARC770 or any torch based backend?

[fredlarochelle]

Haven't looked into it, but it might be related to the 32bit memory counters on Alchemist hardware. High -level overview of the situation is that you can't allocate the whole 16GB of memory as a single block, you are limited to 4GB blocks, but you can allocate multiple blocks.

[martin19]

Didn't get pytorch backend to work - I think it is a version mismatch btw. IPEX and pytorch. Versions from IPEX github page won't work for me. OpenCL backend works out of the box for llama on ARC770. Inference is quite slow.

time GPU=1 python3 llama.py --model=/home/martin/projects/ml/models/LLaMA/7B --size="7B" --temperature=0 --count=50 --prompt="Hello." using GPU backend using LLaMA-7B model ram used: 13.48 GB, freqs_cis : 100%|█████████████████████████████████████████████████████████████████████████████████████████| 292/292 [00:03<00:00, 82.47it/s] loaded weights in 3542.99 ms, 13.48 GB loaded at 3.80 GB/s Hello. I'm a 20 year old male. I'm a student at the University of Texas at Austin. I'm a sophomore majoring in Computer Science. I'm a member of the UT Austin chapter of real 0m35,414s user 0m2,245s sys 0m27,076s

[stalkerg]

You can use OpenCL by Rusticl

[ColonelPhantom]

Why would you use Rusticl over Intel NEO? Unlike RustiCL, NEO also supports a whole bunch of Intel vendor-specific extensions, including the one necessary to use XMX matrix acceleration.

[stalkerg]

because it's well integrated with mesa and easy to use

[ColonelPhantom]

Don't most distros ship intel neo too? I don't really see how rusticl is much if any easier

[martin19]

Hi,
I have one ARC770 16GB here to experiment with. I'd like to run the latest mistral model on it via tinygrad. What needs to be done to implement first-class support in tinygrad? is the torch.device way described above as good as it gets?
I'm also interested in quantization to run larger models according to this link hardware support should be there https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-the-xe-hpg-architecture.html.

Got the benchmark joint_matrix_bfloat16.cpp running on latest oneAPI ("2024.0") , according to the logs it's running on ARC. Cannot tell if this is fast or slow compared to other platforms.

SYCL_PI_TRACE=1 ./a.out
M2048N2048K4096
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_opencl.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_level_zero.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_unified_runtime.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[all]: Requested device_type: info::device_type::automatic
SYCL_PI_TRACE[all]: Requested device_type: info::device_type::automatic
SYCL_PI_TRACE[all]: Requested device_type: info::device_type::automatic
SYCL_PI_TRACE[all]: Requested device_type: info::device_type::automatic
SYCL_PI_TRACE[all]: Requested device_type: info::device_type::automatic
SYCL_PI_TRACE[all]: Requested device_type: info::device_type::automatic
SYCL_PI_TRACE[all]: Selected device: -> final score = 1550
SYCL_PI_TRACE[all]:   platform: Intel(R) Level-Zero
SYCL_PI_TRACE[all]:   device: Intel(R) Arc(TM) A770 Graphics
submit:  304 ms
compute: 53 ms
Elapsed time in milliseconds (reference): 338817 ms
failed

haven't found out yet why the accuracy check fails.

[martin19]

Been reading on "small type" support on intel hardware. One pointer: OpenVino has implemented experimental int4 in their last release -however I dont know if that works on Alchemist. I think int4 is only used as sort of compression for weight storage in gpu memory while computations run using floats converting them on the fly. Where in the software stack does this actually happen?

[fredlarochelle]

@martin19 Alchemist hardware also supports INT2 lol.

From what I have seen, pretty much all the functionalities are there in SYCL, they just haven't been implemented in the upper layers of the stack. SYCL is akin to CUDA on an Nvidia card, pretty much everything goes through that. Now, I haven't looked much into SYCL, but a couple months back, computations using INT2, INT4, INT8 was working in Fortran through IFX.

[ColonelPhantom]

I don't think SYCL is the bottom of the stack on Intel. Sure, a lot of things (probably including IPEX, oneDNN, and probably openVINO?) are built on it, but SYCL is 'just' the host language (like CUDA C++ is).

In fact, I'm pretty sure dpcpp/icpx can run SYCL on top of OpenCL! Although usually they seem to prefer to run on top of Level Zero instead. So L0/OpenCL are what the CUDA runtime is, and then instead of NVPTX, Intel uses mostly industry-standard SPIR-V (possibly with custom extensions, I don't know).

[martin19]

things get dark after spir-v and L0. I see https://mlir.llvm.org/doxygen/JointMatrixOps_8cpp_source.html
anyone know which ops in tinygrad can use mentioned opencl extensions and if the abstraction level in fits these ops? probably use them in the opencl renderer.

[ColonelPhantom]

If I were implementing XMX support in Tinygrad, I'd probably use one of the cl_intel_subgroup_matrix_multiply_accumulate or cl_intel_subgroup_split_matrix_multiply_accumulate extension.

I'd generate this using similar codepaths as what is used for RDNA3 WMMA and Apple Tensor Core.