Experimental AMDGPU support

[mfherbst]

Should work in theory, but so far entirely untested.

OK before this works we need at least these things in Yggdrasil and properly exposed (e.g. via AbstractFFT):

• [New package] Add rocFFT 4.2.0 JuliaPackaging/Yggdrasil#4256

[pxl-th]

FFT seems to be working fine.
Segfalts come from missing rocSOLVER qr used in ortho_qr.
I'm going to add it in AMDGPU.

[pxl-th]

Now amgpu_simple.jl fails with:

ERROR: LoadError: Scalar indexing is disallowed.
Invocation of getindex resulted in scalar indexing of a GPU array.
This is typically caused by calling an iterating implementation of a method.
Such implementations *do not* execute on the GPU, but very slowly on the CPU,
and therefore are only permitted from the REPL for prototyping purposes.
If you did intend to index this array, annotate the caller with @allowscalar.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] assertscalar(op::String)
   @ GPUArraysCore ~/.julia/packages/GPUArraysCore/uOYfN/src/GPUArraysCore.jl:103
 [3] getindex
   @ ~/.julia/packages/GPUArrays/5XhED/src/host/indexing.jl:9 [inlined]
 [4] mul!(Hψ::ROCMatrix{ComplexF64, AMDGPU.Runtime.Mem.HIPBuffer}, H::Hamiltonian, ψ::ROCMatrix{ComplexF64, AMDGPU.Runtime.Mem.HIPBuffer})
   @ DFTK ~/code/DFTK.jl/src/terms/Hamiltonian.jl:83
 [5] *(H::Hamiltonian, ψ::ROCMatrix{ComplexF64, AMDGPU.Runtime.Mem.HIPBuffer})
   @ DFTK ~/code/DFTK.jl/src/terms/Hamiltonian.jl:88
 [6] top-level scope
   @ ~/code/DFTK.jl/examples/amdgpu_simple.jl:23
in expression starting at /home/pxl-th/code/DFTK.jl/examples/amdgpu_simple.jl:23

[pxl-th]

Same issue is with CUDA though

[mfherbst]

Hi @pxl-th Thanks for your help. Yes, that probably should not happen. I'll take a look.

[mfherbst]

I tried to use the code with AMDGPU.jl master, but I get some issues, see JuliaGPU/AMDGPU.jl#461.