ROCm on 0.9.0-rc.5

[rudiservo]

Hey, just wanted to leave some feedback.

I have been testing StarCoder2-7b and 15b Q8_0 on a 7900XTX
RC4 was extremely slow responding and usually stalled on both models.
RC5 is much faster and more responsive, it's actually usable, still if it could be a bit faster it would be great.

I am using ROCm 6.0 on ubuntu 22.04
7950X
7900XTX

StarCoder2-15b-Q8_0 is using 18023MiB

hopefully you can squeeze more performance out of it.
Thanks.

[wsxiaoys]

Thanks for testing out! A bug fix related to ouput token length has been cherrypicked to 0.9 release, thus improving the latency.

#1626

[rudiservo]

yea has you can see on the GPU usage graph it really fixed it.
Before it did 100% GPU for about 5 to 10 sec and cut off without any response.
This was in PHP, HTML, Javascript and Ansible, do not know if that matters.

There are big improvements in ROCm 6.0, but it's not compatible with older AMD cards.

Thanks for the bugfix.