New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MPS support #40

Open

tmm1 opened this issue Jun 28, 2023 · 1 comment

tmm1 commented Jun 28, 2023

It would be nice to support Metal Performance Shaders backend for pytorch on macOS

the main changes for inference seem to be:

load_in_8bit must be False
device_map set to {"":"mps"}

see:

https://github.com/Birch-san/falcon-play/blob/cbf9b2aebe7eef3eea305a511d6cdda17282ca8a/scripts/chat_play.py#L154-L157

https://github.com/Birch-san/falcon-play/blob/cbf9b2aebe7eef3eea305a511d6cdda17282ca8a/scripts/chat_play.py#L175

cc @Birch-san are there any other good docs or resources for adding mps support?

The text was updated successfully, but these errors were encountered:

Contributor

bdqnghi commented Jul 4, 2023

this is great, can you help to propose any pull request that i can merge?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment