-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use safetensors
to store tensors instead of pickle
#32
Comments
Hi @alvarobartt, thank you for reaching out! I'm glad the project was of use to you :)
|
Cool @matthias-wright! I'm also busy these days (also some bank holidays in Spain next week) but I'll try to submit a PR in case you're interested, your project is indeed the most similar to I'll keep you updated anyway! |
Also regarding how "complicated" that could be in terms of code, I'd say that it's just unpickling the weights, converting to |
Hi @matthias-wright, I've been playing around for a couple days with your project and it's so cool, thanks for building some pure
flax
models here 馃憤馃徎Don't know if you're aware, but @huggingface developed a new format for storing tensors named
safetensors
as most of the serialized models from PyTorch usepickle
to store the tensors, which seems to be not super efficient plus it has some known security issues. So I want to know whether you're considering to port the current tensors to usesafetensors
instead.I've recently built
safejax
so as to easily do that, which means that the storage is optimal and more safe! If this is something you could consider to improveflaxmodels
please let me know and I can try to help if applicable!P.S. Did you consider publishing the Python package to PyPI tracking it through GitHub Release so that it attracts more users due to the ease of installation through
pip
from PyPI instead of from source as in theREADME.md
?The text was updated successfully, but these errors were encountered: