Do you want to grow this project?

[shroominic]

Idk what your plan is with this project so I just wanted to ask if you want to grow it and advance into enabling

• more available models (7B, 13B), CodeLLama, ...
• support quantized models
• improved abstractions over multiple files
• gpu support
• documentation
• jsonformer or guidance on top of this
• langchain or openai api integration

We could create different TODO Issues for the featues to enable work by the community.
If you dont want to grow it maybe we could create a community fork building on top of it.
I really like the idea of doing inference in mojo so really greatful for this project and I think this could be a good opportunity to learn more about mojo by building some features :)

[tairov]

Hi @shroominic , thanks for reaching out! I really appreciate you taking the time to share your thoughts on how this project could grow.

You've raised some great ideas, part of them already are in the air in discussions on platforms (github, discord, twitter)
Those are all awesome suggestions that would really help take things to the next level.
I'd also add one major item to your todo list, it seems that the Mistral model support is quite an interesting project to implement.
I'm definitely open to collaborating with others to build it out into something more "full-featured".
Since Mojo is still quite a new toy, I think consolidating "llama, llm" related efforts in one place makes sense for now.
Also, since my bandwidth is quite limited I'm happy to add you or anyone else as collaborators. Major criteria from my side is that people should have strong desire and coincide capabilities to contribute on this axis.

Another aspect that community could benefit of - is the current momentum. Llama2.🔥 got a great traction over social networks and it was featured few times in the official Modular blog posts. This must help us pull more attention, strengthen our results, leverage progress we had so far. We can really accelerate things.

Let me know if you'd be interested in driving any particular features from your list ( or if you have strong desire & capabilities to contribute in todos management 😄 ). I'm excited to see where we can take this together!

PS. If you're ok, for now I'll add a todo list in readme file that'll contain items from your list as well.

[shroominic]

I dont know too much about .bin slicing and low level inference but I am really hyped about doing something like this
https://github.com/MeetKai/functionary, https://github.com/1rgs/jsonformer or https://github.com/guidance-ai/guidance
on top of CodeLlama.
I want to use if to enable local models with structured output into my https://github.com/shroominic/funcchain project. But I also want to get deeper into mojo and model inference so thats kind of the excuse to build other features and dig into that stuff :D

I mostly have experience in python and a bit ocaml, java so another goal would be to learn more about memory management and the more low level stuff introduced in mojo.
Also I'm open to be collaborator and help manage the project but need to PR something productive first!
You mentioned a discord but I cant find a link...