Add support for Gemini

[davidmigloz]

https://langchaindart.com/#/modules/model_io/models/chat_models/integrations/googleai

[Milksheyke]

I'd love to take a stab at this, if you don't mind!
Is there a contribution guide, or should I write some spaghetti and make a PR?

[davidmigloz]

Hey @Milksheyke!

Great to hear that you are interested in contributing to the project 🙂

At the moment I don't have any contribution guide, unfortunately. We can maybe discuss how to implement it here.

I already added support for Gemini on LangChain.dart, so the integration should be straightforward. We may need to think about how we adapt the UX flow to support different models (I would also like to add support for Ollama to run local models). Do you have any ideas?

[Milksheyke]

I'm fairly new to cooperating on code, so please forgive me if I misunderstood the question and stating the obvious here.

I was thinking a dropdown with the supported models, where the displayed fields dynamically change based on the selected model.

Based on the selected model, we execute the appropriate cubit event.
We could also use tabs instead of the dropdown so that the instructions, fields, layout, etc can be bundled together as a separate Widget for each supported model that display within _S3ApiKeys.

I've never worked with Ollama, but by local model, do you mean:

• A model running on the user's machine which they're using to visit the website, with the browser facilitating communication?
• A model running on the user's own server/cloud instance, with HTTP/Websockets for communication between pixel2flutter and the LLM based on the endpoint provided by the user?
• Both?

[davidmigloz]

sounds good!

With local models I mean that the models are running on the user's machine. There are a couple of multi-modal models that Ollama supports (bakllava and llava). Ollama server runs locally and exposes an API to interact with the model.

Regarding your second point, some users would like to be able to proxy the calls to OpenAI/Google (#2), so adding an option to customize the base URL/headers would be useful. We don't need to do it as part of this ticket, but good to have it in mind while we redesign the UI.

[tinoper]

Hey there, I'm catching up on this issue. I 'd like to help to add Gemini support to pixel2flutter. Have you made any progress on it ?

[davidmigloz]

hey @tinoper,

I didn't find time to work on this, unfortunately, so happy if you could take it over!

[tinoper]

Ok, I'll give it a try!
By the way, there is a discord link for LangChain.dart but it doesn't works. I have a question about ChatGoogleGenerativeAI in langchain_google package so it could be useful to join to discord LangChain community. By the way, great work!

[davidmigloz]

Awesome! This invite link should work: https://discord.gg/x4qbhqecVR

[tinoper]

Thank you, I'm in.
I found that ChatGoogleGenerativeAI doesn't have set method that the same openAI chat model already have.
On the other hand, reading a thread where you were explaining that langchain dart were considering deprecating googleai_dart and using the official client google_generative_ai. Would it be better for Pixel2Flutter to try adding google_generative_ai and forget about langchain?

[davidmigloz]

Hey @tinoper ,

Indeed, those methods were missing from ChatGoogleGenerativeAI. I've just added them (davidmigloz/langchain_dart#357).

I'm not planning to release a new version of LangChain.dart in the coming days, as I'm doing some refactoring that I want to release altogether. But you can depend on the commit that contains the API key getters/setters (I've pushed a branch with the dependencies pointing to that commit, you can work on top of it).

One of the main points of LangChain is to provide a unified API for different LLM providers, which is something useful for this project. The migration from googleai_dart to google_generative_ai is something that will happen internally on LangChain.dart, it won't affect its public API.

[tinoper]

The ChatGoogleGenerativeAI class doesn't have the supported streaming method (line 198). Not sure if it's worth it considering they might switch to the official package soon.

[davidmigloz]

Ahh true, I forgot streaming was pending to be implemented. I'll take over the ticket to migrate ChatGoogleGenerativeAI to the official client from @luisredondo (davidmigloz/langchain_dart#332). I can quickly implement it tonight. I'll let you know when it's done.

[luisredondo]

Hey @davidmigloz! Sorry, I have not prioritized this work; I have been very busy! I made an initial effort some time ago. In case it helps you, here is that attempt: luisredondo/langchain_dart#1

Let me know if I can help with something.

[davidmigloz]

@tinoper ChatGoogleGenerativeAI now supports streaming (davidmigloz/langchain_dart#360), I've updated the dependencies in the google_ai branch to point to the latest LangChain.dart commit. Hopefully there isn't more blockers!

[davidmigloz]

@luisredondo no worries! I imagined that. I've halted the migration as the official client is still not as feature complete as googleai_dart (davidmigloz/langchain_dart#332 (comment)). Thanks for your work!

[tinoper]

@davidmigloz Thank you for your quick response. Now I have the stream, but I'm encountering an issue when mapping it similar to what GenerateCodeFromImageUseCase class does. The mapper located in langchain_google/lib/src/chat_models/google_ai/mappers.dart is giving me the error message 'Google AI does not support system messages at the moment. Attach your system message in the human message.'

It's not clear to me if this is something we should address in the mapper or in the content we're sending. I'm trying to send it in the same way we do with the OpenAI option for consistency.

[davidmigloz]

@tinoper step by step 😄

Google models don't support system messages (messages with role system) yet, as other providers like OpenAI do.
We should probably define two prompt templates, the existing one for GPT-4 and a new one for Gemini using a single human message.

Anyway, the way you prompt Gemini is normally slightly different than the way you prompt GPT-4. OpenAI models are quite good at following instructions, whereas Google models are better at following examples. So we'll have to experiment until we find a good prompt for Gemini.

[davidmigloz]

@tinoper the new release of LangChain.dart is out, so there's no need to override the dependencies anymore
https://github.com/davidmigloz/langchain_dart/releases/tag/langchain-v0.5.0

[tinoper]

Hey @davidmigloz , big changes in LangChain.dart then but before updating I'd like to try getting something by querying Gemini. It's not clear to me if PromptValue is able to handle the query without ChatMessage.system(_systemPrompt). If that's the case, I can only remove that line and include another complete prompt in ChatMessage.human.

Maybe to avoid making too much noise here, I'll contact you via Discord if you agree.

I've tried using the Google client outside of here to see how it handles it and the prompt should change quite a bit but it could serve as an alternative.

[davidmigloz]

Hey @tinoper ,

Yes, PromptValue doesn't require a system message. You can include everything as a human message.

I've quickly integrated ChatGoogleGenerativeAI provider in the generate_code_from_image.dart use case . I've pushed the changes in this branch: 1891d23

I've tested it hardcoding my Google AI key, and it seems to be working fine. You can take it over from there 🙂

What's missing:

• Update the UI to allow selecting OpenAI or GoogleAI
• Allow to enter Google AI key
• Optimize Gemini prompt (I've just copy-pasted the one from OpenAI)

Thanks for your work and feel free to contact me on Discord!

[ps6067966]

Hey @tinoper ,

Yes, PromptValue doesn't require a system message. You can include everything as a human message.

I've quickly integrated ChatGoogleGenerativeAI provider in the generate_code_from_image.dart use case . I've pushed the changes in this branch: 1891d23

I've tested it hardcoding my Google AI key, and it seems to be working fine. You can take it over from there 🙂

What's missing:

• Update the UI to allow selecting OpenAI or GoogleAI
• Allow to enter Google AI key
• Optimize Gemini prompt (I've just copy-pasted the one from OpenAI)

Thanks for your work and feel free to contact me on Discord!

Wow

[tinoper]

@davidmigloz , sorry for the hold-up. What do you think of a segmented button like this for entering the API key?
We can always add more options later, after GoogleAI.

[davidmigloz]

@tinoper Looks good! Thanks for following up