Investigate/document how to use with OpenRouter

[ErikBjare]

As I mentioned in #1082 (review), OpenRouter is an easy way to run lots of different LLMs through a OpenAI-compatible API (notably recommended by Aider).

We should consider investigating this as a user-friendly way to use open models without needing your own hardware, and document it accordingly.

@zigabrencic did some investigation on this. Opening this issue to track our investigation and potentially documenting to users how to use it.

One notable downside of it, and open models generally, is that it may lead to all kinds of bugs caused by less powerful models. We might want to mention this in docs so that users are aware, and don't open spurious bug reports.

[zigabrencic]

@vibor you can assign me to this one. Will start looking into it at towards the end of the week.

[viborc]

Done, thanks, @zigabrencic and @ErikBjare!

[TheoMcCabe]

I wrote an article on this

https://medium.com/@tedisaacs/from-openai-to-opensource-in-2-lines-of-code-b4b8d2cf2541

[zigabrencic]

@TheoMcCabe that's great. Thanks for sharing the post.

The issue I came across with gpte was that specific open model {sizes, types} result into rather poor performance of gpte. Running on the local machine has the same issue as with OpenRouter.

In other words. Making the API calls to open router via langchain is the easy part. Making gpte work with open models, is well harder. Hopefully running new gpte benchmarks with Open Models gives us some clarity here.

I'm waiting for llama3 to come out this week to see if it's any better. If it is I suspect llama3 to be also available on OpenRouter as well.

I'm not against adding a few lines from your post to the docs right away. We just need to warn the users as Erik mentioned above about experimentally of the feature.

[TheoMcCabe]

Yeah i saw the same thing with mynt. The only things that could handle the logic required to contruct the requests properly were anthropic or open ai models. Even google gemini sucked!

There are other benefits to using open router other than just getting access to additional models which work well with gpte... like being visible on the open router app rankings. And having access to new models early (which may in future work well with gpte)

I'd say the models not working very well shouldnt be a blocker on us completing this integration

[TheoMcCabe]

I think this can be closed now as you PR is merged @zigabrencic ?