🐛 fix: Azure OpenAI Vision models issue #2429

AnoyiX · 2024-05-08T07:16:14Z

💻 变更类型 | Change Type

🔀 变更说明 | Description of Change

Fixed an issue where the server side was abnormal when using the Vision model in Azure OpenAI.

修复 Azure OpenAI 在使用 Vision 模型时，服务端异常的问题。

📝 补充信息 | Additional Information

在使用 Vision 模型时，如果用户上传图片，那么 Messages 中会存在 type 为 image_url 的内容，示例如下：

{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Describe this picture:"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "<image URL>"
          }
        }
      ]
    }
  ]
}

然而，官方的 SDK @azure/openai 在序列化 Message 时，用的 imageUrl 这个键值，从而导致报错 ISSUE #2207

官方 SDK 源码：node_modules/@azure/openai/dist/index.cjs 中的函数 serializeChatMessageImageContentItem

/** serialize function for ChatMessageImageContentItem */
function serializeChatMessageImageContentItem(obj) {
    return {
        type: obj["type"],
        image_url: { url: obj.imageUrl["url"], detail: obj.imageUrl["detail"] },
    };
}

vercel · 2024-05-08T07:16:19Z

@AnoyiX is attempting to deploy a commit to the LobeHub Team on Vercel.

A member of the Team first needs to authorize it.

lobehubbot · 2024-05-08T07:16:32Z

👍 @AnoyiX

Thank you for raising your pull request and contributing to our Community
Please make sure you have followed our contributing guidelines. We will review it as soon as possible.
If you encounter any problems, please feel free to connect with us.
非常感谢您提出拉取请求并为我们的社区做出贡献，请确保您已经遵循了我们的贡献指南，我们会尽快审查它。
如果您遇到任何问题，请随时与我们联系。

codecov · 2024-05-08T07:20:30Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.80%. Comparing base (0deb079) to head (62e1e5c).
Report is 1 commits behind head on main.

❗ Current head 62e1e5c differs from pull request most recent head 2e71044. Consider uploading reports for the commit 2e71044 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2429      +/-   ##
==========================================
+ Coverage   92.79%   92.80%   +0.01%     
==========================================
  Files         302      302              
  Lines       17621    17648      +27     
  Branches     1267     2041     +774     
==========================================
+ Hits        16352    16379      +27     
  Misses       1269     1269

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

arvinxx · 2024-05-08T09:21:43Z

这个问题不算是 azure-openai 官方 SDK 的问题么？

lobehubbot · 2024-05-08T09:21:54Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is this problem not a problem with the azure-openai official SDK?

AnoyiX · 2024-05-08T09:46:17Z

这个问题不算是 azure-openai 官方 SDK 的问题么？

也不能完全说是官方 SDK 的问题，官方的 SDK 中字段的定义基本都是驼峰格式的，如果单独使用官方的 SDK 去测试，按官方的格式来，是没有问题的。现在 LobeChat 传的 messages 字段格式不转换成驼峰格式的话，就和官方不兼容。

lobehubbot · 2024-05-08T09:46:30Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

Is this problem not a problem with the azure-openai official SDK?

It cannot be said that it is entirely a problem with the official SDK. The definitions of fields in the official SDK are basically in camel case format. If you use the official SDK alone for testing and follow the official format, there will be no problem. If the format of the messages field passed by LobeChat is not converted into camel case format, it will be incompatible with the official version.

src/libs/agent-runtime/azureOpenai/index.ts

vercel · 2024-05-10T03:54:56Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
lobe-chat-community	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 10, 2024 7:32am

AnoyiX · 2024-05-10T03:58:38Z

代码中，给 maxTokens 增加了一个默认值，是因为在默认情况下，使用 Azure 的 Vision 模型，如果不传 max_tokens 参数，AI 回复的内容不全，仅会响应少量文本，这里设置一个较大的默认值，就会接收到完整的 AI 回复。

这应该是 Azure 官方 API 的问题，这里给 maxTokens 增加了一个默认值仅仅是为了提升 LobeChat 的用户体验。

lobehubbot · 2024-05-10T03:58:51Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

In the code, a default value is added to maxTokens because by default, using Azure's Vision model, if the max_tokens parameter is not passed, the content of the AI reply will be incomplete and only a small amount of text will be responded. Here is a larger value. A large default value will receive a full AI reply.

This should be a problem with the official Azure API. A default value is added to maxTokens here just to improve the user experience of LobeChat.

arvinxx · 2024-05-10T06:54:49Z

这应该是 Azure 官方 API 的问题，这里给 maxTokens 增加了一个默认值仅仅是为了提升 LobeChat 的用户体验。

合理的 👍

lobehubbot · 2024-05-10T06:55:01Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

This should be a problem with the Azure official API. A default value is added to maxTokens here just to improve the user experience of LobeChat.

Reasonable 👍

arvinxx · 2024-05-10T06:58:14Z

@AnoyiX 跑下 lint 的 fix，构建没通过

AnoyiX · 2024-05-10T06:59:13Z

@AnoyiX 跑下 lint 的 fix，构建没通过

正在处理，等我几分钟

lobehubbot · 2024-05-10T06:59:22Z

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

@AnoyiX ran lint’s fix and the build failed.

Processing, wait a few minutes

lobehubbot · 2024-05-10T07:35:59Z

❤️ Great PR @AnoyiX ❤️

The growth of project is inseparable from user feedback and contribution, thanks for your contribution! If you are interesting with the lobehub developer community, please join our discord and then dm @arvinxx or @canisminor1990. They will invite you to our private developer channel. We are talking about the lobe-chat development or sharing ai newsletter around the world.
项目的成长离不开用户反馈和贡献，感谢您的贡献! 如果您对 LobeHub 开发者社区感兴趣，请加入我们的 discord，然后私信 @arvinxx 或 @canisminor1990。他们会邀请您加入我们的私密开发者频道。我们将会讨论关于 Lobe Chat 的开发，分享和讨论全球范围内的 AI 消息。

### [Version 0.156.1](v0.156.0...v0.156.1) <sup>Released on **2024-05-10**</sup> #### 🐛 Bug Fixes - **misc**: Azure OpenAI Vision models issue. <br/> <details> <summary><kbd>Improvements and Fixes</kbd></summary> #### What's fixed * **misc**: Azure OpenAI Vision models issue, closes [#2429](#2429) ([9b8a4b1](9b8a4b1)) </details> <div align="right"> [![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top) </div>

lobehubbot · 2024-05-10T07:40:04Z

🎉 This PR is included in version 0.156.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

### [Version 1.36.1](v1.36.0...v1.36.1) <sup>Released on **2024-05-10**</sup> #### 🐛 Bug Fixes - **misc**: Azure OpenAI Vision models issue. <br/> <details> <summary><kbd>Improvements and Fixes</kbd></summary> #### What's fixed * **misc**: Azure OpenAI Vision models issue, closes [lobehub#2429](https://github.com/bentwnghk/lobe-chat/issues/2429) ([9b8a4b1](9b8a4b1)) </details> <div align="right"> [![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top) </div>

### [Version 0.156.1](lobehub/lobe-chat@v0.156.0...v0.156.1) <sup>Released on **2024-05-10**</sup> #### 🐛 Bug Fixes - **misc**: Azure OpenAI Vision models issue. <br/> <details> <summary><kbd>Improvements and Fixes</kbd></summary> #### What's fixed * **misc**: Azure OpenAI Vision models issue, closes [lobehub#2429](lobehub#2429) ([9b8a4b1](lobehub@9b8a4b1)) </details> <div align="right"> [![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top) </div>

AnoyiX added 4 commits May 7, 2024 12:05

💄 style: Improve llm list when all closed

a6cb1d0

Merge branch 'lobehub:main' into main

dedbfa2

🐛 fix: Azure OpenAI vision models issue (lobehub#2207)

b98de56

🐛 fix: Azure OpenAI vision models issue (lobehub#2207)

b91dc6f

arvinxx requested changes May 8, 2024

View reviewed changes

src/libs/agent-runtime/azureOpenai/index.ts Outdated Show resolved Hide resolved

AnoyiX added 2 commits May 10, 2024 10:26

Merge branch 'lobehub:main' into main

f6fb302

🐛 fix: Azure OpenAI vision models issue (lobehub#2207)

4d45e8e

vercel bot had a problem deploying to Preview May 10, 2024 04:00 Failure

AnoyiX added 2 commits May 10, 2024 15:01

🐛 fix: Azure OpenAI vision models issue (lobehub#2207)

62e1e5c

🐛 fix: Azure OpenAI vision models issue (lobehub#2207)

2e71044

vercel bot deployed to Preview May 10, 2024 07:32 View deployment

arvinxx merged commit 9b8a4b1 into lobehub:main May 10, 2024
3 checks passed

lobehubbot added the released label May 10, 2024

TheNameIsNigel pushed a commit to TheNameIsNigel/lobe-chat that referenced this pull request May 15, 2024

🐛 fix: Azure OpenAI Vision models issue (lobehub#2429)

2e54535

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 fix: Azure OpenAI Vision models issue #2429

🐛 fix: Azure OpenAI Vision models issue #2429

AnoyiX commented May 8, 2024 •

edited by arvinxx

vercel bot commented May 8, 2024

lobehubbot commented May 8, 2024

codecov bot commented May 8, 2024 •

edited

arvinxx commented May 8, 2024

lobehubbot commented May 8, 2024

AnoyiX commented May 8, 2024

lobehubbot commented May 8, 2024

vercel bot commented May 10, 2024 •

edited

AnoyiX commented May 10, 2024

lobehubbot commented May 10, 2024

arvinxx commented May 10, 2024

lobehubbot commented May 10, 2024

arvinxx commented May 10, 2024

AnoyiX commented May 10, 2024

lobehubbot commented May 10, 2024

lobehubbot commented May 10, 2024

lobehubbot commented May 10, 2024

🐛 fix: Azure OpenAI Vision models issue #2429

🐛 fix: Azure OpenAI Vision models issue #2429

Conversation

AnoyiX commented May 8, 2024 • edited by arvinxx

💻 变更类型 | Change Type

🔀 变更说明 | Description of Change

📝 补充信息 | Additional Information

vercel bot commented May 8, 2024

lobehubbot commented May 8, 2024

codecov bot commented May 8, 2024 • edited

Codecov Report

arvinxx commented May 8, 2024

lobehubbot commented May 8, 2024

AnoyiX commented May 8, 2024

lobehubbot commented May 8, 2024

vercel bot commented May 10, 2024 • edited

AnoyiX commented May 10, 2024

lobehubbot commented May 10, 2024

arvinxx commented May 10, 2024

lobehubbot commented May 10, 2024

arvinxx commented May 10, 2024

AnoyiX commented May 10, 2024

lobehubbot commented May 10, 2024

lobehubbot commented May 10, 2024

lobehubbot commented May 10, 2024

AnoyiX commented May 8, 2024 •

edited by arvinxx

codecov bot commented May 8, 2024 •

edited

vercel bot commented May 10, 2024 •

edited