Feat add Amazon Bedrock support #1231

usamimeri · 2024-04-26T09:46:17Z

User description

Features
#1174

add support for Amazon Bedrock Provider (Meta, Mistral, Amazon, Ai21, Anthropic, Cohere)
add test for Amazon Bedrock

config

llm:
  api_type: "bedrock" 
  model: "meta.llama3-70b-instruct-v1:0"  # see metagpt/provider/bedrock/utils.py
  region_name: "REGION"
  access_key: "YOUR_ACCESS_KEY"
  secret_key: "YOUR_SECRET_KEY"

test result

platform win32 -- Python 3.9.19, pytest-8.1.1, pluggy-1.5.0
rootdir: D:\python_projects\MetaGPT
plugins: anyio-4.3.0, asyncio-0.23.6, cov-5.0.0, html-4.1.1, metadata-3.1.1, mock-3.14.0, timeout-2.3.1, xdist-3.5.0
asyncio: mode=strict
collected 129 items

tests\metagpt\provider\postprocess\test_base_postprocess_plugin.py .                                                                                                        [  0%]
tests\metagpt\provider\postprocess\test_llm_output_postprocess.py .                                                                                                         [  1%]
tests\metagpt\provider\test_amazon_bedrock_api.py ............................................................................................                              [ 72%]
tests\metagpt\provider\test_anthropic_api.py .                                                                                                                              [ 73%]
tests\metagpt\provider\test_azure_llm.py .                                                                                                                                  [ 74%]
tests\metagpt\provider\test_base_llm.py ..                                                                                                                                  [ 75%]
tests\metagpt\provider\test_dashscope_api.py .                                                                                                                              [ 76%]
tests\metagpt\provider\test_general_api_base.py .......                                                                                                                     [ 82%]
tests\metagpt\provider\test_general_api_requestor.py ...                                                                                                                    [ 84%]
tests\metagpt\provider\test_google_gemini_api.py .                                                                                                                          [ 85%]
tests\metagpt\provider\test_human_provider.py .                                                                                                                             [ 86%]
tests\metagpt\provider\test_metagpt_llm.py .                                                                                                                                [ 86%]
tests\metagpt\provider\test_ollama_api.py .                                                                                                                                 [ 87%]
tests\metagpt\provider\test_openai.py ........                                                                                                                              [ 93%]
tests\metagpt\provider\test_qianfan_api.py .                                                                                                                                [ 94%]
tests\metagpt\provider\test_spark_api.py ...                                                                                                                                [ 96%]
tests\metagpt\provider\test_zhipuai_api.py ..                                                                                                                               [ 98%]
tests\metagpt\provider\zhipuai\test_async_sse_client.py .                                                                                                                   [ 99%]
tests\metagpt\provider\zhipuai\test_zhipu_model_api.py .                                                                                                                    [100%]

======================================================================== 129 passed, 7 warnings in 10.59s ========================================================================

completion test

models = ["cohere.command-text-v14", "mistral.mixtral-8x7b-instruct-v0:1", "meta.llama3-70b-instruct-v1:0",
              "anthropic.claude-3-haiku-20240307-v1:0", "ai21.j2-grande-instruct", "amazon.titan-text-express-v1"]
    for model_id in models:
        print(f"============{model_id}=============")
        my_config.model = model_id
        api = AmazonBedrockLLM(my_config)
        msg = [{"role": 'user', "content": "Please say: I am \{YOUR_MODEL_NAME\}"}]
        print(api.completion(msg))
        api._chat_completion_stream(msg)

2024-04-27 13:13:15.799 | INFO     | metagpt.const:get_metagpt_package_root:29 - Package root set to D:\python_projects\MetaGPT
============cohere.command-text-v14=============
 Hello, World!
 Hello, World!
============mistral.mixtral-8x7b-instruct-v0:1=============
 Hello World! I'm a helpful, respectful, and honest assistant, always ready to provide accurate and appropriate information. How can I assist you today? Is there a specific question or topic you'd like to discuss? I'm here to help make your life easier and more enjoyable.
 Hello World! I'm a helpful, respectful, and honest assistant, always ready to provide accurate and appropriate information. How can I assist you today? Is there a specific question or topic you'd like to discuss? I'm here to help make your life easier and more enjoyable.
============meta.llama3-70b-instruct-v1:0=============
Hello World!
Hello World!
============anthropic.claude-3-haiku-20240307-v1:0=============
Hello World!
Hello World!
============ai21.j2-grande-instruct=============
assistant: Hello World
============amazon.titan-text-express-v1=============
Sorry, but this model is not capable of saying "Hello World".
Sorry, but this model is not capable of saying "Hello World".

codiumai-pr-agent-pro · 2024-04-26T09:47:04Z

PR Description updated to latest commit (4c77d6c)

Copy walkthrough table to "Files Changed" Tab

codiumai-pr-agent-pro · 2024-04-26T09:47:17Z

PR Review

⏱️ Estimated effort to review [1-5]	4, because the PR introduces a significant amount of new code across multiple files, including new classes, methods, and configurations specific to the Amazon Bedrock API integration. The complexity of the code, especially with the handling of asynchronous operations and streaming responses, requires a thorough understanding of both the existing system architecture and the new API's capabilities.
🧪 Relevant tests	Yes
🔍 Possible issues	Possible Bug: The code assumes that all responses from the Amazon Bedrock API will successfully convert to JSON without handling potential exceptions that might occur during the parsing.
🔍 Possible issues	Performance Concern: The synchronous calls to the Amazon Bedrock API might lead to blocking I/O operations that could affect the performance of the entire application, especially under high load.
🔒 Security concerns	No

✨ Review tool usage guide:

Overview:
The review tool scans the PR code changes, and generates a PR review which includes several types of feedbacks, such as possible PR issues, security threats and relevant test in the PR. More feedbacks can be added by configuring the tool.

The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.

When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...

With a configuration file, use the following template:

[pr_reviewer]
some_config1=...
some_config2=...

See the review usage page for a comprehensive guide on using this tool.

codiumai-pr-agent-pro · 2024-04-26T09:48:15Z

PR Code Suggestions

Category	Suggestions
Maintainability	Improve variable naming for clarity. Consider using a more descriptive variable name than `resp_cont_tmpl` for better code readability. The current name does not clearly indicate the purpose or content of the template. tests/metagpt/provider/req_resp_const.py [69] message=ChatCompletionMessage( - role="assistant", content=resp_cont_tmpl.format(name=name)), + role="assistant", content=response_content_template.format(name=name)),
Maintainability	Reduce code duplication by using a factory function for object creation. Refactor the repeated creation of `CompletionUsage` objects with identical parameters into a shared function or a factory pattern to reduce code duplication and improve maintainability. tests/metagpt/provider/req_resp_const.py [80-81] -usage=CompletionUsage(completion_tokens=110, - prompt_tokens=92, total_tokens=202) +def create_default_completion_usage(): + return CompletionUsage(completion_tokens=110, + prompt_tokens=92, total_tokens=202) +usage = create_default_completion_usage()
Possible issue	Ensure variable is defined before use to prevent runtime errors. To avoid potential runtime errors, ensure that `resp_cont_tmpl` is defined and accessible within the scope where it is used, or pass it explicitly to functions that require it. tests/metagpt/provider/req_resp_const.py [69] +# Ensure resp_cont_tmpl is defined earlier in the code or passed as a parameter message=ChatCompletionMessage( role="assistant", content=resp_cont_tmpl.format(name=name)),
Possible issue	Add validation for empty or None prompt text in `get_request_body` method. The method `get_request_body` in the `AmazonProvider` class should handle the case where `messages_to_prompt` returns `None` or an empty string, which could lead to invalid JSON structure. It's recommended to add a check and raise an appropriate exception if necessary. metagpt/provider/bedrock/bedrock_provider.py [69-72] +prompt_text = self.messages_to_prompt(messages) +if not prompt_text: + raise ValueError("Prompt text is empty or None") body = json.dumps({ - "inputText": self.messages_to_prompt(messages), + "inputText": prompt_text, "textGenerationConfig": generate_kwargs })
Best practice	Use constants or configuration for token values to enhance maintainability. Instead of hardcoding token values directly in the function, consider defining them as constants or retrieving them from a configuration file to make the code more flexible and maintainable. tests/metagpt/provider/req_resp_const.py [73-74] -usage=CompletionUsage(completion_tokens=110, - prompt_tokens=92, total_tokens=202), +# Define token values as constants or retrieve from a config +DEFAULT_COMPLETION_TOKENS = 110 +DEFAULT_PROMPT_TOKENS = 92 +DEFAULT_TOTAL_TOKENS = 202 +usage=CompletionUsage(completion_tokens=DEFAULT_COMPLETION_TOKENS, + prompt_tokens=DEFAULT_PROMPT_TOKENS, total_tokens=DEFAULT_TOTAL_TOKENS),
Best practice	Prevent key conflicts in `generate_kwargs` in `get_request_body` of `AnthropicProvider`. The `get_request_body` method in the `AnthropicProvider` class should ensure that the `generate_kwargs` does not contain keys that might conflict with the hardcoded keys in the dictionary, potentially leading to unexpected behavior. metagpt/provider/bedrock/bedrock_provider.py [21-23] +reserved_keys = {"messages", "anthropic_version"} +if any(key in generate_kwargs for key in reserved_keys): + raise KeyError("generate_kwargs contains keys that are reserved: " + ", ".join(reserved_keys)) body = json.dumps( {"messages": messages, "anthropic_version": "bedrock-2023-05-31", **generate_kwargs})
Enhancement	Dynamically compute `total_tokens` to ensure accuracy. To ensure the `total_tokens` value is always accurate, compute it dynamically based on the sum of `completion_tokens` and `prompt_tokens` rather than hardcoding it. tests/metagpt/provider/req_resp_const.py [73-74] -usage=CompletionUsage(completion_tokens=110, - prompt_tokens=92, total_tokens=202), +completion_tokens = 110 +prompt_tokens = 92 +usage=CompletionUsage(completion_tokens=completion_tokens, + prompt_tokens=prompt_tokens, total_tokens=completion_tokens + prompt_tokens),
Enhancement	Validate `llama_version` input in the constructor of `MetaProvider`. In the `MetaProvider` class, the constructor should validate the `llama_version` argument to ensure it only receives 'llama2' or 'llama3' as valid inputs. metagpt/provider/bedrock/bedrock_provider.py [41-43] def __init__(self, llama_version: Literal["llama2", "llama3"]) -> None: + if llama_version not in ["llama2", "llama3"]: + raise ValueError("Invalid llama_version, must be 'llama2' or 'llama3'") self.llama_version = llama_version
Error handling	Add error handling for JSON parsing in `get_choice_text_from_stream`. The `get_choice_text_from_stream` method in the `AmazonProvider` class should handle potential JSON parsing errors with a try-except block to prevent crashes due to malformed data. metagpt/provider/bedrock/bedrock_provider.py [78-81] -rsp_dict = json.loads(event["chunk"]["bytes"]) -completions = rsp_dict["outputText"] +try: + rsp_dict = json.loads(event["chunk"]["bytes"]) + completions = rsp_dict["outputText"] +except json.JSONDecodeError: + raise ValueError("Failed to decode JSON from stream")
Error handling	Add key existence validation in `get_choice_text_from_stream`. The `get_choice_text_from_stream` method in the `BaseBedrockProvider` class should validate the structure of the `rsp_dict` before accessing nested keys to avoid `KeyError`. metagpt/provider/bedrock/bedrock_provider.py [22-25] rsp_dict = json.loads(event["chunk"]["bytes"]) +if "outputText" not in rsp_dict: + raise KeyError("Expected key 'outputText' not found in response dictionary") completions = self._get_completion_from_dict(rsp_dict)

✨ Improve tool usage guide:

Overview:
The improve tool scans the PR code changes, and automatically generates suggestions for improving the PR code. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on a PR.

When commenting, to edit configurations related to the improve tool (pr_code_suggestions section), use the following template:

/improve --pr_code_suggestions.some_config1=... --pr_code_suggestions.some_config2=...

With a configuration file, use the following template:

[pr_code_suggestions]
some_config1=...
some_config2=...

See the improve usage page for a comprehensive guide on using this tool.

codecov-commenter · 2024-04-26T09:57:49Z

Codecov Report

Attention: Patch coverage is 38.38863% with 130 lines in your changes are missing coverage. Please review.

Project coverage is 33.64%. Comparing base (f201b2f) to head (98cb452).

Files	Patch %	Lines
metagpt/provider/bedrock/amazon_bedrock_api.py	37.03%	51 Missing ⚠️
metagpt/provider/bedrock/utils.py	16.27%	36 Missing ⚠️
metagpt/provider/bedrock/bedrock_provider.py	46.87%	34 Missing ⚠️
metagpt/provider/bedrock/base_provider.py	52.63%	9 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1231       +/-   ##
===========================================
- Coverage   70.22%   33.64%   -36.59%     
===========================================
  Files         316      320        +4     
  Lines       18860    19070      +210     
===========================================
- Hits        13245     6416     -6829     
- Misses       5615    12654     +7039

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

fix naming

better629 · 2024-04-28T09:51:03Z

metagpt/configs/llm_config.py

@@ -32,6 +32,7 @@ class LLMType(Enum):
    MISTRAL = "mistral"
    YI = "yi"  # lingyiwanwu
    OPENROUTER = "openrouter"
+    AMAZON_BEDROCK = "amazon_bedrock"


If BEDROCK is distinguishable, no need to use AMAZON_BEDROCK, but you can comment it with "amazon bedrock"

metagpt/provider/bedrock/.gitignore

metagpt/provider/bedrock/amazon_bedrock_api.py

metagpt/provider/bedrock/bedrock_provider.py

tests/metagpt/provider/req_resp_const.py

metagpt/provider/bedrock/amazon_bedrock_api.py

metagpt/provider/bedrock/base_provider.py

better629 · 2024-04-29T03:20:05Z

metagpt/provider/bedrock/utils.py

+        content = message["content"]
+        prompt += GENERAL_TEMPLATE.format(role=role, content=content)
+
+    if role != "assistant":


role will have a NameError if messages is empty

better629 · 2024-04-29T03:21:02Z

metagpt/provider/bedrock/utils.py

+
+
+def get_max_tokens(model_id) -> int:
+    return (NOT_SUUPORT_STREAM_MODELS | SUPPORT_STREAM_MODELS)[model_id]


.get(model_id, default_val) to avoid KeyError

better629 · 2024-04-29T03:24:53Z

metagpt/provider/bedrock_api.py

+            messages, self._const_kwargs)
+        response_body = self.invoke_model(request_body)
+        completions = self.__provider.get_choice_text(response_body)
+        return completions


completion return response structure not text from get_choice_text. see acompletion_text in BaseLLM

better629 · 2024-04-29T03:28:33Z

metagpt/provider/bedrock_api.py

+        response = self.__client.invoke_model_with_response_stream(
+            modelId=self.config.model, body=request_body)
+        usage = self._get_usage(response)
+        self._update_costs(usage)


Here doesn't have the price of each 1k tokens for model, so _update_costs will fail. Don't you notice this in your log?

better629 · 2024-04-29T03:29:06Z

metagpt/provider/bedrock_api.py

+        if self.config.model in NOT_SUUPORT_STREAM_MODELS:
+            logger.warning(
+                f"model {self.config.model} doesn't support streaming output!")
+            return self.completion(messages)


notice here, completion return object not str, see comment like before.

better629 · 2024-05-06T11:21:25Z

lgtm

geekan

LGTM

usamimeri and others added 22 commits April 25, 2024 12:00

implement framework

fd4c0fd

implement base provider

4f14ee7

support mistral

ec7df8a

add generate_kwargs

4d1fb20

add stream

a741488

implement meta

9775a2b

support anthropic

187e9ef

implement all model

6355c5c

change provider to private

a6058ca

add titan

f45a379

add some type hint

aded1dc

update provider package

6452adf

update max tokens and support max_tokens field

784c626

fix llama3 chat template bug

0cdca1b

add claude chat template

e9723f4

add docs

6561c7a

lazy installation

cafe666

add test

4c394a1

stream test

8fafa2e

update

83c8ccb

update test

b0ed292

Merge branch 'geekan:main' into main

4c77d6c

usamimeri requested a deployment to unittest April 26, 2024 09:46 — with GitHub Actions Waiting

codiumai-pr-agent-pro bot added enhancement New feature or request tests labels Apr 26, 2024

codiumai-pr-agent-pro bot added the Review effort [1-5]: 4 label Apr 26, 2024

Update amazon_bedrock_api.py

6776e5c

fix naming

usamimeri requested a deployment to unittest April 26, 2024 15:12 — with GitHub Actions Waiting

Merge branch 'geekan:main' into main

3411e7d

usamimeri requested a deployment to unittest April 27, 2024 03:58 — with GitHub Actions Waiting

usamimeri added 2 commits April 27, 2024 12:18

fix claude streaming bug

a05d257

remove opus since unavailable now and fix bug

98cb452

usamimeri requested a deployment to unittest April 27, 2024 05:09 — with GitHub Actions Waiting

better629 reviewed Apr 28, 2024

View reviewed changes

usamimeri added 3 commits April 28, 2024 19:18

resolve problems

79251cd

add token counts

cb8e14d

delete cache

aa8e123

usamimeri requested a deployment to unittest April 28, 2024 11:32 — with GitHub Actions Waiting

fix

986e3c8

usamimeri requested a deployment to unittest April 29, 2024 01:38 — with GitHub Actions Waiting

rename bedrock class and add more tests

3f108ab

usamimeri requested a deployment to unittest April 29, 2024 02:47 — with GitHub Actions Waiting

better629 reviewed Apr 29, 2024

View reviewed changes

usamimeri added 2 commits April 29, 2024 15:04

add pre-commit

f14a1f6

resolve problem and add cost manager

0006b62

usamimeri requested a deployment to unittest April 29, 2024 09:29 — with GitHub Actions Waiting

change log for non-streaming model

f7b29ed

usamimeri requested a deployment to unittest April 29, 2024 09:36 — with GitHub Actions Waiting

better629 mentioned this pull request May 6, 2024

New provider: Amazon Bedrock (AWS) #1174

Closed

usamimeri mentioned this pull request May 6, 2024

Add Bedrock API docs geekan/MetaGPT-docs#147

Open

Merge branch 'geekan:main' into main

0916399

usamimeri requested a deployment to unittest May 16, 2024 12:51 — with GitHub Actions Waiting

usamimeri temporarily deployed to pre-commit May 16, 2024 12:52 — with GitHub Actions Inactive

geekan approved these changes May 17, 2024

View reviewed changes

geekan merged commit ca67c8f into geekan:main May 17, 2024
1 of 3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat add Amazon Bedrock support #1231

Feat add Amazon Bedrock support #1231

usamimeri commented Apr 26, 2024 •

edited

codiumai-pr-agent-pro bot commented Apr 26, 2024

codiumai-pr-agent-pro bot commented Apr 26, 2024

codiumai-pr-agent-pro bot commented Apr 26, 2024

codecov-commenter commented Apr 26, 2024 •

edited

better629 Apr 28, 2024

better629 Apr 29, 2024

better629 Apr 29, 2024

better629 Apr 29, 2024

better629 Apr 29, 2024

better629 Apr 29, 2024

better629 commented May 6, 2024

geekan left a comment



		def get_max_tokens(model_id) -> int:
		return (NOT_SUUPORT_STREAM_MODELS \| SUPPORT_STREAM_MODELS)[model_id]

Feat add Amazon Bedrock support #1231

Feat add Amazon Bedrock support #1231

Conversation

usamimeri commented Apr 26, 2024 • edited

User description

codiumai-pr-agent-pro bot commented Apr 26, 2024

codiumai-pr-agent-pro bot commented Apr 26, 2024

PR Review

codiumai-pr-agent-pro bot commented Apr 26, 2024

PR Code Suggestions

codecov-commenter commented Apr 26, 2024 • edited

Codecov Report

better629 Apr 28, 2024

Choose a reason for hiding this comment

better629 Apr 29, 2024

Choose a reason for hiding this comment

better629 Apr 29, 2024

Choose a reason for hiding this comment

better629 Apr 29, 2024

Choose a reason for hiding this comment

better629 Apr 29, 2024

Choose a reason for hiding this comment

better629 Apr 29, 2024

Choose a reason for hiding this comment

better629 commented May 6, 2024

geekan left a comment

Choose a reason for hiding this comment

usamimeri commented Apr 26, 2024 •

edited

codecov-commenter commented Apr 26, 2024 •

edited