TEM Client Submitting Invalid JSON data to API Endpoint

[philipsoutham]

It appears that the TEM client (after version 2) is now placing the from details object in an JSON array instead of the expected JSON object.

Tested on Python version: ">=3.10,<3.13"
Affected scaleway-sdk-python versions: ">=2.0.0,<=2.1.0"

Code to reproduce:

import asyncio
from pprint import pprint

from scaleway_async import Client
from scaleway_async.tem.v1alpha1.api import TemV1Alpha1API
from scaleway_async.tem.v1alpha1.types import CreateEmailRequestAddress

from app.core import settings, setup_logging
setup_logging()


_scaleway_client = Client(
    access_key=settings.SCALEWAY_ACCESS_KEY,
    secret_key=settings.SCALEWAY_SECRET_KEY,
    default_organization_id=settings.SCALEWAY_ORGANIZATION_ID,
    default_project_id=settings.SCALEWAY_PROJECT_ID,
    default_region=settings.SCALEWAY_REGION,
    default_zone=settings.SCALEWAY_ZONE,
)

_transactional_email_api = TemV1Alpha1API(_scaleway_client)


async def main():
    """
    Real email addresses are masked for privacy/security
    """
    to_email = "******@*************.com"
    from_email = "do-not-reply@*********.fm"
    email_msg = "Here is a test email."
    resp = await _transactional_email_api.create_email(
        subject=email_msg,
        text=email_msg,
        html=email_msg,
        project_id=settings.SCALEWAY_PROJECT_ID,
        from_=CreateEmailRequestAddress(email=from_email, name=from_email),
        to=[CreateEmailRequestAddress(email=to_email, name=to_email)],
    )
    pprint(resp)


if __name__ == "__main__":
    asyncio.run(main())

Faulty Behavior

with scaleway-sdk-python versions: ">=2.0.0,<=2.1.0" response is:

🚀⚙️🔍 2024-04-30 11:04:23,603 - DEBUG - /usr/lib/python3.10/asyncio/selector_events.py#__init__:54 - Using selector: EpollSelector
🚀⚙️🔍 2024-04-30 11:04:23,604 - DEBUG - /home/********/.cache/pypoetry/virtualenvs/***********************-HSjrFIFp-py3.10/lib/python3.10/site-packages/scaleway_core/api.py#log_request:59 - --------------- Scaleway SDK REQUEST 1 ---------------

POST https://api.scaleway.com/transactional-email/v1alpha1/regions/fr-par/emails?
accept: application/json
x-auth-token: *****
user-agent: scaleway-sdk-python/2.1.0
Content-Type: application/json; charset=utf-8

{"subject": "Here is a test email.", "text": "Here is a test email.", "html": "Here is a test email.", "from": [{"email": "do-not-reply@*********.fm", "name": "do-not-reply@*********.fm"}], "to": [{"email": "******@*************.com", "name": "******@*************.com"}], "project_id": "********-****-****-****-************"}
-------------------------------------------------------
🚀⚙️🔍 2024-04-30 11:04:23,605 - DEBUG - /home/********/.cache/pypoetry/virtualenvs/***********************-HSjrFIFp-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py#_new_conn:1019 - Starting new HTTPS connection (1): api.scaleway.com:443
🚀⚙️🔍 2024-04-30 11:04:24,543 - DEBUG - /home/********/.cache/pypoetry/virtualenvs/***********************-HSjrFIFp-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py#_make_request:474 - https://api.scaleway.com:443 "POST /transactional-email/v1alpha1/regions/fr-par/emails HTTP/1.1" 400 67
🚀⚙️🔍 2024-04-30 11:04:24,545 - DEBUG - /home/********/.cache/pypoetry/virtualenvs/***********************-HSjrFIFp-py3.10/lib/python3.10/site-packages/scaleway_core/api.py#log_response:84 - --------------- Scaleway SDK RESPONSE 1 ---------------
HTTP 400 NOK
content-type: application/json
content-length: 67
date: Tue, 30 Apr 2024 18:04:24 GMT
x-request-id: a1cdae83-36b5-4080-8f3e-81069d49113a
content-security-policy: default-src 'none'; frame-ancestors 'none'
strict-transport-security: max-age=63072000
x-content-type-options: nosniff
x-frame-options: DENY
server: Scaleway API Gateway (fr-par-3;edge02)

{"message":"proto: syntax error (line 1:112): unexpected token ["}
-------------------------------------------------------
🚀💣🔍 2024-04-30 11:04:24,546 - CRITICAL - /home/********/****src****/***********************/app/core/logging.py#handle_exception:64 - 🐞🔍 Uncaught exception
Traceback (most recent call last):
  File "/home/********/****src****/***********************/scratch_email_bug_test.py", line 43, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/********/****src****/***********************/scratch_email_bug_test.py", line 31, in main
    resp = await _transactional_email_api.create_email(
  File "/home/********/.cache/pypoetry/virtualenvs/***********************-HSjrFIFp-py3.10/lib/python3.10/site-packages/scaleway_async/tem/v1alpha1/api.py", line 124, in create_email
    self._throw_on_error(res)
  File "/home/********/.cache/pypoetry/virtualenvs/***********************-HSjrFIFp-py3.10/lib/python3.10/site-packages/scaleway_core/api.py", line 188, in _throw_on_error
    raise ScalewayException(res)
scaleway_core.api.ScalewayException: {"message":"proto: syntax error (line 1:112): unexpected token ["}

Expected Behavior

With scaleway-sdk-python version: "~1.8.0" the email is send just fine.

🚀⚙️🔍 2024-04-30 11:06:48,909 - DEBUG - /usr/lib/python3.10/asyncio/selector_events.py#__init__:54 - Using selector: EpollSelector
🚀⚙️🔍 2024-04-30 11:06:48,910 - DEBUG - /home/********/.cache/pypoetry/virtualenvs/***********************-HSjrFIFp-py3.10/lib/python3.10/site-packages/scaleway_core/api.py#log_request:59 - --------------- Scaleway SDK REQUEST 1 ---------------

POST https://api.scaleway.com/transactional-email/v1alpha1/regions/fr-par/emails?
accept: application/json
x-auth-token: *****
user-agent: scaleway-sdk-python/1.8.0
Content-Type: application/json; charset=utf-8

{"from": {"email": "do-not-reply@*********.fm", "name": "do-not-reply@*********.fm"}, "html": "Here is a test email.", "project_id": "********-****-****-****-************", "subject": "Here is a test email.", "text": "Here is a test email.", "to": [{"email": "******@*************.com", "name": "******@*************.com"}]}
-------------------------------------------------------
🚀⚙️🔍 2024-04-30 11:06:48,911 - DEBUG - /home/********/.cache/pypoetry/virtualenvs/***********************-HSjrFIFp-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py#_new_conn:1019 - Starting new HTTPS connection (1): api.scaleway.com:443
🚀⚙️🔍 2024-04-30 11:06:50,354 - DEBUG - /home/********/.cache/pypoetry/virtualenvs/***********************-HSjrFIFp-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py#_make_request:474 - https://api.scaleway.com:443 "POST /transactional-email/v1alpha1/regions/fr-par/emails HTTP/1.1" 200 528
🚀⚙️🔍 2024-04-30 11:06:50,355 - DEBUG - /home/********/.cache/pypoetry/virtualenvs/***********************-HSjrFIFp-py3.10/lib/python3.10/site-packages/scaleway_core/api.py#log_response:84 - --------------- Scaleway SDK RESPONSE 1 ---------------
HTTP 200 OK
content-type: application/json
content-length: 528
date: Tue, 30 Apr 2024 18:06:50 GMT
x-request-id: 8c4663dd-f54b-4b9d-a583-e65cd12136ad
content-security-policy: default-src 'none'; frame-ancestors 'none'
strict-transport-security: max-age=63072000
x-content-type-options: nosniff
x-frame-options: DENY
server: Scaleway API Gateway (fr-par-3;edge02)

{"emails":[{"id":"e95a2b7c-683d-4a6f-91a2-bba688720f41", "message_id":"a694ec13-6813-4f77-89ff-c9c65aaebb3d", "project_id":"********-****-****-****-************", "mail_from":"do-not-reply@*********.fm", "rcpt_to":"******@*************.com", "mail_rcpt":"******@*************.com", "rcpt_type":"to", "subject":"Here is a test email.", "created_at":"2024-04-30T18:06:50.020850616Z", "updated_at":"2024-04-30T18:06:50.020850616Z", "status":"sending", "status_details":"ready to send", "try_count":0, "last_tries":[], "flags":[]}]}
-------------------------------------------------------
CreateEmailResponse(emails=[Email(id='e95a2b7c-683d-4a6f-91a2-bba688720f41',
                                  message_id='a694ec13-6813-4f77-89ff-c9c65aaebb3d',
                                  project_id='********-****-****-****-************',
                                  mail_from='do-not-reply@*********.fm',
                                  rcpt_to='******@*************.com',
                                  mail_rcpt='******@*************.com',
                                  rcpt_type='to',
                                  subject='Here is a test email.',
                                  created_at=datetime.datetime(2024, 4, 30, 18, 6, 50, 20850, tzinfo=tzutc()),
                                  updated_at=datetime.datetime(2024, 4, 30, 18, 6, 50, 20850, tzinfo=tzutc()),
                                  status='sending',
                                  status_details='ready to send',
                                  try_count=0,
                                  last_tries=[],
                                  flags=[])])

[RemiBauzac]

Hello @Laure-di @yfodil

I have the same issue. This is because the "from" is set as a tuple in marshal_CreateEmailRequest, and then converted to an array by the json serializer (for the final request). That's why we have unexpected token [.

The issue is here :

scaleway-sdk-python/scaleway/scaleway/tem/v1alpha1/marshalling.py

Line 640 in 89e0dc9

output["from"] = (marshal_CreateEmailRequestAddress(request.from_, defaults),)

The output["from"] is set as a tuple (why ?). If you remove the tuple creation, and just let the marshal_CreateEmailRequestAddress, it works fine.

[Laure-di]

Hey everyone!

Good news! The latest release of the scaleway-async package has fixed the issue. To get the update, just run this command:

pip install --upgrade scaleway-async

Thanks a bunch for your patience and support!