Batch sending of emails not working

[jeffoneill]

Hello,

I'm testing my code for the latest version of the library. When I send emails one at a time (one per API call), it works fine. When I send multiple emails in an API call, only one of the emails is received.

Below is an excerpt of JSON sent for a batch of 10 emails (minimally edited to remove personal info). I'm sending to my personal gmail account using the + notation, e.g., recipient+0@gmail.com through recipient+9@gmail.com.

One extra weirdness is that the Sendgrid email activity log says that all 10 emails are delivered even though they are not. I find it very strange that this works sending individually (to all 10 email addresses) but not in batches of 10 at once.

I'd greatly appreciate any help you can provide.

Jeff

{'categories': ['begin'],
 'custom_args': {'eid': '5715143556071424'},
 'from': {'email': 'noreply@example.com'},
 'personalizations': [{'custom_args': {'vkey': 'aglzfm'},
                       'send_at': 1486243132,
                       'substitutions': {'{{description}}': u'<p>xgfddfdfhdf</p>',
                                         '{{item_title}}': u'Best Cat',
                                         '{{manager_email}}': u'manager@example.com',
                                         '{{manager_name}}': u"Jeff O'Neill",
                                         '{{unsubscribe_link}}': u'https://www.example.com/remove/',
                                         '{{vote_link}}': u'https://www.example.com/vote/'},
                       'to': [{'email': u'recipient+0@gmail.com'}]},
                       [...]
                      {'custom_args': {'vkey': 'aglzfm'},
                       'send_at': 1486243141,
                       'substitutions': {'{{description}}': u'<p>xgfddfdfhdf</p>',
                                         '{{item_title}}': u'Best Cat',
                                         '{{manager_email}}': u'manager@example.com',
                                         '{{manager_name}}': u"Jeff O'Neill",
                                         '{{unsubscribe_link}}': u'https://www.example.com/remove/',
                                         '{{vote_link}}': u'https://www.example.com/vote/'},
                       'to': [{'email': u'recipient+9@gmail.com'}]}],
 'reply_to': {'email': u'manager@example.com'},
 'subject': u'Best Cat',
 'template_id': '6060c6e9-0376-4b52-ac15-e6befd906072'}

[thinkingserious]

Hello @jeffoneill,

It looks like you are trying to achieve this use case: https://sendgrid.com/docs/Classroom/Send/v3_Mail_Send/personalizations.html#-Sending-Two-Different-Emails-to-Two-Different-Groups-of-Recipients

At first glance, your payload looks correct.

I will need to dig in and try to reproduce. To that effect, I'm adding this to our backlog for further investigation.

With Best Regards,

Elmer

[jeffoneill]

Here is a simpler way to look at the issue.

When a single API call has multiple personalizations that send emails to the same gmail account, the gmail account will receive only one of the emails, even though the emails being sent are different from one another.

Consider this example:

data = {
    'from': {'email': 'noreply@example.com'},
    'personalizations': [
        {'substitutions': {'{{description}}': u'<p>first</p>'},
         'to': [{'email': u'recipient@gmail.com'}]},
        {'substitutions': {'{{description}}': u'<p>second</p>'},
         'to': [{'email': u'recipient@gmail.com'}]},
        {'substitutions': {'{{description}}': u'<p>third</p>'},
         'to': [{'email': u'recipient@gmail.com'}]}
        ],
    'reply_to': {'email': u'jeff.oneill@example.com'},
    'subject': u'Best Cat',
    'template_id': '6060c6e9-0376-4b52-ac15-e6befd906072'}
response = SG.client.mail.send.post(request_body=data)

Three emails should be sent to the same gmail account. The three emails are different because they each have different substitutions. However, only one of these three emails will be delivered.

Only one email is delivered even if the email addresses are different (but still end up at the same gmail account). For example, changing them to recipient+1@gmail.com, recipient+2@gmail.com, and recipient+3@gmail.com still results in a single email being delivered.

I suspect that Google is doing something here. It makes sense for Google to discard identical emails, but here the emails are different from each other.

[thinkingserious]

@jeffoneill,

Thanks for the additional information, that should help me reproduce.

[jeffoneill]

I suspect that the problem is that Sendgrid is using the same SMTP-ID for all of the personalizations. Gmail is receiving multiple emails to the same account with the same SMTP-ID, assumes that they are the same email, and delivers only one.

Below is an example of delivery metadata reported by Sendgrid and you can see that the messages all have the same SMTP-ID.

EMAIL recipient@gmail.com
REASON 250 2.0.0 OK 1486478238 n5si11909140iof.45 - gsmtp 
SMTP-ID <srIZMvKuRHOuGd0rHM93Eg@ismtpd0006p1iad1.sendgrid.net>
PROCESSED STRING February 7, 2017 - 09:37:18 AM
MSGID srIZMvKuRHOuGd0rHM93Eg.filter0807p1mdw1-6811-5899DB9B-43.2

EMAIL recipient@gmail.com
REASON 250 2.0.0 OK 1486478238 b20si11895501ioe.107 - gsmtp 
SMTP-ID <srIZMvKuRHOuGd0rHM93Eg@ismtpd0006p1iad1.sendgrid.net>
PROCESSED STRING February 7, 2017 - 09:37:18 AM
MSGID srIZMvKuRHOuGd0rHM93Eg.filter0807p1mdw1-6811-5899DB9B-43.0

EMAIL recipient@gmail.com
REASON 250 2.0.0 OK 1486478238 g140si7310879itg.69 - gsmtp 
SMTP-ID <srIZMvKuRHOuGd0rHM93Eg@ismtpd0006p1iad1.sendgrid.net>
PROCESSED STRING February 7, 2017 - 09:37:18 AM
MSGID srIZMvKuRHOuGd0rHM93Eg.filter0807p1mdw1-6811-5899DB9B-43.1

[thinkingserious]

Hello @jeffoneill,

This is an issue where you might want to dig in with our support team, until this ticket reaches the top of my queue. They can be reached at https://support.sendgrid.com.

BTW, I ran this quick test, and all three emails were delivered:

curl -X "POST" "https://api.sendgrid.com/v3/mail/send" \
        -H "Authorization: Bearer $SENDGRID_API_KEY"  \
        -H "Content-Type: application/json" \
        -d @post.json

post.json

{
    "from": {
        "name": "Example User",
        "email": "dx@sendgrid.com"
    },
    "personalizations": [
        {
            "to": [
                {
                    "email": "elmer.thomas+1@sendgrid.com"
                }
            ],
            "subject": "Test Subject1",
            "substitutions": {
                "-name-": "Elmer"
            }
        },
        {
            "to": [
                {
                    "email": "elmer.thomas+2@sendgrid.com"
                }
            ],
            "subject": "Test Subject2",
            "substitutions": {
                "-name-": "Robbin"
            }
        },
        {
            "to": [
                {
                    "email": "elmer.thomas+3@sendgrid.com"
                }
            ],
            "subject": "Test Subject3",
            "substitutions": {
                "-name-": "Thomas"
            }
        }
    ],
    "content": [
        {
            "type": "text/plain",
            "value": "Hello -name-"
        },
        {
            "type": "text/html",
            "value": "Goodbye -name-"
        }
    ]
}

[jeffoneill]

Elmer, Thanks for the feedback and trying that out. Is your Sendgrid email based on gmail (G suite or whatever they call it now)? Jeff *Jeff O'Neill | Founder and Director* jeff.oneill@opavote.com

…

On Wed, Feb 8, 2017 at 1:56 PM, Elmer Thomas ***@***.***> wrote: Hello @jeffoneill <https://github.com/jeffoneill>, This is an issue where you might want to dig in with our support team, until this ticket reaches the top of my queue. They can be reached at https://support.sendgrid.com. BTW, I ran this quick test, and all three emails were delivered: curl -X "POST" "https://api.sendgrid.com/v3/mail/send" \ -H "Authorization: Bearer $SENDGRID_API_KEY" \ -H "Content-Type: application/json" \ -d @post.json post.json { "from": { "name": "Example User", "email": ***@***.***" }, "personalizations": [ { "to": [ { "email": ***@***.***" } ], "subject": "Test Subject1", "substitutions": { "-name-": "Elmer" } }, { "to": [ { "email": ***@***.***" } ], "subject": "Test Subject2", "substitutions": { "-name-": "Robbin" } }, { "to": [ { "email": ***@***.***" } ], "subject": "Test Subject3", "substitutions": { "-name-": "Thomas" } } ], "content": [ { "type": "text/plain", "value": "Hello -name-" }, { "type": "text/html", "value": "Goodbye -name-" } ] } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#294 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKKSAIsGWpJHqEq4oWe8d27WhsyH59Nks5rag_5gaJpZM4L4lAk> .

[thinkingserious]

Yes. I also tried sending to my personal GMail account: https://www.screencast.com/t/MmI1KW8Izsb

[jeffoneill]

I'll follow up with support. Your example also worked for me, but if you change the emails to all have the same subject then only one is delivered. *Jeff O'Neill | Founder and Director* jeff.oneill@opavote.com

…

On Wed, Feb 8, 2017 at 2:03 PM, Elmer Thomas ***@***.***> wrote: Yes. I also tried sending to my personal GMail account: https://www.screencast.com/t/MmI1KW8Izsb — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#294 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKKSLly0fsccy__1mSrJwCAh9hVUTtMks5rahGZgaJpZM4L4lAk> .

[mbernier]

@jeffoneill This could actually be gmail protecting you, rather than our pipeline doing something weird. Our received event is actually informed by the recipient email server. Gmail will thread emails based on a couple things like subject, reply-to, and others. So they may be just "protecting" you from multiple of the same emails that were all received by them at relatively the same instant.

This actually saves them from having to write multiples of the same email to disk (considering the volume of emails they receive this is extremely smart) and it will keep the recipients from complaining to gmail, or marking a legit sender's emails as spam when they made a simple mistake. (I am not saying you made a mistake, just that in a normal sending situation this would likely happen as a result of a mistake).

If you send one at a time, a second or so apart - you should receive all of them.
Also, if you send to multiple addresses you should see all of them show up.

If these two use cases do not produce the correct result, please reach out to our support team ASAP.

[jeffoneill]

Hi Matt, That makes sense if the content of the emails is the same, but here the content of the emails is different. I am trying to send two different emails to the same user, but gmail is only delivering one of them. I also tried including the "send_at" parameter and sending the emails at 10 second intervals (with a single API call from me to Senggrid) and Google still only delivered one of them. If I send the emails at essentially the same time using a different API call for each, they all get delivered. The difference between batch sending and individual sending (from me to Sendgrid) appears to be that Sendgrid reuses the SMTP ID for batch sends. Google turns the SMTP ID into the MESSAGE ID of the email, and these need to be unique but they are not... I appreciate all your input and I'm looking forward to hearing back from your support team. Jeff

…

On Wed, Feb 8, 2017 at 5:40 PM, Matt Bernier ***@***.***> wrote: @jeffoneill <https://github.com/jeffoneill> This could actually be gmail protecting you, rather than our pipeline doing something weird. Our received event is actually informed by the recipient email server. Gmail will thread emails based on a couple things like subject, reply-to, and others. So they may be just "protecting" you from multiple of the same emails that were all received by them at relatively the same instant. This actually saves them from having to write multiples of the same email to disk (considering the volume of emails they receive this is extremely smart) and it will keep the recipients from complaining to gmail, or marking a legit sender's emails as spam when they made a simple mistake. (I am not saying you made a mistake, just that in a normal sending situation this would likely happen as a result of a mistake). If you send one at a time, a second or so apart - you should receive all of them. Also, if you send to multiple addresses you should see all of them show up. If these two use cases do not produce the correct result, please reach out to our support team ASAP. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#294 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKKSCvNvdue3fFkiwQ8h6iXlZmBv22mks5rakRAgaJpZM4L4lAk> .

[mbernier]

I reached out to our deliverability folks to see if they have seen this behavior with Gmail before.

I will let you know as soon as I have an answer.

[thinkingserious]

Hello @jeffoneill,

All of our research points to this being an issue on GMail's side. I recommend you reach out to their support team for further assistance.

Please let us know if there is anything else we can help with.

With Best Regards,

Elmer

[jeffoneill]

Hi Elmer, I appreciate you all taking the time to look into this, but from my perspective, this really seems like a Sendgrid issue and not a Gmail issue. (1) If I send two emails with *two separate API calls* at virtually the same time, both get delivered. (2) If I send the *exact same two emails in a single API call*, only one gets delivered. For both cases, the data I send to you is the SAME but the data you send to Gmail is DIFFERENT and that causes one email to not be delivered!!! This seems like actual mistake on your part that causes email to not be delivered, and since email delivery is a key metric, I don't understand why you won't fix this. I know I'm a tiny customer so I shouldn't be taking up too much of your time on this, but this does affect all your customers so I hope you consider this issue more before closing it. Thanks again, I do love your product overall. Jeff

…

On Wed, Feb 15, 2017 at 12:21 PM, Elmer Thomas ***@***.***> wrote: Hello @jeffoneill <https://github.com/jeffoneill>, All of our research points to this being an issue on GMail's side. I recommend you reach out to their support team for further assistance. Please let us know if there is anything else we can help with. With Best Regards, Elmer — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#294 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKKSIACK4imcNBSRKJP-2WwxzpAJMk7ks5rczQXgaJpZM4L4lAk> .