ServiceBusReceiver receive_messages max_wait_time race condition locks the message but sends back empty list

[cabal-daniel]

• Package Name: pip install azure-servicebus

• Package Version: 7.12.2

• Operating System: MacOS and Ubuntu

• Python Version: 3.8.18

Describe the bug

Receiving code:

with sb_client.get_queue_receiver(queue) as receiver:
    messages = receiver.receive_messages(
        max_message_count=max_message_count,
        max_wait_time=max_wait_time,
    )
    print(messages)
    for message in messages:
           receiver.complete_message(message)

If max_wait_time is set in ServiceBusReceiver.receive_messages, then a race condition can happen where a message gets locked by the receive_message action but receive_messages returns back an empty list, locking this message for the duration of the lockout period.

To Reproduce
Steps to reproduce the behavior:

1. We used max_wait_time to 2 and are receiving 1 message at a time
2. Spin up 10 workers, constantly calling the Receiving code above in a polling interval in a while loop
3. Have a sender send 100 or so messages to the queue 1 at a time with a decently large body
   Actual behaviour: some of the messages will get locked in the queue. From our experiments roughly 2-4 messages out of 100 get put into the locked state without them being complete. The receive_messages method shouldn't lock a message unless it can return that message in full.

Expected behavior
All the messages will get completed. No message would get locked out.

I suspect the code for that behaviour is somewhere in here: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/servicebus/azure-servicebus/azure/servicebus/_servicebus_receiver.py#L454

Perhaps it can be configured so that max_time_out can allow a bit of "grace period" to allow the full message to be received and then terminate afterwards once the max_time_out is reached.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @EldertGrootenboer.

[kashifkhan]

Thank you for the feedback @cabal-daniel . We will investigate and get back to you asap.

[cabal-daniel]

Have you guys looked into this? It's happening in our prod environments a lot. We can't not set the max time out so messages get dropped

[cabal-daniel]

@rakshith91
@swathipilrakshith91 and swathipil

It seems like you guys are the original authours for this line. We are seeing frames being dropped by service bus. Could you please possibly comment on a remedy for this line?

https://github.com/Azure/azure-sdk-for-python/blob/d335a043d50cb3f4c484202dedf77bb[…]b4/sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/client.py