[QUERY] Message completion slower with AMQP vs NetMessaging transport

[jsquire]

Issue Transfer

This issue has been transferred from the Azure SDK for .NET repository, #40981.

Please be aware that @michaelmcmaster is the author of the original issue and include them for any questions or replies.

Details

Library name and version

Azure.Messaging.ServiceBus 7.16.2

Query/Question

Background

I have an application that makes heavy use of ServiceBus sessions. I'm updating the application's ServiceBus client library from WindowsAzure.ServiceBus to Azure.Messaging.ServiceBus and testing is showing a reduction (roughly half as many messages/second) in performance using the newer Azure.Messaging.ServiceBus client. After significant tinkering and benchmarking, it appears the performance reduction is related to the "transport type" being used by the client; the AMQP transport takes roughly twice as long to perform the message Complete operation.

This "slow completion" (aka. message settlement) behavior is alluded to in #13920 as well, but doesn't have any resolution. My goal is to get the Azure.Messaging.ServiceBus client to the same performance level as the older WindowsAzure.ServiceBus client, but all the experiments I've tried... aside from scaling out the number of concurrent sessions... have proved fruitless. I've tried client-per-thread (instead of singleton), I've tried a range of Prefetch (0 to 100, settled on 10), I've tried using the Processor (instead of Receiver), and I've tested just a single session w/single thread... across all the experiments, the client's Complete method is significantly slower when using AMQP transport.

Benchmark Scenario

Setup

• Topic (duplicate detection enabled, message ordering enabled)
• Subscription (requires session enabled)
• Publish 1000 messages (10 sessions with 100 messages each) into Topic

Client Pseudocode

• Construct a single Subscription Client
  • Use default client options except increase PrefetchCount to 10
• Construct 10 threads which do the following
  • Using the (singleton) subscription client, Accept a message session
  • Using the (accepted) message session, enter the receive loop:
    • Receive a single message (using PeekLock)
    • Process the message (simple deserialize + log)
    • Complete the message
    • Repeat the receive loop until Receive returns null (no message)
• Capture (via stopwatch) the execution time for every ServiceBus library call

Benchmark Results

WindowsAzure.ServiceBus 7.0.1

TransportType: NetMessaging (client default):

Completed [10] sessions containing [1000] messages in [00:00:05.722] (174.735 msg/sec)
Receive  - Minimum:[0.000 ms], Average:[4.496 ms], Maximum:[148.000] ms
Process  - Minimum:[0.000 ms], Average:[0.501 ms], Maximum:[51.000] ms
Complete - Minimum:[33.000 ms], Average:[51.870 ms], Maximum:[241.000] ms

TransportType: Amqp (set via connection string):

Completed [10] sessions containing [1000] messages in [00:00:12.478] (80.135 msg/sec)
Receive - Minimum:[0.000 ms], Average:[0.819 ms], Maximum:[52.000] ms
Process - Minimum:[0.000 ms], Average:[0.355 ms], Maximum:[35.000] ms
Complete - Minimum:[63.000 ms], Average:[123.271 ms], Maximum:[398.000] ms

Azure.Messaging.ServiceBus 7.16.2

TransportType: Amqp (client default):

Completed [10] sessions containing [1000] messages in [00:00:11.846] (84.410 msg/sec)
Receive - Minimum:[0.000 ms], Average:[0.991 ms], Maximum:[86.000] ms
Process - Minimum:[0.000 ms], Average:[0.612 ms], Maximum:[61.000] ms
Complete - Minimum:[38.000 ms], Average:[116.760 ms], Maximum:[259.000] ms

Observation

Comparing the NetMessaging transport (WindowsAzure.ServiceBus) against the Amqp transport (both old and new) libraries, the old NetMessaging Receive average is slightly slower... but the Complete is much faster.

Question

I've tuned the client's options within the constraints of the application's design. Is there anything that can be done to improve the Azure.Messaging.ServiceBus client's Complete performance?

Environment

Windows 10, .NET Framework 4.8.9181.0, x64

[jsquire]

//cc: @EldertGrootenboer

[EldertGrootenboer]

Thank you for your feedback. We have opened an investigation task for this in our backlog, and will update this issue when we have more information.

[EldertGrootenboer]

This item in our backlog, however we currently don't have an ETA on when development might start on this. For now, to help us give this the right priority, it would be helpful to see others vote and support this item.