WIP: prototype new spec for messaging systems

[blumamir]

Prototyping amqplib (rabbitmq)

General

• The library works only in “Push” model (TCP socket to the server with callback). No ambient context.
• “deliver” span will always be the root span
• Always consume a single message. No batch.
• Scheduling messages is available with a plugin. Not native to the broker.
• Current implementation (prior to this PR) is using parent semantics (single “process” span per message, child of the “send” span)
• The same message can be consumed from the same queue more than once in case of nack \ timeout etc.
• A single message can be consumed from multiple, unbounded different queues (topic, fanout)

Routing Combinations

• Rabbitmq always “publish” to exchange and “consume” from queue.

Exchange is the pubsub

• An empty value means default exchange (direct)
• 4 exchange types: direct, fanout, topic, headers
• Always interesting

Queue is what consumers receive messages from

• Can be a user-supplied string “my-queue” - interesting
• Can be a “temporary queue” with a random name generated in the server. Not very interesting. Not always possible to tell if a queue is temporary or not.

Routing key is set in send and is used to forward messages from exchange to queue

• Fanout - not used (user can still, however, fill any value in it)
• Direct - match value given in bind
• Topic - routing key is topic in form “foo.bar.baz” and queues subscribe with “.bar.”.

Exchange name - known to instrumentation
Routing key - known to instrumentation
Exchange type - not known to instrumentation
Binding pattern - not known to instrumentation
Queue name - known to instrumentation.

Routing Scenarios

• Consume from queue on default exchange (this is equivalent to sendToQueue)
• Consume from queue on direct non-default exchange (can’t tell this is the case for sure. Can guess if queue name == routing key)
• Consume from queue on fanout exchange (can’t tell for sure. Can guess if routing key is an empty string). The queue name can be temporary
• Consume from queue on topic exchange (no deterministic way to know this is the case, can fallback to this option in case other options are rolled out).

Changes

• Consumer side span - always add link and don’t use propagated context as parent
• Add link attribute “messaging.operation”: “send” to give context to the link.
• Change messaging operation from “process” to “deliver”
• Move all consume message specific attributes to be link attributes:
  “messagin.rabbitmq.exchange_name” (new attribute)
  “messagin.rabbitmq.routing_key”
  “messaging.message_id”
  “messaging.conversation_id”
• Destination and destination kind:
  For consumer - change to queue semantics
  For producer - destination is exchange name and type is topic.
• Span name:
  Relax the current specification requirement
  Include only exchange name in publish span (assuming it’s low cardinality)
  Replace “send” in publish span to “publish” to make it consistent with the library’s API.
  Replace “process” with “deliver” in consumer name
• End deliver span as soon as the consume callback returns (either promise or non-promise)
• Add settlement span:
  “messaging.operation”: “settle”
  Kind = client
  Add 2 links per message pointing to “send” and “deliver” contexts. Add message attribute “messaging.operation” to distinguish.
  Add all message attributes from “deliver” to the “deliver” link on “settlement”
  Set span to error if settlement is “reject”, “nack”, “nackAll”, “channelClose”, “channelError”
  Quietly settle messages that timeout after enough time

Open Questions

• Which attributes should go into the link attributes?
• Should we additionally store these attributes on the span itself?
  Symmetric with publish
  Feasible for rabbit, not in other clients
  No good support in jaeger and vendors
• Create link even if no context is available? (my opinion: yes)
• Does it makes sense to include parent propagation style? How will it behave in case of fanout/topic? How will it behave with scheduling message plugin?
• What should be the destination attribute for producer? Exchange name or routing key? Kind “topic” is not very suitable to exchange name
• Should instrumentation include specific semantics for “default exchange” (e.g. “sendToQueue”)?
• “Create” span doesn’t feel right. Should it be joined with the “publish” span on single message publish?
• Should we include “messaging.operation” in publish span (“send”)?
• Do we need to signal in some way that the deliver is for a single message and not batch with one message?
• Settlement:
  Add requeue attribute?
  Add “allUpTo” attribute?
  Semantic attribute for settlement type (ack, nack, reject, ackAll, nackAll, channelClose, channelError)?
  Semantic meaning to the error in this case (reject)