Getting the total number of messages in a stream

[tinakurian-pki]

Hello!

I was wondering, is there a way to get the total number of messages in a stream at any given time? The reason being is that i'd like to eventually offer a rest endpoint for a UI to seek through the stream using pagination. Currently, i've followed the example and set the messageCount to 1_000_000 and a timeout of 1 minutes. However, this seems less than ideal for a rest endpoint and seeking through the stream. The number of messages being published to the stream is unknown, as it is tracking audit information.

So because the number of messages being published is unknown, 1_000_000 is just a wild guess and i just have to hope we never exceed that value. Setting it to more or increasing the timeout is definitely an option, however, since we are using REST, we'd like the endpoint to be fairly responsive.

If there is no way to get the message count, is there another way to do this that is more accurate? (get the current total of messages (at any given time) and use that, or more reliably set the timeout so i'm not waiting unnecessarily?)

Thanks in advance.

[acogoluegnes]

It's not possible using the stream Java client, but you can use the /api/queues/{vhost}/{stream-name} REST endpoint to get an estimation of the total number of messages.

E.g. for a stream stream in the / virtual host, it is http://localhost:15672/api/queues/%2F/stream and the estimated number of messages is in the messages property:

{
  "garbage_collection": {
    "max_heap_size": -1,
    "min_bin_vheap_size": -1,
    "min_heap_size": -1,
    "fullsweep_after": -1,
    "minor_gcs": -1
  },
  "consumer_details": [
    {
      "arguments": {},
      "channel_details": {
        "connection_name": "127.0.0.1:55370 -> 127.0.1.1:5552",
        "name": "",
        "node": "rabbit@acogoluegnes-nuc",
        "number": 0,
        "peer_host": "127.0.0.1",
        "peer_port": 55370,
        "user": "guest"
      },
      "ack_required": false,
      "active": true,
      "activity_status": "up",
      "consumer_tag": "stream.subid-0",
      "exclusive": false,
      "prefetch_count": 0,
      "queue": {
        "name": "stream",
        "vhost": "/"
      }
    }
  ],
  "arguments": {
    "x-max-length-bytes": 20000000000,
    "x-queue-leader-locator": "least-leaders",
    "x-queue-type": "stream",
    "x-stream-max-segment-size-bytes": 500000000
  },
  "auto_delete": false,
  "consumers": 1,
  "deliveries": [],
  "durable": true,
  "effective_policy_definition": {},
  "exclusive": false,
  "incoming": [],
  "members": [
    "rabbit@acogoluegnes-nuc"
  ],
  "memory": 9216,
  "messages": 21943,
  "messages_details": {
    "rate": 100.4
  },
  "messages_ready": 21943,
  "messages_ready_details": {
    "rate": 100.4
  },
  "messages_unacknowledged": 0,
  "messages_unacknowledged_details": {
    "rate": 0
  },
  "name": "stream",
  "node": "rabbit@acogoluegnes-nuc",
  "online": [
    "rabbit@acogoluegnes-nuc"
  ],
  "operator_policy": null,
  "policy": null,
  "readers": {
    "rabbit@acogoluegnes-nuc": 1
  },
  "reductions": 0,
  "reductions_details": {
    "rate": 0
  },
  "state": "running",
  "type": "stream",
  "vhost": "/"
}

Again, this is just an estimation.

We're also working on a protocol command that would return some information on a stream, something along the way of StreamMetadata Environment#metadata(String stream), with information like first / last offsets, and the estimated number of messages. Applications could call from time to time, but it would not be meant to be called repeatedly, like in the message handler callbacks. Is that something that could work for you?

[tinakurian-pki]

Thanks for the speedy response! That makes sense. I think I have an idea on how I could use the estimation. The protocol command would be great to have. For now, i'll try out the rest endpoint! Thanks for your help!