Consumer and Stream metadata internal fields

[Jarema]

Overview

ADR-33 https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-33.md introduced metadata fields for consumers and stream.

Users can use those, but we also reserved internal prefixes:

_nats

format

_nats_field_about
or
_nats.field.about

The latter one is closer to how NATS subjects are structured.

prefixes

I think it would be good to have subject-like structure:

_nats.server.created.version
_nats.client.created.library

I assume past tense when viable.

list of internal keys

This thread discusses all fields set by server and client.
After gathering the list, ADR-33 will be updated.

I've included some rough ideas below.
Please add more.

The list will be updated.

Server

• _nats.server.created.version - immutable, specifying the server version that created the resource.
• _nats.server.updated.version - server version that executed the last update
• _nats.server.updated.time - when resource was last time updated
• _nats.server.kind - type/kind/purpose of the resource - kv, object store, mqtt etc.

Client

• _nats.client.created.library - library language
• _nats.client.created.version - library language version
• _nats.client.updated.library - last updated by library
• _nats.client.updated.version - library language version

[ripienaar]

+1 for . seperator.

It would be nice if the server maintained like _nats.server.revision that it increments on each edit?

[wallyqs]

yes revision sounds like a great idea, not sure if server already tracks this internally

[scottf]

? _nats.server.created.time

[tbeets]

{{.ServerId}} (i.e. capture the particular server id that is adding or updating the metadata...)

[ripienaar]

I find serverid generally not that useful since it canges every start, in a week the value of that field is unusable essentially

[tbeets]

TBH, just wanted to field idea of some server-provided metadata dynamics (with possible mustache notation) rather than the specifics.

[ripienaar]

I do like the template idea against server info

[Jarema]

I also like the templating idea, but at the same time, I assume that all metadata with _nats.server are handler server-side, not passed by a client.

[ColinSullivan1]

The user that last edited the object - NKEY, id, or DN - could be useful. Although it would leak user IDs so there's a potential security concern.

[ripienaar]

We can get it now, but yes think I'd rather not put those in the API where potentially other accounts can get them vie API imports/exports

[wallyqs]

server version when created, edited time and revision number would be great. Also have a way to represent the original intention of the stream like if mqtt, kv, object, etc...

[Jarema]

good idea!
added kind.
Not sure about the name though.

[MauriceVanVeen]

Really cool!
I think also related to this discussion we had a few weeks ago: #190

It's not explicitly part of the ADR, but will the metadata also be exposed in things like Prometheus metrics, to display relevant metadata in graphs for example?

[ripienaar]

I wouldn't imagine exposing all metadata to Prometheus would be a good idea, but I imagine we could have something like prometheus.label.* kind of convention with Surveyor and Prom exporter only exposing those.

[ripienaar]

Since adding metadata to micro some of these seem a bit weird, its not edited, no point in kind etc. So maybe something to consider.

[Jarema]

I agree. I think kind is relevant on stream metadata and maybe consumer. Maybe.

[ripienaar]

I've done a bit of this on a branch in the CLI and just want to be sure we're on the same page.

$ nats s edit X --replicas 3 -f
...
Metadata:

  _nats.client.created.library: natscli
  _nats.client.created.version: development
  _nats.client.updated.library: natscli
     _nats.client.updated.time: 2023-10-25T14:43:33Z
  _nats.client.updated.version: development

It's quite verbose, and would be even more when we add server ones. So I am slightly concerned about the noise this generates UX wise but otherwise it would be useful? I might hide these when rendering basic info but include with -v or something