Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow publishers to specify a different subject for "Nats-Expected-Last-Subject-Sequence" #5280

Open
cchamplin opened this issue Apr 7, 2024 · 5 comments · May be fixed by #5281
Open

Allow publishers to specify a different subject for "Nats-Expected-Last-Subject-Sequence" #5280

cchamplin opened this issue Apr 7, 2024 · 5 comments · May be fixed by #5281
Labels
proposal Enhancement idea or proposal

Comments

@cchamplin
Copy link

cchamplin commented Apr 7, 2024 •
edited

Proposed change

I'd like to request a change to allow publishers to provide a specific subject for use when nats-server enforced "Nats-Expected-Last-Subject-Sequence". Adding an additional header field that allows publishers to optionally change the subject used for "Nats-Expected-Last-Subject-Sequence" from the subject of the message being published to to a subject provided in the header.

The proposal is for a "Nats-Expected-Last-Subject-Sequence-Subject" header that when set with a string value becomes the subject used during enforcement of the last sequence. The header does not necessarily need to be that string exactly, the functionality is the actual proposal.

Use case

The specific use case I am trying to solve here is to allow publishing to complex and dynamic subjects while still enabling optimistic concurrency control.

For example in an event sourcing application an aggregate might be defined against the subject:

  • orders.[order_id]
    • orders.1
    • orders.2
    • orders.3
    • ...

Today we can use "Nats-Expected-Last-Subject-Sequence" to ensure that when we publish to orders.2 we are publishing against the known last sequence for messages on the orders.2 subject.

However, it is often desirable to have more complex subject patterns. For example image a pattern that looks like this:

  • orders.[order_id].[event_type].[visibility]
    • orders.1.created.public
    • orders.1.approved.private
    • orders.2.created.public
    • ...

Today we cannot use "Nats-Expected-Last-Subject-Sequence" with this pattern and still have OCC for the aggregate orders.2 when publishing. If a publisher attempts to set the expected sequence and then publish to order.2.rejected.private the sequence for that subject is likely 0 as opposed to sequence for the last event published to order.2.>

By allowing publishers to override and customize the subject used in the expect last subject sequence check OCC can be enforced for the aggregate itself versus the subject being published to. For example by setting the proposed "Nats-Expected-Last-Subject-Sequence-Subject" to "order.2.>"

Relevant Slack Conversations:

Contribution

Possibly
@cchamplin cchamplin added the proposal Enhancement idea or proposal label Apr 7, 2024
cchamplin added a commit to cchamplin/nats-server that referenced this issue Apr 7, 2024
@cchamplin
[ADDED] Publish header "Nats-Expected-Last-Subject-Sequence-Subject"
f5d181d 
This change adds a new header
"Nats-Expected-Last-Subject-Sequence-Subject" when when paired with
"Nats-Expected-Last-Subject-Sequence" allows publishers to customize the
subject used when the server enforces
"Nats-Expected-Last-Subject-Sequence". Publishers can specify a
alternative subject to be used that includes wildcards.

Resolves nats-io#5280

Signed-off-by: Caleb Champlin <caleb.champlin@gmail.com>
This was referenced Apr 7, 2024
Open
Open
@ripienaar
Copy link
Contributor

Looping in some people who know more about event sourcing than me @bruth @codegangsta

@cchamplin
Copy link
Author

If we can get this change accepted or have some discussion around it we'd also like to propose an additional set of headers:

	JSExpectedLastSubjHeader     = "Nats-Expected-Last-Subject-Header"
	JSExpectedLastSubjHeaderSubj = "Nats-Expected-Last-Subject-Header-Subject"

That would allow publishers to specify an expected header and value on the last message for a message subject or an overriding subject.

For example Nats-Expected-Last-Subject-Header: "Aggregate-Version=10"

In this case nats server would grab the last message for the subject on the stream, extract the "Aggregate-Version" header from that message and compare it against 10 and rejecting the publish accordingly. This I think offers the most flexibility and would allow event sourcing implementations to leverage their own versioning for an aggregate root rather than worrying about the intricacies of stream sequences, especially since my understanding is that sequences can potentially change when events are mirrored or sources which makes them pretty undesirable as a source of truth for a version or sequence number in an event sources aggregate.

Flexible on the exact details, pattern, and names of things here, but hoping the idea gets some traction.

@derekcollison
Copy link
Member

Mirrors are bit for bit copies, so sequence and timestamp stay the same. For sources they change, but we preserve the originals in headers.

@bruth
Copy link
Member

bruth commented Apr 9, 2024

@cchamplin This is a great addition and straightforward PR. I will want to play with this a bit to see how much this could be abused (subject length/number of tokens), but I am very supportive of this. Solves a very concrete problem for ES.

cchamplin added a commit to cchamplin/nats-server that referenced this issue Apr 10, 2024
@cchamplin
[ADDED] Publish header "Nats-Expected-Last-Subject-Sequence-Subject"
e9b1b36 
This change adds a new header
"Nats-Expected-Last-Subject-Sequence-Subject" when when paired with
"Nats-Expected-Last-Subject-Sequence" allows publishers to customize the
subject used when the server enforces
"Nats-Expected-Last-Subject-Sequence". Publishers can specify a
alternative subject to be used that includes wildcards.

Resolves nats-io#5280

Signed-off-by: Caleb Champlin <caleb.champlin@gmail.com>
@codegangsta
Copy link
Contributor

Yeah I really like this addition. like @bruth I'd like the opportunity to play with it this week

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
proposal Enhancement idea or proposal
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ADDED] Publish header "Nats-Expected-Last-Subject-Sequence-Subject" cchamplin/nats-server
5 participants
@ripienaar @derekcollison @codegangsta @bruth @cchamplin