Nats-Expected-Stream header brakes cross domain mirrors

[andrewwebber]

Observed behavior

Given a hub and leaf setup, once a leaf has been established a local mirror of an hub stream, publishing a message with a header Nats-Expected-Stream results in a leaf delivery failure. The following message on the leaf can be observed.

[WRN] Error processing inbound mirror message for 'acc' > 'backup_simple': expected stream does not match

Expected behavior

When attempting to mirror or source streams across domains, permissions permitting, there should be no limitation on the messages delivered.

Server and client version

nats-server: 2.10.10
nats-cli: 0.1.3

Host environment

Local golang binaries on Arch

Steps to reproduce

Given the following hub configuration:

port: 4222
server_name: hub-server
jetstream {
    store_dir="./store_hub"
    domain=hub
}

leafnodes {
    port: 7422
}
accounts {
    acc: {
        users: [{user: acc, password: acc}],
        jetstream: enabled
    }
}

Given the following leaf configuration:

port: 4111
server_name: leaf-server
jetstream {
    store_dir="./store_leaf"
    domain=leaf
}

leafnodes {
    no_advertise: true
    remotes = [
        {
            urls: ["nats://acc:acc@0.0.0.0:7422"]
            account: "acc"
        }
    ]
}
accounts {
    acc: {
        users: [{user: acc, password: acc}],
        jetstream: enabled
    }
}

# start the server
nats-server -c ./server/server.conf --js
# start the leaf
nats-server -c ./leaf/leaf.conf --js
# add a stream to the hub
nats -s nats://acc:acc@localhost:4111 stream add simple --js-domain hub
# test publishing persistence
nats -s nats://acc:acc@localhost:4111 pub simple hello --js-domain hub
# establish a mirror from the hub to the leaf
nats -s nats://acc:acc@localhost:4111 stream add --mirror=simple --js-domain leaf backup_simple
# test the mirror is replicated messages
nats -s nats://acc:acc@localhost:4111 pub simple hello2 --js-domain hub
# observer mirror message count equal to the hub
nats -s nats://acc:acc@localhost:4111 stream ls --js-domain leaf
nats -s nats://acc:acc@localhost:4111 stream ls --js-domain hub
# publish a message with the troublesome header
nats -s nats://acc:acc@localhost:4111 pub simple hello3 -H Nats-Expected-Stream:simple --js-domain hub

leaf now outputs to console [WRN] Error processing inbound mirror message for 'acc' > 'backup_simple': expected stream does not match

# observer mirror message count is **not equal** to the hub
nats -s nats://acc:acc@localhost:4111 stream ls --js-domain leaf
nats -s nats://acc:acc@localhost:4111 stream ls --js-domain hub
# publish a new message without the header again
nats -s nats://acc:acc@localhost:4111 pub simple hello4 --js-domain hub
# observer mirror message count has increased, only one message was dropped
nats -s nats://acc:acc@localhost:4111 stream ls --js-domain leaf
nats -s nats://acc:acc@localhost:4111 stream ls --js-domain hub

Work around

Simply make sure the stream name on the leaf matches the upstream stream name in the hub

Risks

With the above work around I would expect more advanced sourcing setups like multiple streams with difference names would eventually hit the same issue.

[andrewwebber]

I forgot to add but I was not able to test if with other headers related to Nats-Expected-Stream would also break mirroring:

• Nats-Expected-Last-Msg-Id
• Nats-Expected-Last-Sequence
• Nats-Expected-Last-Subject-Sequence

https://docs.nats.io/nats-concepts/jetstream/headers#publish