Streams not recoverable when accounts are defined for a NATS JetStream server with previously existing streams

[pcsegal]

Observed behavior

If I have a NATS instance with existing streams and no accounts defined, and then I stop the server, change the configuration to define some accounts, then restart the server, I get a warning that the previously existing streams are not recoverable.

Expected behavior

In a previous version of NATS (2.9.6), the server would start without any error. This behavior happens in at least version 2.10.6.

Server and client version

Server version: 2.10.9
Client version: 0.1.4

Host environment

Ubuntu 22.04.1

Steps to reproduce

1. Run the test script given below. This script performs the following steps: 1) start a NATS server with no accounts defined, 2) create a stream, 3) stop the server, 4) change the server configuration to define some accounts and 5) restart the server.
2. Open the file generated by this script: /tmp/nats-accounts-issue/nats-with-accounts.log. This file shows the server logs after NATS was restarted in step 5 listed above. The relevant part of the server logs that I see when running this script locally is the following:

[29785] 2024/03/15 15:14:06.853400 [INF] Listening for client connections on 0.0.0.0:4222
[29785] 2024/03/15 15:14:06.853886 [INF] Server is ready
[29785] 2024/03/15 15:14:08.857491 [WRN] Healthcheck failed: "JetStream stream '$G > KV_example' could not be recovered"

Below is the test script and relevant files.

test.sh

#!/bin/bash

set -euo pipefail

# Clean up: make sure there are no NATS servers left over from a previous run of this script.

pkill -f -9 'nats.*nats-without-accounts.*' || true
pkill -f -9 'nats.*nats-with-accounts.*' || true

# Clean up data directories used in this example to make sure we start from scratch.

rm -rf /tmp/nats-accounts-issue || true

mkdir -p /tmp/nats-accounts-issue

# Start nats server without account configuration.

nats-server -c nats-without-accounts.conf > /tmp/nats-accounts-issue/nats-without-accounts.log 2>&1 &

sleep 2

# Wait for nats to be ready.

curl --fail --silent --retry 5 --retry-delay 1 http://localhost:8222/healthz > /dev/null

sleep 3

# Add a KV bucket.

nats -s localhost:4222 kv add example

# Shutdown NATS server.

pkill -f "nats.*nats-without-accounts.*" || true
sleep 5

# Start NATS server with some accounts configured.

nats-server -c nats-with-accounts.conf > /tmp/nats-accounts-issue/nats-with-accounts.log 2>&1 &

sleep 2

# Wait for nats to be ready.

curl --fail --silent --retry 5 --retry-delay 1 http://localhost:8222/healthz > /dev/null

nats-without-accounts.conf

server_name: nats
port: 4222
http_port: 8222

jetstream: {
  enabled: true
}

nats-with-accounts.conf

server_name: nats
port: 4222
http_port: 8222

jetstream: {
  enabled: true
}

accounts {
    $SYS: {
        users: [
            {user: admin, password: "password"}
        ]
    },
    "acc": {
        "jetstream": "enabled",
        "users": [
            {
                "pass": "password",
                "user": "acc"
            }
        ]
    },
}

[bjorndm]

Might it be that this is due to upgrading? Some care is needed it seems: 2.10 Upgrade Guide

[derekcollison]

JetStream assets are pinned to accounts. If you want to move them between accounts, start with your same config and create a backup / snapshot of the stream.

Then delete the stream (and all assets from the $G account).

Change configuration, and restore the snapshot into the new account.