KV create can fail with "wrong last sequence" on tombstoned entries

[davidmcote]

Observed behavior

As near as I can tell, all KV client libraries use similar logic for create(): update(expectedRevision = 0) || update(delete marker revision).

https://github.com/nats-io/nats.go/blob/main/jetstream/kv.go#L909-L928

// Create will add the key/value pair iff it does not exist.
func (kv *kvs) Create(ctx context.Context, key string, value []byte) (revision uint64, err error) {
	v, err := kv.Update(ctx, key, value, 0)
	if err == nil {
		return v, nil
	}

	if e, err := kv.get(ctx, key, kvLatestRevision); errors.Is(err, ErrKeyDeleted) {
		return kv.Update(ctx, key, value, e.Revision())
	}

	// Check if the expected last subject sequence is not zero which implies
	// the key already exists.
	if errors.Is(err, ErrKeyExists) {
		jserr := ErrKeyExists.(*jsError)
		return 0, fmt.Errorf("%w: %s", err, jserr.message)
	}

	return 0, err
}

Every call to KeyValue.delete() writes a new tombstone entry with a new revision number. If a call to delete() occurs between the check for ErrKeyDeleted finds a previously deleted key and the second call to kv.Update() above, the client gets a wrong last sequence error despite the key being tombstoned.

The JetStream protocol does not appear to have a mechanism for an atomic KV Create which avoids the client library needing to check for a delete marker.

Expected behavior

wrong last sequence is an unexpected failure for an application to receive when calling a KV method that does not have a kvLatestRevision parameter.

Server and client version

nats-server: 2.10.11
natscli: 0.0.35

Host environment

Mac 13.4.1 (22F82)
2.2 GHz 6-Core Intel Core i7
32 GB 2400 MHz DDR4
Intel UHD Graphics 630 1536 MB

Steps to reproduce

1. Launch nats-server -js
2. Create a KV bucket nats kv add mybucket
3. Start a client that rapidly writes tombstones: while true ; do nats kv del mybucket mykey -f ; done
4. Start a client that uses KV's atomic create:

$ while true ; do nats kv create mybucket mykey myvalue || break ; nats kv del mybucket mykey -f ; done
myvalue
myvalue
myvalue
myvalue
myvalue
myvalue
myvalue
myvalue
myvalue
nats: error: nats: wrong last sequence: 2083

[davidmcote]

FWIW, this scenario is somewhat contrived and not something my application has experienced.

I only noticed it through code inspection of NATS clients while contributing an unrelated patch.

[derekcollison]

I think this is more a client thing. Looping in @piotrpio and @Jarema