Cluster size scalability question

[melaraj2]

I see references to "a recommended cluster size" in the documentation. This statement is incongruent with true horizontal scalability. If you can't make a cluster of whatever size, then there is really no horizontal scalability. I understand the need to keep jetstream streams to 3 replicas, but does that really have anything to do with the overall size of the cluster? can we make a 10-server cluster, but make a policy to create streams that only live in 3 cluster members?

Along the same lines of scalability, are there any plans for jetstream stream partitioning?

[derekcollison]

Yes clusters themselves can be any size, although if they get too big we might recommend a super cluster (many clusters). This is how we setup NGS ourselves.

In terms of JetStream partitioning, we have the ability to map subjects into partitioned subjects transparently today, so you can utilize multiple streams that are partitioned.

We have also been considering segmented streams.

And lastly, if you are using JetStream as a KV, when you do a get from a KV any mirrors will also transparently share the load as of 2.9.

[melaraj2]

Excellent, thank you. I think the documentation states that only 3 or 5 jetstreams-enabled servers should be in a cluster. Perhaps, it should say that when creating a stream, it should have 3 or 5 replicas. I have a cluster with 6 nat-servers, they are all identical, and all have jetstream enabled. I want to make sure this is ok for production.

Also, would Synadia offer a managed clusters product in the future? Similar to confluent and Kafka?

[derekcollison]

Yes that might be a better way to describe it. Most CPs only have 3 AZs which define availability, so having R5 makes no sense in that situation really.

Synadia offers NGS and of course you can extend NGS with your own servers, which can run JetStream. We are also starting to support managed clusters in customers CP resources.

[kruegernet]

@derekcollison What is the status of thought on segmented streams today?

[derekcollison]

Still on the fuzzy roadmap, with 2.11 there will be a new filestore impl that will allow us to get to PB. Right now we are good IMO generally for large streams in TBs with reasonable subject cardinality and interior deletes. I am currently working on the subject cardinality problem.

Once those sort themselves out I think we will revisit in earnest the concept of segmented streams.

[kruegernet]

Thanks for the update. I ask next because I believe it's related for a lot of users to segmenting: is there a plan to allow changes to partition counts of subject partitioning via the or API from client code rather than involving server reloads of conf?

[derekcollison]

Not following. JetStream is a system concept, not an individual server one per se. Yes the server operator gives some broad parameters on where to store, and how much etc.

[kruegernet]

Sorry, I believe I was misunderstanding the docs. Clearly if I can do:
nats server mapping "foo.*.*" "foo.{{wildcard(1)}}.{{wildcard(2)}}.{{partition(3,1,2)}}" from CLI, then I don't need to update server conf files and cause all the cluster nodes to reload to affect changes here to the partition count.

[derekcollison]

Ah apologies you are talking about subject mappings and partitioning. These can be in account JWTs to in decentralized config mode. So you just update the account JWT and push to the system.

[ripienaar]

And also per stream config since 2.10

[jnmoyne]

To resume what has been said and explain a bit more:

As of 2.10 there are now two places where you can do "partitioning" in NATS: you can do subject transformation as part of streams which allows you to do partitioning for the consumers of the stream (which is the functionality most people are looking for). And you can change the number of partitions at any time by updating the stream's config. In the more rare cases where you want to partition a message flow into partitioned streams (plural) then you need to do Core NATS subject transformation which if you do not want it to be part of the server config you can do with the operator JWT-based security model where the subject transforms are part of the account JWT and as Derek mentioned you can update the mapping at any time by generating a new JWT and pushing it to the resolver.

[bjorndm]

How many KV buckets can one node reasonably support? And one NATS network? For my use case, which is storing private data, I'm testing and would like to create 1 million buckets. My idea is to give each user their own private bucket.

But on MacOS but around 20000 KV buckets I'm getting some performance problems on the creation of the buckets. Also, the startup is very slow with 20000 buckets, because the buckets are restored in series. Arguably they should be restored concurrently, 100 buckets at the time or so.

[derekcollison]

Keys can contain multiple tokens. So you can put many users in a single KV with proper permissions (we know we need to make this easier, but it is possible today).

uuid.>

[bjorndm]

Yes , that could also work, but I have some problems with the permissions for such an approach. So I filed #5204