Changing MutableState cache to host-level by default #5894
Conversation
I am explicitly setting cli dev server cache size to be 8k in dynamic config yaml file: https://github.com/temporalio/temporal/pull/5894/files#diff-18c8bb8717f6896034d89c0eb07448ffce8fe45aeb752362b798ce18ef22e92f This should limit the cache size in dev server. |
The cli ( I'd also recommend trying to keep those files as short as possible. In local testing we should run as close to the default as possible. |
Right! Default host level cache size is 128K entries. That is very large for local clusters. Previously it was per-shard and cli server only has 1 shard. Do you think if setting this dynamic config key(cache size) in cli a good idea? |
## What was changed Setting the size of host level mutable state cache to 8k. ## Why? There is a pending change is server repo(temporalio/temporal#5894) that will enable host-level mutable state cache. This cache was shard-level with a size of 512. Cli server will only have one shard. Host level cache has a default size of 128k to accomodate for all shards. But this size is too big a cluster with 1 shard. Setting this size to more reasonable 8k for server started from cli command. ## Checklist 1. Closes 2. How was this tested: 3. Any docs updates needed?
|
temporalio/cli#574 |
What changed?
We had added a configuration flag to make MutableState cache a host-level cache.
Changing dynamic config to enable this flag by default and adjusting cache size to 128K. We found that this size is appropriate for most workloads we see.
Also setting host-level cache size to be 8K in development dynamic config.
Why?
Host level cache is better for cache utilization and it prevents the problem of a smaller shard-level cache getting full with pinned elements.
How did you test it?
Running the cluster with load.
Potential risks
None
Documentation
Is hotfix candidate?
No