Implement Local Cache for ConfigStore

[rannn505]

Suggestion

We propose the integration of a local caching mechanism for ConfigStore which will significantly enhance performance by reducing fetch times and providing a fallback during outages. This cache should be configurable via the .configu file, allowing users to toggle caching on or off, or specify a custom SQLite file for the cache data.

• Add an optional string cache property to root of the .configu file.
• Add an optional boolean cache property under stores.[storeName].cache of the .configu file.
• Enable local cache if one of the stores.[storeName].cache props are true. -> this enables the cache for that spesific store.
• If there arent any stores with cache: true, disable cache.
• if a custom path string is provided to the root cache prop -> use the specified SQLite file otherwise use the default cache path at $cacheDir/configs.sqlite:
• Automatically save to cache after each successful eval command execution involving that ConfigStore.
• In case of a ConfigStore fetch error during eval (not an actual eval error like validate the cfgu against a Config), attempt to fetch the Config from the local cache if caching is enabled.
• Add a --force-cache flag to the eval CLI command to allow fetching Configs strictly from the cache.

Motivation

Introducing a local cache for ConfigStore aims to:

• Minimize latency during configuration fetch operations.
• Provide resilience against temporary ConfigStore outages or network issues.
• Streamline CI/CD processes by allowing cache artifacts to be saved and reused, improving build times and reliability.

Context

Additional considerations include:

• Ensuring cache integrity and freshness—cache should be updated or invalidated based on configurable criteria.
• Testing within CI environments like GitHub Actions to confirm the cache's effectiveness as a stored artifact and its reusability across multiple build runs.
• Establishing a clear cache management strategy for users to understand when and how cache is utilized or ignored.

[rannn505]

DRAFT

Suggestion

Creates local sqllite
Controlled from . configu - path to db file
CI ONLY?
Offering for Platform? Cache feat?

Motivation

No response

Context

No response

[RonConfigu]

Development Plan

• Modify SQLiteConfigStore so that the table is dynamic
• Add cache?: boolean | string to BaseConfig
• Create getCacheStoreInstanceByStoreFlag to BaseCommand. Adding to base and not to eval for future uses in cache in other commands.
• Create class ConfigStoreError extends ConfigError {} to identify errors from the store
• Modify ORMConfigStore to except tableName in params
• Modify SQLiteConfigStore to except tableName in params
• Modify EvalCommand to throw ConfigStoreError where:
  • store.init() throws
  • any error inside evalFromStoreSet
  • validateResult throws and the origin is EvaluatedConfigOrigin.StoreSet
• Implement caching and fetch from cache in cli eval
• Add --force-cache to cli eval
• Create Github action
• Update docs (if necessary)

Questions / Requests

Notes / Comments / Additional

[rannn505]

cache on the main base-config with the file path and cache on each store with enable disable for it.

if any store is "cache enabled" and the file path is not defined use the default one.
I will fine tune the ticket now so it reflects the final product request

[rannn505]

please attach the .configu, .cfgu and GH action workflows file of the test you created here @RonConfigu

[RichardAkman]

An initial implementation of this feature has been implemented and released to both @configu/node & @configu/cli. The implementation is currently missing the cache fallback requirement from this issue due to a concern raised during the product review where authentication errors also caused cache fallbacks that go against the product vision. Masking authentication errors should be allowed only if the user explicitly asks for it. Until we come back to the issue, we have disabled the fallback mechanism on the SDK level. Once we determine the definitions of authentication errors on a store-level, we will return to this issue and implement the requirements fully.

[RonConfigu]

please attach the .configu, .cfgu and GH action workflows file of the test you created here @RonConfigu

@rannn505 @RichardAkman
Sure, files with txt because GH doesn't allow these file types
dot-configu.txt
cache.cfgu.json
gh.yaml.txt