Keyv v5 - Features and Release Plans 🎉

[jaredwray]

Hello Everyone👋,

We're excited to bring you some updates about the forthcoming Keyv version 5, slated to be released in the coming months. While this update will introduce some breaking changes, it also offers a plethora of new features designed to improve your experience. With an ever-present commitment to maintaining the highest possible compatibility with the Map API and facilitating the use of many backend storage adapters, we're eager to unveil what's in store.

Don't hesitate to get in touch if you have any questions concerns - you can reach us right here or at me@jaredwray.com.

New Features and Progress:

• Migration to Typescript and Core Types via Keyv
• Keyv Hooks
• Built-in Storage Tiering
• Statistics
• Async / Await in TypeScript instead of Promise
• Full Control over Name Spacing
• Breaking Change: URI Breaking Changes on Constructor
• Breaking Change: Simplified Compression Adapters

Here's a quick rundown of the new built-in functions and compatibility enhancements:

Compatibility with browsers

We are working to support Keyv the main module as something that can work in the browser. There is more details on this but the just is swapping out buffer and event handling. Read more here: #252

URI and Proposed Breaking Change 🔥

After feedback from the community and looking into this more (thanks @hugo082) we have decided to longer support the uri on the constructor. This would be a major breaking change 😬 but we believe everything else will be worth it.

import KeyvMemcache from '@keyv/memcache';
import Keyv from 'keyv';

// don't work anymore
const keyv = new Keyv('memcache://user:pass@localhost:11211');

// works
const keyv = new Keyv({ store: new KeyvMemcache('memcache://user:pass@localhost:11211') });

// proposals moving forward. 
const keyv = new Keyv(new KeyvMemcache('memcache://user:pass@localhost:11211'));
const keyv = new Keyv(KeyvMemcache, 'memcache://user:pass@localhost:11211');
const keyv = new Keyv({ store: new KeyvMemcache('memcache://user:pass@localhost:11211') });

Async / Await in TypeScript

As we transition to TypeScript, we are moving away from the Promise infrastructure to make the code more readable and accessible.

Built-in Offline Mode

If you select a storage adapter and set the offline mode to true, Keyv will default to in-memory if your storage adapter is offline.

import Keyv from 'keyv';

const keyv = new Keyv('redis://user:pass@localhost:6379', {offline: true});
keyv.on('storage_adapter_state_change', handleStorageAdapterStateChange);

Built-in Storage Tiering

By default, you can now establish a tiered storage architecture (e.g., primary and secondary) without using @keyv/tiered.

import Keyv from 'keyv';

const keyv = new Keyv({primaryStorage: new Map(), secondaryStorage: 'redis://user:pass@localhost:6379'});

Simplified Compression Adapters

Going forward, all compression adapters will only require the compress() and decompress() functions.

Integrated Types for Compression and Storage Adapters

With integrated type definitions for Compression and Storage Adapters, we're fostering a more robust third-party storage and compression adapter ecosystem.

Full Control over Name Spacing

This update gives you unprecedented control over name spacing. You can now define the default, add complexity via a function, or remove it entirely.

import Keyv from 'keyv';

//example of setting the default
const keyv = new Keyv({namespace: fooNameSpace});
console.log(keyv.namespace); //returns `fooNameSpace`

//example of a complex namespace via function
const keyv = new Keyv({namespaceHandler: (namespace, key, value, options) => {
    return `${namespace:key}`; //return a string of whatever you want
} });
console.log(keyv.namespace); //returns `fooNameSpace`

//example of removing it all together resulting in no more prefix and also telling other storage adapters it is not valid. This will also be a breaking change for non-compatible storage adapters that have not updated. 
const keyv = new Keyv({namespace: undefined});
console.log(keyv.namespace); //returns undefined

Hooks Included

You will be able to add hooks to our core functions before and after execution:

export enum KeyvHooks {
	PRE_SET = 'preSet',
	POST_SET = 'postSet',
	PRE_SET_MANY = 'preSetMany',
	POST_SET_MANY = 'postSetMany',
	PRE_GET = 'preGet',
	POST_GET = 'postGet',
	PRE_GET_MANY = 'preGetMany',
	POST_GET_MANY = 'postGetMany',
	PRE_DELETE = 'preDelete',
	POST_DELETE = 'postDelete',
	PRE_CLEAR = 'preClear',
	POST_CLEAR = 'postClear',
}

more details to come on this.

Statistics

We will be adding more information around this but Keyv will have the ability to do stats and give you access to them. By default it will be turned off.

Example:

const keyv = new Keyv({ stats: true});
keyv.set('foo', 'bar');
console.log(keyv.stats.data.sets); // this will equal 1

Stay tuned for more updates as we approach the release of Keyv version 5. Your continued support and feedback are invaluable to us, and we look forward to unveiling more features and improvements.

[hugo082]

Hi,

Thanks for the update!

With the current version, there is an import design issue that breaks the compatibility with bundlers (used by Next.js even for backend environments). please refer to #45 for more details.

To summarize, here is the part of the code that is responsible of the compatibility break. We are using require(adapters[adapter]) which is a require with expression and produces the following warning: the request of a dependency is an expression.

Is it possible to upgrade the v5 to be compatible with bundlers and remove this part of the code? It will produce a breaking change similar to:

import KeyvMemcache from '@keyv/memcache';
import Keyv from 'keyv';

// don't work anymore
const keyv = new Keyv('memcache://user:pass@localhost:11211');

// works
const keyv = new Keyv({ store: new KeyvMemcache('memcache://user:pass@localhost:11211') });

// proposals
const keyv = new Keyv(new KeyvMemcache('memcache://user:pass@localhost:11211'));
const keyv = new Keyv(KeyvMemcache, 'memcache://user:pass@localhost:11211');

[jaredwray]

@hugo082 - thanks and yes we are highly considering to move to a more stable model instead of using require. Will update as we get closer to alpha versions in the next couple weeks.

[lukechilds]

Awesome progress, looking forward to seeing v5!

[jaredwray]

Awesome progress, looking forward to seeing v5!

@lukechilds - thanks so much one thing we will be discussing is if we want to go with the proposal as the require() functionality is going to cause issues with other libraries. Thoughts?

[jaredwray]

@hugo082 - updated with proposal

[jaredwray]

Added in information around hooks and statistics.

[lukechilds]

@lukechilds - thanks so much one thing we will be discussing is if we want to go with the proposal as the require() functionality is going to cause issues with other libraries. Thoughts?

Yeah if you wanna support bundling / frontend then I think probs best to remove dynamic require. I do think inferring deps via URI is pretty slick API design if it's possible to keep though. Potentially there could be some workaround that keeps the current behaviour without doing a dynamic require. Maybe some kind of shim that statically requires each dep on a best effort basis like:

storage-adaptors.js

try {
  module.exports.sqlite = require('sqlite')
} catch {}

try {
  module.exports.redis = require('redis')
} catch {}

then you could require the shim at runtime without error, check which storage adaptor you want to use, and return an error if it's undefined.

Potentially that could still cause issues with bundlers if they can't find redis at bundle time or whatever I'm not sure. Maybe defining them as peer deps could help? Could be worth investigating if you want to keep the URI API though.

[lukechilds]

But yeah if it's gonna be too much of a headache or isn't possible I think only accepting a storage adapter instance makes sense like:

const keyv = new Keyv({ store: storageAdaptor })

[hugo082]

Potentially that could still cause issues with bundlers if they can't find redis at bundle time or whatever I'm not sure. Maybe defining them as peer deps could help? Could be worth investigating if you want to keep the URI API though.

The main issue I see with the try / catch approach is that the bundle cannot statically check so will try to bundle all dependencies (redis + sqlite).
There is maybe a solution to keep the same API by using TS Template literal types (which was introduced in typescript 4.1). I'm not sure it works but it may looks like

type SQLiteURI = `sqlite://${string}`;
type RedisURI = `redis://${string}`;
type URI = SQLiteURI | RedisURI;

const isRedisUri = (uri: URI): uri is RedisURI => uri.startsWith('redis://');
const isSQLiteUri = (uri: URI): uri is SQLiteURI => uri.startsWith('sqlite://');

const uri = 'redis://...'; // <= as to be a literal type, not only string

const clientFromUri = async (uri: URI) => {
  if (isRedisUri(uri)) {
    return await import('sqlite')
  } else if (isSQLiteUri(uri)) {
    return await import('redis')
  }

  assertType<never>(uri);
};

But it will still create a breaking change as the uri property as to be a literal type URI, not a string

[jaredwray]

Hi All - I wanted to update you on the progress leading up to v5 release. We have now split the repo into two branches:

• main - this is now the v5 branch and we will be adding to it there. We do plan to put a link to the v4 branch.
• v4 - this is now the v4 branch and the goal is to maintain this through 2024.

The current branch is now completely in typescript and also has updated types. Our next step is to work on publishing the v5 alpha version for people to start testing and then roll out the rest of the features hooks, stats, browser compatibility, compression fixes, offline / tiering mode (multi storage adapters), and namespace enhancements.

[jaredwray]

Hi All - Keyv Hooks are now up and running and we will be releasing v5.0.0-alpha.0 on NPM this weekend for testing. Just FYI.

[gperdomor]

Hi all... I just discover this package and looks awesome... I have on doubt, any particular reason why you use ioredis instead official redis package?

Maybe the V5 can introduce a new store with the official package?

[gperdomor]

@jaredwray or maybe rename the current redis implementation to @keyv/ioredis in v5, in that way a new package @keyv/redis with official redis package can be created, I can help if you want :D

[jaredwray]

@gperdomor - thanks for your suggestion and I will plan to look at moving @keyv/redis to the standard driver as that does make sense. Working on getting v5 out and then will look at that.

[gperdomor]

Hi again @jaredwray, it's me again... I want to know if there is any update regarding to the V5 release or if existe any beta version available?

[gperdomor]

seems like ioredis eventually will be deprecated according to this comment, so now it's more important a node-redid adapter 🙏🏻 redis/ioredis#1870 (comment)

[jaredwray]

seems like ioredis eventually will be deprecated according to this comment, so now it's more important a node-redid adapter 🙏🏻 redis/ioredis#1870 (comment)

Correct and we will be moving to the redis driver

[jaredwray]

Hi All! Wanted to update this thread as https://www.npmjs.com/package/keyv/v/5.0.0-rc.1 is now published as next and you can download it via npm install keyv@next to get the latest. Please note, you will also want to use the latest storage adapters if using typescript as it is compatible with v5. We will be publishing them over the next 24 hours and will tag the storage adapters with next. 🎉

This includes ESM and CJS support (CJS will get phased out most likely with Keyv v6), hooks, statistics, and moving to async/await with typescript and the following breaking change:

URI and Proposed Breaking Change 🔥

After feedback from the community and looking into this more (thanks @hugo082) we have decided to longer support the uri on the constructor. This would be a major breaking change 😬 but we believe everything else will be worth it.

import KeyvMemcache from '@keyv/memcache';
import Keyv from 'keyv';

// don't work anymore
const keyv = new Keyv('memcache://user:pass@localhost:11211');

// works
const keyv = new Keyv({ store: new KeyvMemcache('memcache://user:pass@localhost:11211') });

// proposals moving forward. 
const keyv = new Keyv(new KeyvMemcache('memcache://user:pass@localhost:11211'));
const keyv = new Keyv(KeyvMemcache, 'memcache://user:pass@localhost:11211');
const keyv = new Keyv({ store: new KeyvMemcache('memcache://user:pass@localhost:11211') });

Finally, After more consideration we are going to get v5 live and past RC (we are planning that to be in June after testing) the plan moving forward is to do the final feature sets since they are non-breaking.

I will be closing this thread down and adding issues for the following features when it makes sense to track:

• Storage Tiering - we are considering not adding this in because it seems to really be a feature that we want to support in caching systems such as cache-manager and cacheable long term but would love your feedback 👍 or 👎
• Namespacing - this is currently being planned out and should be non breaking change but will be later.
• Simplified Compression Adapters - this will be added as an enhancement as time permits this year.