Caching in Dev Mode

[mdugue]

Summary

Hi, I would like to use built-in caching in npm dev mode. In my project I have a Server Component which is querying GPT API. The API response is slow and billed on a per request basis, so quering every time HMR kicks in is not a convenient solution. I can imagine that many similar situation exists, where requests are not "cheap" regarding time or money.

For me, neither app dir revalidate nor react cache() nor some custom runtime-cache would work during development (though perfectly once built for production).

I found in https://stackoverflow.com/questions/69388338/nextjs-using-a-memory-cache-in-dev-does-not-work the hint to build a custom file based "cache" but was wondering if there is a more "idiomatic nextsjs" approach.

Is there anything I am missing? Any hints?

Additional information

Operating System:
      Platform: darwin
      Arch: arm64
      Version: Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000
    Binaries:
      Node: 16.18.1
      npm: 8.19.2
      Yarn: 1.22.19
      pnpm: 8.2.0
    Relevant packages:
      next: 13.3.0
      eslint-config-next: 13.3.0
      react: 18.2.0
      react-dom: 18.2.0

Example

import 'server-only'
import { Configuration, OpenAIApi } from 'openai'

const openai = new OpenAIApi(new Configuration({apiKey: process.env.OPENAI_API_KEY}))

// Server Component rendering a TLDR via GPT for a given text
export default async function GptTldr(props: { text: string }) {
	const response = await openai.createCompletion({
		model: 'text-davinci-003',
		prompt: `${props.text} tl;dr`,
		temperature: 0.7,
		max_tokens: 372,
		top_p: 1,
		frequency_penalty: 0,
		presence_penalty: 1,
	})
	return <>{response.data.choices[0].text}</>
}

[mdugue]

did this problem never arise anybody else or is my issue unclear 🤔?

[dsuttonpreece]

I'm also having this issue. Every time the next page is rebuild it fires a new request irrespective of cache settings. I understand why they do this in development mode, but would be good if you could force caching across page builds.

[mdugue]

Hey @dsuttonpreece, welcome on board 😅. I was wondering why so little people seem to stumble upon this topic. Do you have any plan already how you want to solve this in your case?

[dsuttonpreece]

Unfortunately I don't have a workaround. The api I'm trying to use offers a free mock api, so I'm just switching to that for local dev. Then going to use SSR caching in prod to try limit the number of calls we use. Luckily the data we're interested in from the api doesn't change very often so I don't think it's going to be a huge issue in prod.

Would love to hear if you come up with something though!

[Wursteintopf]

Would also love a solution for this. Maybe a flag in the config to turn it on/off?

[johnsimeroth]

Looks like this topic hasn't received any attention from the next team yet, but I would also like a way to do this.

[quintesse]

I'd like to have a solution too. In our case it's not solely about caching though, it's about trying to create a single worker thread to do specific work. Firing up multiple versions of that can cause serious issues. But if we can't create singletons, we can't reliably determine if there's only a single worker or not.

[zoul]

Can you do something like locking a file exclusively and the first worker that succeeds in acquiring the lock may continue?

[quintesse]

thing is that AFAIK on unix-like systems there's no way you can "lock" a file. There are other ways to get "locks" but then we'd have to write specific code for specific OSes which seems like a bit too much work for something so basic.

[zoul]

FWIW, I have created a custom memoization function that can be used to turn any other function to its cached variant:

import { Cache } from "file-system-cache";
import crypto from "crypto";

const sharedCache = new Cache({ ttl: 60 * 5 });

type Producer<T> = (...args: any[]) => Promise<T>;

export function memoize<ReturnType, F extends Producer<ReturnType>>(
  originalFunction: F,
  key: string = originalFunction.name
): F {
  if (!key) {
    console.error("Cache key is empty, things will probably break.");
  }
  const memoized = async (...args: Parameters<F>): Promise<ReturnType> => {
    const compoundKey = hashDigest([key, ...args]);
    const cachedValue = await sharedCache.get(compoundKey);
    if (cachedValue) {
      return cachedValue;
    } else {
      const freshValue = await originalFunction(...args);
      sharedCache.set(compoundKey, freshValue);
      return freshValue;
    }
  };
  return memoized as F;
}

function hashDigest(params: string[]): string {
  const shasum = crypto.createHash("sha1");
  shasum.update(params.join(":"));
  return shasum.digest("hex").slice(0, 10);
}

It’s used like this:

const originalFunction = /* call some API or whatever */
const cached = memoize(originalFunction); // now use this instead of originalFunction

I would still prefer some built-in solution in Next, though.

[AlexLomm]

Would be nice to make the memoize function have the same signature to react's cache, that way something like this would become possible:

import { cache as reactCache } from 'react';

// ...

// substitute the default `cache` with memoize in dev env
export const cache =
  process.env.NODE_ENV === 'production' ? reactCache : memoize;

[rszalski]

I'm in the same boat, using a Snowflake SDK client.

Tried following workarounds like this Prisma doc would be totally acceptable but this seems to no longer work. Not sure if this is due to App Router vs Page Router differences or simply something changed in one of the bigger Next.js releases.

Another use case, beyond cost (which I totally understand) is being rate limited. Assuming the reload does clear cache so that it doesn't have a memory leak, when dealing with connections it might still result in a client being blocked for abusing.

The fewer differences between environments the better IMO.

See also:

• https://stackoverflow.com/questions/75272877/how-to-prevent-next-js-from-instantiating-a-singleton-class-object-multiple-time
• https://stackoverflow.com/questions/64712126/nextjs-initialize-an-object-once
• Global variables #15054

[sudhir4ev]

I can confirm the solution prescribed in Prisma doc for preserving instances in development mode works only when using pages router. Its absolutely useless when using app router. 😠

[AaronLayton]

I have come across this as well, I can't really test the caching strategy in dev mode without somehow turning this on

[Andrewjeska]

Would also love this feature. I get 1000 API calls per day.

[AaronLayton]

@leerob this would be a great feature when developing locally so we can test things. (also a DevTools for NextJS would be great, hint hint 😉 )

[magaldima]

This would be very helpful!

[mix3d]

Throwing my hat in the ring that this would have saved a few hours of troubleshooting in my dev env. I want to test that the caching is working correctly, I shouldn't have to do that "in prod".

[laurencefass]

Is this possible in 14.1? Are there any other open related tickets or response from core devs?

[amirrezaDev1378]

This is really needed in development mode.

[Dreathorian]

yep running into db timeout issues when adjusting my css...

[nhducit]

We have the official Redis custom cache handler, but we cannot use it in the local environment. How can we verify the caching work?

https://github.com/vercel/next.js/blob/canary/examples/cache-handler-redis/README.md

[AaronLayton]

All caching in NextJS 15 is opt-in now, so this will probably solve your issue
https://nextjs.org/blog/next-15-rc#caching-updates

[mix3d]

No, the issue is that we want to enable caching *in Dev mode, locally*. We shouldn't have to deploy just to confirm caching is working, or blow up a rate limited API while developing.

…

On Fri, May 24, 2024, 2:14 AM Aaron Layton ***@***.***> wrote: All caching in NextJS 15 is opt-in now, so this will probably solve your issue https://nextjs.org/blog/next-15-rc#caching-updates — Reply to this email directly, view it on GitHub <#48481 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALH4WI6F6Y6SEKG5Z7Q7KDZD3ZHZAVCNFSM6AAAAAAXBFHZBCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNBUGE2TM> . You are receiving this because you commented.Message ID: <vercel/next. ***@***.***>

[baba43]

I keep stumbling across this problem.

It means that you can't develop your application the way it should actually be developed. If you want to use caching sensibly, you can't do this because you can not test and you can not benefit from it.

For example, I don't want a token to be generated again and again to save the API. The normal cache would actually be ideal for this. Unfortunately, it doesn't work during development and so I have to find some stupid workarounds.

Caching should ALWAYS work locally. We theoretically have native functions in the browser to skip the cache. Where possible, you should use them.

For me, this (and this: #58736) has been one of the most important pain points when using Next.js because you write so much about caching and while I would love to use it properly, you make it so hard and annoying to actually implement it in a beneficial way.