Feature request: actual custom key generation

[teodorescuserban]

I see that this feature has been requested at least twice but got entrenched in a particular use case (#318 that just disables the body and #444 which goes on to discuss and fix Content-Length issues.

My particular use case is that I want to remove an argument from the query just for the cache key, the request should go to the upstream server unchanged. However, because the argument that I want removed only identifies the client, and because all data is public, I'd like to cache a response no matter the client argument is, because it will be the same exact response.

Also, I am not interested in the method, protocol and host and port as part of the key. For me, just the uri_path and the query (without a specific parameter) is required.

Any chance to be able to get a way to add a custom key?

I should also mention that I am using the cache-handler caddy plugin.

Thank you.

[teodorescuserban]

Something like https://github.com/sillygod/cdp-cache?tab=readme-ov-file#cache_key

[darkweak]

Hey, @teodorescuserban I'm actually implementing it in the core.

[darkweak]

The PR is ready @teodorescuserban can you try it?

[teodorescuserban]

@darkweak, thank you for your PR!

Unfortunately, I am not able to build caddy with the specific commit:

59.74 2024/05/13 08:24:28 [INFO] exec (timeout=0s): /usr/local/go/bin/go build -o /usr/bin/caddy -ldflags -w -s -trimpath -tags nobadger
125.3 # github.com/darkweak/souin/plugins/caddy
125.3 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512124841-f4b3aefa97bb/configuration.go:389:11: ck.DisableScheme undefined (type configurationtypes.Key has no field or method DisableScheme)
125.3 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512124841-f4b3aefa97bb/configuration.go:391:11: ck.Template undefined (type configurationtypes.Key has no field or method Template)
125.3 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512124841-f4b3aefa97bb/configuration.go:393:11: ck.Hash undefined (type configurationtypes.Key has no field or method Hash)
125.3 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512124841-f4b3aefa97bb/configuration.go:477:18: config_key.DisableScheme undefined (type configurationtypes.Key has no field or method DisableScheme)
125.3 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512124841-f4b3aefa97bb/configuration.go:479:18: config_key.Template undefined (type configurationtypes.Key has no field or method Template)
125.3 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512124841-f4b3aefa97bb/configuration.go:481:18: config_key.Hash undefined (type configurationtypes.Key has no field or method Hash)
125.3 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512124841-f4b3aefa97bb/httpcache.go:192:108: dc.Key.DisableScheme undefined (type configurationtypes.Key has no field or method DisableScheme)
125.3 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512124841-f4b3aefa97bb/httpcache.go:192:133: dc.Key.Hash undefined (type configurationtypes.Key has no field or method Hash)
125.3 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512124841-f4b3aefa97bb/httpcache.go:192:192: dc.Key.Template undefined (type configurationtypes.Key has no field or method Template)
125.6 2024/05/13 08:25:34 [INFO] Skipping cleanup as requested; leaving folder intact: /tmp/buildenv_2024-05-13-0823.1014515775
125.6 2024/05/13 08:25:34 [FATAL] exit status 1
------
Dockerfile:13
--------------------
  12 |
  13 | >>> RUN CGO_ENABLED=0 GOARCH=${TARGETARCH} GOOS=${TARGETOS} \
  14 | >>>     xcaddy build \
  15 | >>>     --with github.com/darkweak/souin/plugins/caddy@f4b3aefa97bbdc8b97776854c3a0bf92172f161a
  16 |
--------------------
ERROR: failed to solve: process "/bin/sh -c CGO_ENABLED=0 GOARCH=${TARGETARCH} GOOS=${TARGETOS}     xcaddy build     --with github.com/darkweak/souin/plugins/caddy@f4b3aefa97bbdc8b97776854c3a0bf92172f161a" did not complete successfully: exit code: 1

[teodorescuserban]

I realised that was not the last commit in the PR; the last one doesn't work as well

139.0 # github.com/darkweak/souin/plugins/caddy
139.0 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512154727-aa3a9480609c/configuration.go:389:11: ck.DisableScheme undefined (type configurationtypes.Key has no field or method DisableScheme)
139.0 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512154727-aa3a9480609c/configuration.go:391:11: ck.Template undefined (type configurationtypes.Key has no field or method Template)
139.0 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512154727-aa3a9480609c/configuration.go:393:11: ck.Hash undefined (type configurationtypes.Key has no field or method Hash)
139.0 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512154727-aa3a9480609c/configuration.go:477:18: config_key.DisableScheme undefined (type configurationtypes.Key has no field or method DisableScheme)
139.0 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512154727-aa3a9480609c/configuration.go:479:18: config_key.Template undefined (type configurationtypes.Key has no field or method Template)
139.0 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512154727-aa3a9480609c/configuration.go:481:18: config_key.Hash undefined (type configurationtypes.Key has no field or method Hash)
139.0 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512154727-aa3a9480609c/httpcache.go:192:108: dc.Key.DisableScheme undefined (type configurationtypes.Key has no field or method DisableScheme)
139.0 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512154727-aa3a9480609c/httpcache.go:192:133: dc.Key.Hash undefined (type configurationtypes.Key has no field or method Hash)
139.0 /go/pkg/mod/github.com/darkweak/souin/plugins/caddy@v0.0.0-20240512154727-aa3a9480609c/httpcache.go:192:192: dc.Key.Template undefined (type configurationtypes.Key has no field or method Template)

[darkweak]

xcaddy build --with github.com/darkweak/souin/plugins/caddy@aa3a9480609c4a6c7c1d79815b119fd214ac0dda --with github.com/darkweak/souin@aa3a9480609c4a6c7c1d79815b119fd214ac0dda still doesn't work?

[teodorescuserban]

Alright, the build does work. My bad, @darkweak.
Will test it.

[teodorescuserban]

OK, I tested it. It seems to work well. However, in my use case I would need to be able to feed a custom variable in the key template. This doesn't seem to work properly.

This is my Caddyfile:

{
	order cache before rewrite

	debug
	log {
		output file /logs/main.log
		format json {
			time_format iso8601
		}
	}
	local_certs
	cache {
		default_cache_control public
		api {
			souin
		}
	}
}

:80 {
	header Content-Type "text/html; charset=utf-8"
	respond "Nothing here... Yet??????"
}

http://test.example.local {
	map {http.request.uri.query} {cacheKey} {
		~^(.*)(?:[?&]some_id=[^&]*)$ "?${1}"
		~^(some_id=[^&]*&?)(.*)$ "?${2}"
		~^([^&]*)(&some_id=[^&]*)(&?.*)$ "?${1}${3}"
		~^$ ""
		~^(.+)$ "?${1}"
	}
	header Custom-Cache-Key {http.request.uri.path}{cacheKey}
	header Content-Type "text/html; charset=utf-8"
	respond "Nothing here... Yet??????"
	cache {
		key {
			template KEY-{http.request.uri.path}-{http.request.uri.query}-{cacheKey}
		}
	}
}

Doing a curl -IX GET "test.example.local/gg/dd?aa=11&rr=22" will show the expected value of the custom variable (in the Custom-Cache-Key header) but it wont show at all in the Cache-Status header:

~> curl -IX GET "test.example.local/gg/dd?aa=11&rr=22"
HTTP/1.1 200 OK
Cache-Control: public
Cache-Status: Souin; fwd=uri-miss; stored; key=KEY-/gg/dd-aa=11&rr=22-
Content-Length: 25
Content-Type: text/html; charset=utf-8
Custom-Cache-Key: /gg/dd?aa=11&rr=22
Server: Caddy
Date: Mon, 13 May 2024 10:16:10 GMT

Any ideas, @darkweak ?

[teodorescuserban]

I mean, this particular issue can be closed by you PR, I am wondering where I should open a new issue about the custom value though :)

[darkweak]

@teodorescuserban It should work with the commit 4e350ccccf7c0e507e44ba0fd34eef324a41dff8
xcaddy build --with github.com/darkweak/souin/plugins/caddy@4e350ccccf7c0e507e44ba0fd34eef324a41dff8 --with github.com/darkweak/souin@4e350ccccf7c0e507e44ba0fd34eef324a41dff8

[teodorescuserban]

@darkweak it works like a charm. erm, actually, id does work very well ;)

Thank you for your quick response!

This ticket can now be closed imho (along with PR merge)

When we should expect this feature to be released in a new version?