dicts: Add option to use the *_byReference variants at creation time #60
Conversation
Makefile
Outdated
| @@ -82,7 +82,7 @@ update-zstd: | |||
| cp zstd/lib/zstd_errors.h . | |||
|
|
|||
| test: | |||
| CGO_ENABLED=1 GODEBUG=cgocheck=2 go test -v | |||
| CGO_ENABLED=1 GOEXPERIMENT=cgocheck2 go test -v | |||
There was a problem hiding this comment.
Recent go toolchains error on this so had to fix:
fatal error: cgocheck > 1 mode is no longer supported at runtime. Use GOEXPERIMENT=cgocheck2 at build time instead.
|
@valyala: You seem to have been busy with bigger things, so sorry to bug you with yet another PR! Have you thought about roping in another maintainer to help field changes? Do you still want contributions for missing surface area with the main |
|
@valyala: Is there interest in these kind of contributions? I'd prefer to not do an internal fork, but there are some capabilities missing here when compared to the upstream lib. Happy to make changes to the PR however you see fit if you have other implementation preferences. |
Summary
Related issue: #59
This adds two additional functions to create a
CDictorDDict:NewCDictByRef(dict []byte) (*CDict, error)NewCDictLevelByRef)NewDDictByRef(dict []byte) (*DDict, error)My particular use-case is having hundreds (~500-2000) of dictionaries cached locally in-memory, and not wanting to potentially have three copies of each.
CDictDDictTo mitigate the risk of the input bytes being garbage collected or moved, they get pinned and unpinned once
gozstd.CDictandgozstd.DDictare released. There's still risk of users mutating the input, but this is advised against in the documentation. Users of this feature implicitly accept the risks of optimization.Test Plan
I've added two types of tests:
gozstd_timing_testbenchmarksTestCompressDecompressDistinctConcurrentDictsByRefI'm happy to add more if you think it reasonable, but I think this stresses the right things: making sure that the same underlying bytes can be used concurrently with
CDictandDDict.Command Output
Full Benchmark Output