Skip to content

Package diskcache provides a on-disk cache for storing http results.

License

Notifications You must be signed in to change notification settings

kenshaw/diskcache

Repository files navigation

diskcache Go Package

Package diskcache provides a standard Go HTTP transport (http.RoundTripper) implementation designed to cache, minify, compress, and transform HTTP responses on disk. Allows definition of caching policies on a per-method, per-host, or per-path basis.

Additionally provides header, body and content transformers that alter cached HTTP response headers and bodies prior to storage on disk. Includes ability to rewrite headers, white/black-list headers, strip XSS prefixes, Base64 encode/decode content, minify content, and marshal/unmarshal data stored on disk using Go's GLib/ZLib compression.

Package diskcache does not act as an on-disk HTTP proxy. Please see github.com/gregjones/httpcache for a HTTP transport implementation that provides a RFC 7234 compliant cache.

Example

A basic Go example:

// _example/example.go
package main

import (
	"fmt"
	"log"
	"net/http"
	"net/http/httputil"
	"os"
	"time"

	// "github.com/spf13/afero"
	"github.com/kenshaw/diskcache"
)

func main() {
	d, err := diskcache.New(
		// diskcache.WithFs(afero.New*(...)),
		// diskcache.WithBasePathFs(path),
		diskcache.WithAppCacheDir("diskcache-example"),
		diskcache.WithTTL(365*24*time.Hour),
		diskcache.WithHeaderWhitelist("Date", "Set-Cookie", "Content-Type"),
		diskcache.WithHeaderTransform(
			`Date:\s+(.+?)`, `Date: Not "$1"`,
		),
		diskcache.WithMinifier(),
		diskcache.WithErrorTruncator(),
		diskcache.WithGzipCompression(),
	)
	if err != nil {
		log.Fatal(err)
	}
	// create client using diskcache as the transport
	cl := &http.Client{
		Transport: d,
	}
	for i, urlstr := range []string{
		"https://github.com/kenshaw/diskcache",      // a path that exists
		"https://github.com/kenshaw/does-not-exist", // a path that doesn't
		// repeat
		"https://github.com/kenshaw/diskcache",
		"https://github.com/kenshaw/does-not-exist",
	} {
		if err := grab(cl, "GET", urlstr, i); err != nil {
			log.Fatal(err)
		}
	}
}

func grab(cl *http.Client, method, urlstr string, id int) error {
	req, err := http.NewRequest(method, urlstr, nil)
	if err != nil {
		return err
	}
	// execute request
	res, err := cl.Do(req)
	if err != nil {
		return err
	}
	fmt.Fprintf(os.Stdout, "------------------- %s %s (%d) -------------------\n", method, urlstr, id)
	buf, err := httputil.DumpResponse(res, true)
	if err != nil {
		return err
	}
	if _, err = os.Stdout.Write(buf); err != nil {
		return err
	}
	fmt.Fprintf(os.Stdout, "\n------------------- END %s %s (%d) -------------------\n\n", method, urlstr, id)
	return nil
}

See the Go package documentation for more examples.

afero support

The afero filesystem package can be used in conjunction with diskcache to satisfy advanced use-cases such as using an in-memory cache, or storing on a remote filesystem.

Notes

Prior to writing diskcache, a number of HTTP transport packages were investigated to see if they could meet the specific needs that diskcache was designed for. There are in fact a few other transport packages that provide similar functionality as diskcache (notably httpcache), however after extensive evaluation, it was decided that existing package implementations did not meet all requirements.

About

Package diskcache provides a on-disk cache for storing http results.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages