The Varnish exhausts resources of the VM

[xiaoyundi]

I've had two varnish problems recently,

1. The memory allocated to the Varnish is 8 GB. The Varnish exhausts 16 GB resources of the VM. It seems that the Varnish memory control mechanism is invalid.
2. Rules can be modified for keys cached in the Varnish. For example, message=xxx&name=xxx and name=xxx&message=xxx can be considered as the same key.

(edit: removed duplicates, no other details were given)

[asadsa92]

A Varnish instance usually set up to use 8 GB for the stevedore (object store) also need additional memory for the other machinery.
Could you share the command line you were running?

[nigoroll]

The usual suspect is jemalloc in combination with Linux transparent huge pages (THP). You might want to have a look at this stackoverflow question.

[nigoroll]

bugwash on the first point: We should improve the documentation:

• mention overhead from the memory allocator
• mention jemalloc + THP issues
• provide links to alternative storage engines

@xiaoyundi on the second point: std.querysort() or vmod_querystring might help you.

[xiaoyundi]

The machine is 4c and 16g. The Varnish allocates 12G memory, which is sufficient for other programs.

[xiaoyundi]

bugwash on the first point: We should improve the documentation:

• mention overhead from the memory allocator
• mention jemalloc + THP issues
• provide links to alternative storage engines

@xiaoyundi on the second point: std.querysort() or vmod_querystring might help you.

I don't see what we can do to avoid this.
and in which file the Environment="MALLOC_CONF=thp:never,narenas:2" configuration is in.

[gquintard]

@xiaoyundi, You allocateed 12G for the regular cache, but Varnish also has the default Transient storage that you can restrict (`varnishd -s Transient=malloc,1G), and memory allocation can be fragmented, plus each thread needs some memory workspace. Here's an old, but hopefully useful article.

As to what to do to avoid this:

• disable Transparent Huge Pages
• plan for overhead by allocating less memory for the cache
• Environment= should be put in the unit file varnish.service if you are using systemd, or maybe in the command line, if you are using docker (-e MALLOC_CONF=thp:never,narenas:2)

This ticket doesn't appear to be a bug, which we try to keep github for. Can I invite you to continue this discussion either on the varnish-misc mailing list, IRC, or the discrod server?

[nigoroll]

Please leave this ticket open until we improved the documentation. PRs welcome!