Use the "local" logging driver as a default for Docker Desktop

[thaJeztah]

Tell us about your request

By default, the docker daemon uses the json-file logging driver. The json-file logging driver historically was the "only" logging driver available (before custom logging drivers were introduced).

While the json-file driver is suitable as a general-purpose logging driver, it has quite some limitations:

• using JSON as a storage format introduces a significant overhead when writing and reading logs (due to mashaling/unmarshaling JSON) (see docker#34211, docker#31935)
• writes are non-atomic, which can sometimes cause partial JSON messages to be read (see docker#29511)
• by default, it does not rotate logs, nor does it restrict the file-size of logs. As a results, log-files can grow unbounded, with a real risk of running out of resources (disk space) (see docker/cli#2422, docker/compose#1083)
• to add to the above, space taken up by these log-files is currently invisible to users (neither docker system df, nor docker ps --size include the disk space taken up by logs; see docker/cli#2715)
• the files used by the json-file logging driver were never meant to be part of the API, but as they were not documented as "off bounds", resulted in various tools to consume them.

The "local" logging driver

Docker 18.09 introduced a new "local" logging driver ([documentation][https://docs.docker.com/config/containers/logging/local/], related github: docker#33475, docker#37092), which:

• Uses an optimized, binary format to store logs, reducing the overhead of JSON (un)marshaling
• Restricts the maximum on-disk size consumed to 100MB per container
• Uses log-rotation by default (max 5 files of 20MB each)
• Explicitly documents the files as an "implementation detail", not to be used by external consumers (see below)

Why is the "local" logging driver not the default for the Engine?

The json-file logging driver has been the default since before docker 1.0, and while the files used by the driver (as with anything inside /var/lib/docker) should be considered an implementation detail, not to be accessed by anything other than the docker daemon, unfortunately quite some tools consider them an "API".

For example, by default, kubernetes creates a symlink to every container's logfile, and aggregates those. There's also other tools that (for example) attempt to rotate the logfiles, or scrape logs for sending them to log aggregators. Such tools are problematic as they can result in log corruption, or prevent containers from being fully removed (being another cause for out of disk problems).

In short, while we would want to, we can't (easily) change the default driver, nor could we enable log-rotation by default (for the json-file driver) as it would break too many users.

However, I think for Docker Desktop, being a more "controlled" environment, the "local" logging driver would (IMO) be a good fit, and in line with recent features that help with cleaning up resources.

Proposal: Use the "local" logging driver as a default for Docker Desktop

For reasons outlined above, I propose to make the local logging driver for Docker Desktop; perhaps have an easy-to-use switch in the preference panel to switch between "local" and "json-file" (for backward compatibility), although documenting how to manually set these options in the daemon.json configuration would work.

Which service(s) is this request for?

Docker Desktop

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Are you currently working around the issue?

Manually editing the daemon.json configuration for docker desktop

[adeniyistephen]

@thaJeztah Are you through with this editing? I would love to work on this too. Smile

[thaJeztah]

This change would have to be made in Docker Desktop itself (as we can't easily change the default for non-Desktop installs without introducing a potentially breaking change in behaviour), and I think the parts that need changes would be in the closed-source bits of docker desktop

[agalazis]

You mention that manipulating those files manually is not right, but docker doesn't provide any tools for performing every day tasks like purging logs in any indirect way. (Is there any way I can purge logs using local driver?). To me providing a common interface to manipulate logs via docker cli is more important than changing defaults.