Where to begin debugging custom graphdriver (entrypoint not found, stream copy error)

[Pippadi]

I'm writing a custom graphdriver with docker/go-plugins-helpers. Right now, it's just a dumb wrapper for overlay2 with a different name.

This is at a point where the plugin installs, and pulls can be performed successfully. When trying to run a container, though, it always fails with a stream copy error: reading from a closed fifo. Online fora say that this happens in out-of-memory situations, which is unlikely (the machine has more than 6G of 8G of RAM free).

At first, I was getting errors where dockerd was unable to find the /var/lib/docker/custom-graphdriver folder. I got around this by creating a symlink to the appropriate folder in the plugin (ln -s /var/lib/docker/plugins/${plugin_id}/rootfs/var/lib/docker/custom-graphdriver /var/lib/docker/custom-graphdriver). Containers still fail to run because the entrypoint is not found.

I have no clue what to do at this stage. I'd love it if someone could point me in the right direction. Please let me know if any further info is required.

Logs for docker run --rm -it busybox /bin/sh.

[thaJeztah]

The graphdriver-plugins were added as an experimental feature in #13777, but in all honesty never really were worked on after the initial implementation, and for sure are not used widely (and very likely have very little test-coverage, so I would not exclude the possibility that there's bugs related to this feature). There's some additional docs in https://github.com/docker/cli/blob/v25.0.3/docs/extend/plugins_graphdriver.md, but not sure if they would provide much help for debugging.

The original motivation for this feature was to allow moving graph-drivers out of process, so that obsolete graph-drivers (such as AuFS and devicemapper) could be removed from our code-base, but allowing users to install them as optional component for legacy distro versions that did not provide support for overlayfs. That never happened, and now that all supported distros provide support for overlay, those legacy drivers have been deprecated and removed.

All that work happened before the containerd project was started. Containerd re-implements various parts of the docker runtime, including storage, and provides a native plugin-system to allow custom storage-drivers ("snapshotters") to be used. We are currently working on finishing integration with the daemon and this will become the default in future.

In general, I would highly recommend exploring snapshotters as an alternative instead, as it's unlikely more development will happen on graphdriver plugins (also see https://docs.docker.com/storage/containerd/ in the documentation to learn how to enable the containerd image-store integration).

If it's possible for you to share; is there something specific you are trying to implement in your graph-driver?

[Pippadi]

Thank you for the speedy response! containerd plugins (in this case a diffservice plugin) allow me to do exactly what I need to. Unfortunately, this work will not be open-sourced for some time.

Thanks again for pulling me out of that rabbit hole!

[thaJeztah]

Glad it helped!

Unfortunately, this work will not be open-sourced for some time.

No worries! I was mostly curious what you were trying to achieve, in case your use-case was already supported / possible in some other way.