Supporting for docker tmpfs volume mounts in steps

[pat-s]

fix #862

[anbraten]

Could you provide some details what you are trying to achieve by using a tmpfs, so we can maybe help you implementing it?

[pat-s]

Using docker's tmpfs as the volume mount type for the working dir (/woodpecker) instead of a local volume. See also linked issue.

This allows for all actions within /woodpecker to happen in the host's memory instead of on disk. This can result in a step up of up to 20x for certain r/w intensive tasks and also save a lot of r/w cycles on the linked disks.

Downside: no state persistence between containers in a pipelines. I.e. cloning must be disabled and everything must happen in one step.

I want to add a boolean flag (currently named use_tmpfs) to allow for this and users which know what to do can then set it on a step level.

While there is already an undocumented tmpfs step option (which would technically solve the needs of #862 already), this setting is different as it would add an additional tmpfs mount somewhere else. This would require moving all files there first and then executing any task. Or redirecting output files to this additional mount.
The option added by this PR will use a tmpfs mount as the global WP working dir within that container.

[anbraten]

Wouldn't it be possible to pass a tmpfs volume across all steps of a workflow? We could maybe add it as option to https://woodpecker-ci.org/docs/next/usage/workflow-syntax#workspace.

[pat-s]

Wouldn't it be possible to pass a tmpfs volume across all steps of a workflow?

Of course but this wouldn't help/solve the case I am describing. These mounts would live outside of /woodpecker and always require manual action to read/write files to/from this directory.

The additional issue is that you don't always want this. I.e. it only really helps for instances with a lot of RAM and you can use that RAM instead of a physical disk. On small VMs/homelabs with like 4 GB of RAM, you don't wanna use this as otherwise it will stall the host completely. And setting a sensible default there is not so easy (by default all available memory is provided for the mount).

So to me it makes most sense to have both options as "enabled on request" including a size limitation of how much RAM should be used for the mount. And maybe even a top-level env var which allows to prevent these kind of mounts on the admin side. And "proper" documentation on how and when to use this option.

Usage Example: https://vladmihalcea.com/how-to-run-integration-tests-at-warp-speed-with-docker-and-tmpfs/
Drone ref: https://docs.drone.io/pipeline/docker/syntax/volumes/temporary/
Docker: https://docs.docker.com/storage/tmpfs/

Quote from the docker ref:

As opposed to volumes and bind mounts, a tmpfs mount is temporary, and only persisted in the host memory. When the container stops, the tmpfs mount is removed, and files written there won't be persisted.

[anbraten]

Of course but this wouldn't help/solve the case I am describing. These mounts would live outside of /woodpecker and always require manual action to read/write files to/from this directory.

Sure, I was talking about the workspace as well. I basically wanted to suggest to allow a user to use a tmpfs for the /woodpecker workspace instead of the normal volume. So sth like

workspace:
  tmpfs: true

[anbraten]

Regarding the permission part, we could automatically add a label tmpfs=true to pipelines in case a tmpfs workspace (or later on maybe volume) was requested, so only agents with tmpfs=true or tmpfs=* would pick such pipelines.

[pat-s]

I basically wanted to suggest to allow a user to use a tmpfs for the /woodpecker workspace instead of the normal volume

That is the exact purpose of this PR 😄️

How the step setting should be named then and structured is TBD. I already tested it in a previous version (not in this PR) by hardcoding the TMPFS creation. Worked fine.
Now I wanted to get it working using a new step variable. Which is where I am stuck now. Do you see what is missing so that the variable is actually honored?

Regarding the permission part, we could automatically add a label tmpfs=true to pipelines in case a tmpfs workspace (or later on maybe volume) was requested, so only agents with tmpfs=true or tmpfs=* would pick such pipelines.

That might be one option but I guess you need a config setting on the admin side to allow/disallow this for pipelines in general. And/or to limit the size of the tmpfs mount. A label can surely also be added but I think there's no way around global options which apply to all agents, so that this feature doesn't require a dedicated agent necessarily.