Jenkins Image cleanup

[kodebach]

Based on the discoveries from the last meeting and the recent troubles with disk space (see #160) I propose we do more immediate cleanup in the Jenkins pipeline.

Currently, we directly build a lot of images in the last stage of the pipeline:

libelektra/scripts/jenkins/Jenkinsfile

Lines 1104 to 1169 in 90f7081

	tasks << buildPackage(
	"buildPackage/debian/buster",
	DOCKER_IMAGES.buster,
	"buster-unstable",
	"buster",
	DOCKER_IMAGES.buster_installed,
	release.&publishDebPackages,
	this.&updateDebianChangelog
	)
	tasks << buildPackage(
	"buildPackage/debian/bullseye",
	DOCKER_IMAGES.bullseye,
	"bullseye-unstable",
	"bullseye",
	DOCKER_IMAGES.bullseye_installed,
	release.&publishDebPackages,
	this.&updateDebianChangelog
	)
	tasks << buildPackage(
	"buildPackage/ubuntu/bionic",
	DOCKER_IMAGES.bionic,
	"bionic-unstable",
	"bionic",
	DOCKER_IMAGES.bionic_installed,
	release.&publishDebPackages,
	this.&updateDebianChangelog
	)
	tasks << buildPackage(
	"buildPackage/ubuntu/focal",
	DOCKER_IMAGES.focal,
	"focal-unstable",
	"focal",
	DOCKER_IMAGES.focal_installed,
	release.&publishDebPackages,
	this.&updateDebianChangelog
	)
	tasks << buildPackage(
	"buildPackage/fedora/36",
	DOCKER_IMAGES.fedora_36,
	"fedora-36-unstable",
	"fedora-36-unstable",
	DOCKER_IMAGES.fedora_36_installed,
	release.&publishRpmPackages,
	this.&updateFedoraChangelog,
	true
	)
	tasks << buildPackage(
	"buildPackage/fedora/37",
	DOCKER_IMAGES.fedora_37,
	"fedora-37-unstable",
	"fedora-37-unstable",
	DOCKER_IMAGES.fedora_37_installed,
	release.&publishRpmPackages,
	this.&updateFedoraChangelog,
	true
	)
	tasks << buildPackage(
	"buildPackage/opensuse/leap/15.3",
	DOCKER_IMAGES.opensuse_15_3,
	"opensuse-leap-15.3",
	"opensuse-leap-15.3",
	DOCKER_IMAGES.opensuse_15_3_installed,
	release.&publishRpmPackages,
	this.&updateFedoraChangelog,
	true
	)

Each of these buildPackage calls does a dockerUtils.buildImage call (which executes docker build):

libelektra/scripts/jenkins/Jenkinsfile

Lines 1216 to 1229 in 90f7081

	dockerUtils.buildImage(installImage)
	dir("build/package") {
	if (isMaster()) {
	publishPackagesFun(
	'doc.libelektra.org',
	"/packaging/incoming/${repoName}/",
	repoName,
	repoPrefix,
	"$VERSION",
	packageRevision,
	targetDir
	)
	}
	}

Additionally, there are few dockerUtils.buildImageStage (a wrapper around dockerUtils.buildImage) calls for the website and web-ui:

libelektra/scripts/jenkins/Jenkinsfile

Line 749 in 90f7081

tasks << dockerUtils.buildImageStage(DOCKER_IMAGES.webui_base)

libelektra/scripts/jenkins/Jenkinsfile

Lines 1334 to 1338 in 90f7081

	def buildWebsite() {
	def websiteTasks = [:]
	websiteTasks << dockerUtils.buildImageStage(DOCKER_IMAGES.website)
	return websiteTasks
	}

libelektra/scripts/jenkins/Jenkinsfile

Lines 1359 to 1364 in 90f7081

	def buildWebUI() {
	def webuiTasks = [:]
	webuiTasks << dockerUtils.buildImageStage(DOCKER_IMAGES.webui_elektrad)
	webuiTasks << dockerUtils.buildImageStage(DOCKER_IMAGES.webui_webd)
	return webuiTasks
	}

It makes sense to build those images to test, if the build works, but there is no need to keep images around. AFAICT, if the pipeline isn't running on master the images aren't even published anywhere.

So I propose we add a dockerUtils.removeImage in ElektraInitiative/jenkins-library to execute docker image rm. Then we call dockerUtils.removeImage either as soon as the images aren't needed anymore, or even better, we add a post section to the pipeline for proper cleanup.

[lukashartl]

if you don't produce junk, you don't have to clean it up.

I really like the idea of having a proper post section and cleaning everything up that is produced during the pipeline on the agents.

[kodebach]

diff --git a/scripts/jenkins/Jenkinsfile b/scripts/jenkins/Jenkinsfile
index fbb23a3eb..e5dcf1e53 100644
--- a/scripts/jenkins/Jenkinsfile
+++ b/scripts/jenkins/Jenkinsfile
@@ -207,6 +207,10 @@ Logs: ${currentBuild.rawBuild.getLog(20).join('\n')}
          to: "build@libelektra.org"
   }
   throw e
+} finally {
+  stage("cleanup") {
+    sh "docker image list --format='{{ .Repository }}' '*:${dockerUtils.tagArtifact()}' | xargs docker image rm"
+  }
 }
 
 /*****************************************************************************

I think this might actually do the trick. It should add a cleanup stage as the last thing before the Pipeline stops (for whatever reason). What I don't know is whether docker image rm on the worker node is actually enough to solve our space problem (that's why I didn't make a PR).

[github-actions]

I mark this stale as it did not have any activity for one year. I'll close it in two weeks if no further activity occurs. If you want it to be alive again, ping by writing a message here or create a new issue with the remainder of this issue.
Thank you for your contributions 💖