Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clean up dangling images #4915

Merged
merged 1 commit into from May 13, 2024
Merged

Clean up dangling images #4915

merged 1 commit into from May 13, 2024

Conversation

B1F030
Copy link
Contributor

@B1F030 B1F030 commented May 8, 2024

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
Clean up dangling images after each run of hack/local-up-karmada.sh.

Which issue(s) this PR fixes:
Fixes #4914

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

@karmada-bot karmada-bot added the kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. label May 8, 2024
@karmada-bot
Copy link
Collaborator

Welcome @B1F030! It looks like this is your first PR to karmada-io/karmada 🎉

@karmada-bot karmada-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label May 8, 2024
@codecov-commenter
Copy link

codecov-commenter commented May 8, 2024 •
edited

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 53.18%. Comparing base (4ccffcc) to head (5544c49).

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #4915   +/-   ##
=======================================
  Coverage   53.18%   53.18%           
=======================================
  Files         252      252           
  Lines       20510    20510           
=======================================
  Hits        10909    10909           
  Misses       8882     8882           
  Partials      719      719
Flag Coverage Δ
unittests 53.18% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@B1F030
Copy link
Contributor Author

B1F030 commented May 8, 2024 •
edited

The result of this PR change are reflected here.

Total reclaimed space: 0B

If we try this on normal environment, this change could clean up a lot of storage space(may be zero if you have the same image tagged).

@RainbowMango
Copy link
Member

I tried on my side, it shows that reclaimed 0B space, with a bunch of images deleted, and I don't know why :)

-bash-5.0# docker image prune -f
Deleted Images:
deleted: sha256:763d25d3ed94215c35e626c360ae9ce208f7e3770c0ec23fd9c3772b8b7d5821
deleted: sha256:871a9858500a5efb05515c12f72db5ecce35a53b86f7379f8b7765f8a4c369a2
deleted: sha256:b604619e7de83135b7a746deffafd2cbdc8d318f9ef2e0a1c06c45c00d28ab1b
deleted: sha256:8f63671f3e0a248d11f6832d80ddd94c723693bc7d346f9fd6d8364b39717d9c
deleted: sha256:c50dbb61a36f848ac3a2ed40df77be387443b878170e33712f120f392b0a5ede
deleted: sha256:df768018dd3856aeb9d7ddb05bbe6b7ab912d212ec966b182bb28711d4b38f52
deleted: sha256:86a1211a35bd43b220c71727c47e42a5dbf728a49c47003d825c0d02788e31fe
deleted: sha256:90ede044761a80ca632571c676e50521c1da66d4827f16d886c8d2f5b5246602
deleted: sha256:fb30dd1b6009234dee8e3ac2ab39ec0f9d0f7ab2717d838c3dbe18c82678f409
deleted: sha256:9a76b1350c435ff4e4d3100970ef6f9c72e285c22180f98c2e88580d057d8f96
deleted: sha256:e0a618ac6770ad4255ffb8a0d43a724524aaa7b2d2e22c8d59f96c1c3c3edd9d
deleted: sha256:33abba3ccc6af87220c6faed76ca116b29ecdbe74dbcc0cc362c797a33501953
deleted: sha256:d001ec31a9034d9a57585d3d516c1ab634c791a985225a017cab07d968628efd
deleted: sha256:b2eb1e6e448b9b4cac66aa9b0beb25411ef1cc9a86df26361268fe0b7a9efa35
deleted: sha256:631be387b31ea8a6d88de4ca79cb6f1e84ff41d9682cc583527a624706ef2bb9
deleted: sha256:ad3edb405bac1dd6fe95c276b2614f1ec7e996f1d7e3a594a1572f6111c77254
deleted: sha256:17913e88e23853bd5e37097944b6067456c21658470546d79e37cd760d3faa9b
deleted: sha256:cfbfd82225106494d01fe2cdf6a6fbfcf8087ed1468f15a8977df241b2bc5fd9
deleted: sha256:75b7ed548567bb6eb3db2300579ca2471bc51dab4f9ba117bd23643b38056e70
deleted: sha256:151ae598f4141d3f7c98f75a4afddb2030f96f643400f457aab07edc046e83a5
deleted: sha256:17e304c1e2d58f59d89283b64883056d56be0068750abf8f04cd4ddaff9ff5fe
deleted: sha256:3ddd8050c32c42a8631ae0a5218e260a6f7795f6812230144fa192a0f57ebc07
deleted: sha256:2b1032ffcb9b8de165fdd6980e104e21475188faf3ea640755f47ef351c690f5
deleted: sha256:e8a18592841ababd600e4064b1fead3b930bb0ad2ea1a7c301b86cb1db166def
deleted: sha256:711b2db088ffa072057356b94ae3414e7148dd1732231804a5676a124adb494a
deleted: sha256:e9b72c4fb3ab8c718d212b19473750a75f53676cbb77a0dcf1911cfaf504d2d7
deleted: sha256:2e6c46ee4b2c6e09c2ee7c8de36a2de8e5dbaa674e017ece66b40bad4c072542
deleted: sha256:bea50ba1371613d84bffc28c0eba4eb656bb02080101e9f3489a6a4287361b20
deleted: sha256:44406add84efd9e14c386b63ae751c69fd0aaa8b83519bf480dd40701a369a13
deleted: sha256:bbb08bff066b3dcc6da2f66dbadccbe505301fa12e0505395dc7005870968f03
deleted: sha256:a1ebae6324299b44debc323e2ac7e717040bd62f0e26a7934440556e7465bce5
deleted: sha256:9cd0135a60db3405c81a1fc8e1c9d52a9d6ba0e795978795c2398e3f27dadc1d
deleted: sha256:bbd9e8ae8481b875aaf0fe8badde9ca5ddcca491afb96ec85deb20f3188472cf
deleted: sha256:c583b8e929f97cfabae56a6d343090246f01a4d40efcc884a935812bd0905b72
deleted: sha256:2d4ade24e70856b85be77813c1faad33ac9a44b876a6b9f8251373a178989f11
deleted: sha256:4a8774af1bb927742ba5fbfd3eeb650e2af44a0a7f58c6806c814ae441e97265
deleted: sha256:e31a708e51d012bae9b0128e8795db8ca19c06d3465dd9d358b7d7f4d4c0ca6d
deleted: sha256:a9fc1216c695c148928db88565b31def845c9c4b49962340a4a80fbbc7ab48c2
deleted: sha256:450139a88040f632c78019f5db2665f59379b380a2b957520546652fcd9264de
deleted: sha256:80a15fb9343ebe59c5ade8c28116fd1d54c4de24d51771ce75c024c93b1dc2b4
deleted: sha256:2a71427a33ff0795b3373a9afbce86c694d0b6327ccb46406c97189ac5062ee9
deleted: sha256:cc2598578a659bb3bb2f1b4c8cadc95ed26de933379270480ceb9ff3f5803601
deleted: sha256:5b08fa0cbef2872ff513a4ea19742ab1e8ac7e3ca6bd8778ca209e8b2d93ac4c
deleted: sha256:fe29ac36c88e572eae2cad9522b1d1894d5e4368a9ba9343462c0a75ec2bee84

Total reclaimed space: 0B

@B1F030
Copy link
Contributor Author

B1F030 commented May 8, 2024 •
edited

I tried on my side, it shows that reclaimed 0B space, with a bunch of images deleted, and I don't know why :)

Sorry for my mistake, cleaning up dangling images may not reclaim storage space, if you have the same image tagged.
For example:

karmada/karmada-aggregated-apiserver          latest    93d7db7c82c7   14 minutes ago      98.2MB
<none>                                        <none>    d7207b9f233d   About an hour ago   98.2MB

These two images are actually the same, using the same filesystem, and having same layers, just have different id because they are built from different run.
In this case, the Total reclaimed space is correct to be 0.

Which means, if you delete the image karmada/karmada-aggregated-apiserver:latest first(make sure no container is using this image), then run docker image prune -f, you may get Total reclaimed space: 98.2MB.

@zhzhuang-zju
Copy link
Contributor

In this case, the Total reclaimed space is correct to be 0.

I verified locally that deleting the dangling image generated by command docker build ., the Total reclaimed space shown is also 0. Many related issues can be found, such as docker/cli#1751, but it seems that there is no answer yet

@B1F030
Copy link
Contributor Author

B1F030 commented May 8, 2024 •
edited 

appledeMacBook-Pro:~ apple$ docker images
REPOSITORY                             TAG       IMAGE ID       CREATED          SIZE
karmada/karmada-aggregated-apiserver   latest    688235b8f0fb   4 seconds ago    98.7MB
<none>                                 <none>    4d63cb6d9596   25 seconds ago   98.7MB

I tried remove karmada/karmada-aggregated-apiserver first, and then docker image prune -f, still got Total reclaimed space: 0B. Weird, but seems to be a bug of docker.

But this work:

appledeMacBook-Pro:~ apple$ docker image prune -a
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: alpine:3.19.1
untagged: alpine@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b
deleted: sha256:ace17d5d883e9ea5a21138d0608d60aa2376c68f616c55b0b7e73fba6d8556a3
untagged: kindest/node:v1.27.3
untagged: kindest/node@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
deleted: sha256:6e360fda99b513dc6cadb9fb94d9e5910cab395a33641db08c92e93a0abaa6c6
deleted: sha256:7c2896a347b6b25f1f1c2286382791854b18351ca268b6ea8ae2fc8e13314ae1
deleted: sha256:297cac2aca85f29d00df816ee553424e35f2ff2b2d66ef66f092be2b190ac121

Total reclaimed space: 849.7MB

ikaven1024
ikaven1024 reviewed May 8, 2024
View reviewed changes
hack/local-up-karmada.sh Outdated Show resolved Hide resolved
@B1F030
Copy link
Contributor Author

B1F030 commented May 8, 2024 

appledeMacBook-Pro:~ apple$ docker images
REPOSITORY                             TAG       IMAGE ID       CREATED              SIZE
karmada/karmada-aggregated-apiserver   latest    ce11352b8dd2   24 seconds ago       98.7MB
<none>                                 <none>    f0da7dd4ea10   About a minute ago   98.7MB
appledeMacBook-Pro:~ apple$ 
appledeMacBook-Pro:~ apple$ docker inspect f0da7dd4ea10
{...
            "Labels": {
                "maintainer": "karmada.io"
            }
...}
appledeMacBook-Pro:~ apple$ 
appledeMacBook-Pro:~ apple$ docker inspect ce11352b8dd2
{...
            "Labels": {
                "maintainer": "karmada.io"
            }
...}
appledeMacBook-Pro:~ apple$ 
appledeMacBook-Pro:~ apple$ docker image prune --force --filter "label=maintainer=karmada.io"
Deleted Images:
deleted: sha256:f0da7dd4ea10629d0205c320e17e91df9183238ff51f1d347366dd78cc47f55a

Total reclaimed space: 0B
appledeMacBook-Pro:~ apple$ 
appledeMacBook-Pro:~ apple$ docker images
REPOSITORY                             TAG       IMAGE ID       CREATED          SIZE
karmada/karmada-aggregated-apiserver   latest    ce11352b8dd2   34 seconds ago   98.7MB

Works in my local environment.

@B1F030
Copy link
Contributor Author

B1F030 commented May 8, 2024 •
edited

For more information, run docker system df:

# Basic env:
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          3         1         1.496GB   857.5MB (57%)
Containers      1         0         5.583kB   5.583kB (100%)
Local Volumes   1         1         209.3MB   0B (0%)
Build Cache     1         0         0B        0B

# Build once:
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          4         1         1.587GB   948.4MB (59%)
Containers      1         0         5.583kB   5.583kB (100%)
Local Volumes   1         1         209.3MB   0B (0%)
Build Cache     7         0         89.18MB   89.18MB

# Build twice:
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          5         1         1.676GB   1.038GB (61%)
Containers      1         0         5.583kB   5.583kB (100%)
Local Volumes   1         1         209.3MB   0B (0%)
Build Cache     8         0         89.18MB   89.18MB

# After docker image prune --force --filter "label=maintainer=karmada.io":
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          4         1         1.587GB   948.4MB (59%)
Containers      1         0         5.583kB   5.583kB (100%)
Local Volumes   1         1         209.3MB   0B (0%)
Build Cache     8         0         178.4MB   178.4MB

# After sudo docker builder prune --force:
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          4         1         1.587GB   948.4MB (59%)
Containers      1         0         5.583kB   5.583kB (100%)
Local Volumes   1         1         209.3MB   0B (0%)
Build Cache     4         0         0B        0B

As you can see, when we run docker image prune --force --filter "label=maintainer=karmada.io", the RECLAIMABLE reduces from 1.038GB (61%) to 948.4MB (59%), but Build Cache increases from 89.18MB to 178.4MB, only after sudo docker builder prune --force, the Build Cache can be reclaimed.

So, in conclusion, after docker image prune, we got Total reclaimed space: 0B because the dangling image is moved to Build Cache provisionally, we can reclaim those storage space by executing the cleanup command docker builder prune manually.

@karmada-bot karmada-bot added the do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. label May 9, 2024
@B1F030 B1F030 force-pushed the clean-up-dangling-images branch from 2974ce9 to a0356d2 Compare May 9, 2024 07:15
@karmada-bot karmada-bot removed the do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. label May 9, 2024
zhzhuang-zju
zhzhuang-zju reviewed May 9, 2024
View reviewed changes
hack/local-up-karmada.sh Outdated Show resolved Hide resolved
hack/docker.sh Outdated Show resolved Hide resolved
hack/docker.sh Outdated Show resolved Hide resolved
@karmada-bot karmada-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 9, 2024
@B1F030 B1F030 force-pushed the clean-up-dangling-images branch from 5b43668 to 4ea01f8 Compare May 9, 2024 09:45
@B1F030
Copy link
Contributor Author

B1F030 commented May 9, 2024

/cc @zhzhuang-zju

@zhzhuang-zju
Copy link
Contributor

/cc @zhzhuang-zju

Good job! LGTM

cc @ikaven1024

@karmada-bot karmada-bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 10, 2024
@ikaven1024
Copy link
Member

LGTM

@B1F030
Copy link
Contributor Author

B1F030 commented May 10, 2024

/retest

@karmada-bot
Copy link
Collaborator

@B1F030: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@zhzhuang-zju
Copy link
Contributor

/ok-to-test

@B1F030
Copy link
Contributor Author

B1F030 commented May 11, 2024

/retest

@B1F030
Copy link
Contributor Author

B1F030 commented May 12, 2024

@RainbowMango PTAL, thanks.

@RainbowMango
Copy link
Member

/assign

RainbowMango
RainbowMango approved these changes May 13, 2024
View reviewed changes
Copy link
Member

@RainbowMango RainbowMango left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

Thanks for the excellent efforts on this @ikaven1024 @zhzhuang-zju @B1F030

@karmada-bot karmada-bot added the lgtm Indicates that a PR is ready to be merged. label May 13, 2024
@karmada-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RainbowMango

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@karmada-bot karmada-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 13, 2024
@karmada-bot karmada-bot merged commit 124c396 into karmada-io:master May 13, 2024
12 checks passed
@B1F030 B1F030 deleted the clean-up-dangling-images branch May 13, 2024 07:02
@B1F030 B1F030 mentioned this pull request May 15, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RainbowMango RainbowMango RainbowMango approved these changes

@ikaven1024 ikaven1024 ikaven1024 left review comments

@XiShanYongYe-Chang XiShanYongYe-Chang XiShanYongYe-Chang left review comments

@zhzhuang-zju zhzhuang-zju Awaiting requested review from zhzhuang-zju

Assignees

@RainbowMango RainbowMango

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm Indicates that a PR is ready to be merged. ok-to-test size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Dangling images left after each run of hack/local-up-karmada.sh
7 participants
@B1F030 @karmada-bot @codecov-commenter @RainbowMango @zhzhuang-zju @ikaven1024 @XiShanYongYe-Chang