Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed upload creates "shadow" container image #388

Open
hashkeks opened this issue Dec 7, 2021 · 1 comment
Open

Failed upload creates "shadow" container image #388

hashkeks opened this issue Dec 7, 2021 · 1 comment

Comments

@hashkeks
Copy link
Contributor

hashkeks commented Dec 7, 2021

Describe the bug
When there is only slightly more space left on the registry host than a image upload would need, the upload fails with HTTP error 500 (internal server error). The problem is, that even though the upload fails, a "logical" / shadow image remains. This shadow image is then part of a collection and seems to reference an actual image file that is not present. Therefore:

  • MinIO is not cleaned up (we guess that is done with the function delete_minio_container in minio.py) even though there are no real references left but the shadow images resulting from the bug
  • Such buggy shadow images can't be deleted and therefore their collection can't be either.
  • When trying to upload an image again, it fails instantly but still a shadow image with the according name of the upload is created

To Reproduce
For example: we have 212MB of storage left and upload a 161MB image that hasn't been previously uploaded. An image object is created in sregistry to represent the image file in collections, etc. The upload finishes with 100% done, but nonetheless an error message is appearing with HTTP 500. All the space is used up on the MinIO storage (guess due to MinIO data safety mechanisms needing more space than just the image size?). The image only appears in the collection overview, but not inside the collection. If we start the upload again, it fails instantly but still a shadow image is created and remains.

err1
err2

Expected behavior

  1. The upload starts just as usual
  2. While the upload is going on, the container image is created in sregistry and added to the collection. Also the "Builds" counter is increased by one.
  3. Mid-way, when the storage is full, the upload is aborted. An error message follows and the shadow image is deleted, "Builds" counter decreased by one. Everything is like before the upload.
@vsoch
Copy link
Member

vsoch commented Dec 7, 2021

I’m aware of this bug- the reason it happens is because we don’t know the name, tag etc until the last step and then we rename and show in the collection. It should not, however, interfere with deleting a collection, although it will show it to exist.

If you have a suggested fix that resolves the issue you are seeing I’m happy to review a PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants