Container images as artifacts

[verdverm]

Is (could) it be possible to treat pushed container images built with DinD (i.e.) as Argo artifacts?

• it would be nice to see them as such in the UI
• it would be really nice to have them cleaned up with GC

We have branch builds, such that when the PR is merged, we want to clean up all the stuff produced from it, including both buckets and container registry

[Joibel]

It could be, and it would probably be best supported via a custom artifact plugin. #5862

The actual push to the repository would presumably be hidden from workflow's view, as in it wouldn't be the one to perform the upload.

It's a big task to implement, but if you're willing to do this, or hire someone to do it, we'd welcome a PR.

[Joibel]

Indeed, what @agilgur5 said. I wasn't asking for existing plugins to be migrated, just voicing my opinion on where I think this should lead to.

[verdverm]

Still a big ask to develop a novel plugin architecture

[agilgur5]

It's not novel, it's just a new feature.

That is how open source works, someone's gotta build it, someone's gotta refactor it. Those with a need are typically best resourced to do so, especially in an ecosystem that is largely made up of unpaid, volunteer hobbyists (like me).

[verdverm]

The problem I have is that the first reply is "you can add this, if you implement this plugin stuff I want first" despite there being a more straightforward way that requires less effort on the contributors part. Developing the plugin arch is something the core maintainers should do, people who are familiar with the code base, the projects design and direction, not something you ask a first time contributor to do

[agilgur5]

The current artifact architecture is not sustainable long-term. Less effort in the short-term may mean substantially more effort in the long term and vice-versa. Less effort does make it the right approach, as Alan said.

Argo is a large enough codebase that there are few, if any, contributors familiar with the entirety of it. People become familiar with a certain area while writing or reviewing code for a fix or feature, sometimes that's their first time touching it.

I would say we've probably had larger and more complex features implemented by first time contributors and I've seen such in other projects too. It's not very common, but it certainly does happen with some frequency -- it's the best way to get a feature implemented in a project after all. They also tend to be high quality contributions in my experience (meaning quality and number of contributions are not necessarily correlated either). It's not an ideal ask for sure, but it's certainly not unheard of.

---

Also, Alan and I didn't outright reject adding OCI registries as artifact repositories either.
This isn't a feature request though, this is a Q&A discussion -- and my answer to that is below, adding OCI wasn't part of your original question.
There's actually potentially more complexity in an OCI feature than there is in creating plugins, because they don't function the same as S3-like storage. That's probably something that should go through a larger implementation proposal -- unlike plugins, OCI hasn't been requested before and does not have pre-existing proposals.

I would actually lean toward accepting such a feature built-in, since as you said, OCI is a growing ecosystem. My main hesitancy about it being built-in would actually be around the implementation, as it may differ substantially from other artifact repositories, has to answer questions like "should image pull secrets be used?" and similar around native k8s features, and may have a different spec in the Workflow definition. In particular, the last one is the hardest to make breaking changes to and so is the one we should be most careful around, especially so for designs we have existing plans to change, such as artifacts.

If you were to add OCI as a plugin though, it can be very experimental to begin with, can break as many times as you want it to, does not need a proposal, does not even need to be tested, and does not have to answer even half of the questions. It can be half-baked to begin with, and that would be fine. Whereas a core feature has substantially higher requirements than any plugin.

[agilgur5]

Container images are tarballs, so you can upload them to an artifact repository like any other file.

Also there is existing support for Artifactory

[verdverm]

But I cannot "docker pull" from them the same way as a container registry

logically, an image produced in CI is an artifact

Many other ecosystems are moving to using OCI registries as their foundation. Helm & CUE come to mind

[agilgur5]

Correct, those are not the same type of registry/repository. I don't know of a CI provider that allows you to store artifacts in an OCI registry, storing artifacts in S3-compatible storage has been the norm for some time.

But yes, OCI registries now have various features that make them good targets for other packaging tools as they're very generic and versatile.
But they're two different ecosystems and forms of storage. For instance, for CI you'd use buckets with automatic deletion after a period of time; that kind of feature does not exist in OCI registries. The auth patterns and multi-part are also handled differently, especially in k8s. More differences beyond that.

Would be cool if you could use either, but I'm just telling you what's available now.

Notably, you didn't ask for a docker pull capability or any OCI specifics in your question -- you just asked whether an image can be an Argo artifact, the answer to that is yes.