Re-introduce a per-operation automatic build process

[mmourafiq]

This discussion was started on slack and was moved to github for future reference.

Original content:

After some thinking about the impact and the size of this feature request https://polyaxon.slack.com/archives/C6QBND2SV/p1618316341162200
I think we will probably deploy an experimental version next release (v1.9) or the one after (v1.10) to Polyaxon Cloud. Once we think the feature is stable and does not require any major migrations (both on the spec and db level) we will move it to CE & EE.

• This feature will not replace the ad-hoc build operations, users can still create independent polyaxonfiles with a kaniko/dockerize hub ref.
• The no-build requirement that the platform provides at the moment will stay the same, so users who have stable pipelines that do not require frequent changes to their images can safely ignore this feature.
• A new section build will be introduced, where users can signal to the platform that a build is required prior to starting the main operation, the build section will provide the necessary fields to provide stuff like, queue, preset, resources, node selectors, ... specific to the build.
• Cache (and invalidation) requires some improvements on the commercial offering, but they are planned anyway. I am not sure yet how it would work for CE, but we will work on it when we get to that point and will add docs around edge cases.
• By providing a build section, Polyaxon will take care of generating the image based on the project and the uuid, project:build-uuid, and will set that image automatically on the container of the main operation. The registry to use will be still configured via a connection.
• A new status building will be added to show that a build is progressing before compiling the main manifest (without the build). And a build icon in the UI to redirect to the build run for viewing info, logs, … about the build operation.
• When the build and matrix sections are used together, a single build operation will be scheduled and will be used for all runs.
• To make the process predictable and easy, when the build section is used with -u/--upload flag of the run command, the uploaded artifacts will automatically be injected in the build operation and not the main operation, which means users have to copy any necessary artifacts to their generated image.

[polyaxon-team]

https://polyaxon.com/docs/automation/builds/