Provide a single command to create_or_update the pipeline in CLI

[jiyongjung0]

System information

• All
• Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.
Currently pipeline create command can be used when there is no pipeline, and update command can be used when there *is an existing pipeline. It would be better to have just one command that can be used in CI/CD to create/update pipelines.

For example,

tfx pipeline create --overwrite ...

or

tfx pipeline update --missing_ok ...  (We can support --missing_ok for the `delete` command.)

Will this change the current API? How?
No.

Who will benefit with this feature?
Mostly users with CI/CD flow which doesn't have human intervention.

Do you have a workaround or are completely blocked by this? :
We can use pipeline list and examine the output but it is quite cumbersome and need a conditional branch.

[SaschaHeyer]

Hi @jiyongjung0

thanks so much for your efforts on this topic.
I prefer tfx pipeline create --overwrite it feels more natural and easier to understand

[jiyongjung0]

@SaschaHeyer
Thank you for the feedback. May I ask one more thing? Kubeflow Pipelines has a concept of pipeline version, and when we do tfx pipeline update, we actually create a new version of the pipeline instead of deleting existing one. If we delete a pipeline it might not displayed well in Kubeflow Pipelines web UI.

However, overwrite feels like we should delete the existing pipeline to me, but it might not a best fit for repeated CI/CD jobs. Do you think that is it ok to update instead of deleting the existing pipeline?

[SaschaHeyer]

Yes you're right updating make more sense. We should keep the concept of versions

[dylan-stark-anthem]

I've had the same issue as Sascha. For my situation, I have a small number of pipelines, each with a large, growing number of versions.

The tfx pipeline update --missing_ok is most appealing for us because we are typically attempting to create a new version in a CICD build context that would not necessarily have the local pipeline_args.json snippet. (As a workaround we run the create step manually, add the initial JSON to the repo, and reuse that in CICD setup by setting KUBEFLOW_HOME.)

[FatimahAdwan]

Hi, I would love to contribute to this, pls let me know soon if it is possible

[jiyongjung0]

I think that both of create --overwrite and update --missing_ok should work in the same way, so it is a matter of taste or understandability.

Contributions are always welcome! Please send a PR with a reference to this issue. Thanks!

[dylan-stark-anthem]

@jiyongjung0, thanks for the clarification. From my first readI thought pipeline update --missing_ok would create a new version of the pipeline even when the local pipeline metadata is "missing" (e.g., when you're running the update in a different context than you ran the original create).

[FatimahAdwan]

@jiyongjung0 alright then!. thanks