Unability to set affinity rules

[szczeles]

Currently, the plugin doesn't allow to set affinity on ContainerOp. In our setup, with multitenant Kubeflow deployment, we need to separate physically pipelines started by different teams (by running them on separate node pools) for proper costs allocation.

The configuration file of the plugin allows to specify tolerations, but unfortunately it's not enough to guarantee that a pods will be scheduled there (see the first use case in the docs).

It looks like setting up affinity can be done directly on ContainerOps: https://github.com/kubeflow/pipelines/blob/1.8.22/sdk/python/kfp/dsl/_container_op.py#L959, but this may be tricky to parse yaml config as an affinity setup.