improve: allow customizing clusterDomain

[cyanwind23]

Hi deployKF team,

I think you missed this config. In my case, I have many clusters with different cluster domains and I want to config in my custom-values.yaml file. Perhaps someone else need this in future, so I create this pull request.

I hope this is useful!

[thesuperzapper]

@cyanwind23 Thanks for your suggestion.

First, I would like to know the situation which is causing your cluster domain to not be cluster.local, given this is such a widespread standard?

Also, there are a few big problems with the PR:

1. You can't update the files under the upstream/ folders under kubeflow-tools, you have to use kustomize patches.
2. I think there are at least a few hard-coded cases that assume cluster.local with no way to change it (e.g. in controllers), but I could be wrong. (A big one was the profile-controller & kfam, but I think this was resolved in Kubeflow 1.8, released yesterday)

If you are very interested in getting custom cluster domains to work, I would greatly appreciate your help in testing and seeing what breaks when you have a non-default domain, but some of it will be slightly non-obvious.

[cyanwind23]

Hi @thesuperzapper,

First, I would like to know the situation which is causing your cluster domain to not be cluster.local, given this is such a widespread standard?

When I installed K8s cluster, I changed my cluster domain to prepare for cross-cluster communication if necessary. I know changing the default value cluster.local is not a common case. Also, It might not be a good thing to do until now, as mentioned in this discussion. But I think my situation is not the only one. I saw some people did. I don't know what their purpose is, there's a lot or even simply to identify the cluster such as cluster.dev, cluster.stag or cluster.prod, etc., maybe.

You can't update the files under the upstream/ folders under kubeflow-tools, you have to use kustomize patches.

Sorry for my mistake, I will check it.

I think there are at least a few hard-coded cases that assume cluster.local with no way to change it (e.g. in controllers), but I could be wrong.

Maybe you're right. I also think so although I don't know if there is a 3rd party app hard-coded value cluster.local in the code. However, in my opinion, K8s allow us to change cluster domain value therefore the K8s components themselves must support that (not hard-coded), and 3rd party apps also should do so.

Thank you for your attention to this matter.

[cyanwind23]

I have fixed my code to not change upstream manifests and tested on my cluster with customized clusterDomain and it works.

I noticed this line pipelines-profile-controller/sync.py#L241, cluster.local is hard-coded but it can be overwritten by istio-authorization-config.yaml#L40-L58