Using a custom domain configured SSL/TLS when logging with Google

[tam-pham-ts]

Hi everyone,

I am trying to use my dns domain, authenticating with Google and configuring SSL/TLS for our system.
I have configured:

• Istio Gateway hostname to my dns domain
• Change the credential name of Istio Gateway to my certificate
• Used SSO authenticate with Google

And the result is that when I login with Google, it returns callback url https://mydomain.com/oauth2/callback and throw 500 Internal Server Error. Could you give me some suggestions that I can use for my context (custom domain, SSL/TLS and SSO with Google)?

I hope there will have someone interested with this problem.

Thank you.

[thesuperzapper]

Hey @tam-pham-ts so I can help, can you clarify a few things:

1. Can you provide any more logs about what the actual error is beyond 500?
2. How are have you exposed the deploykf_istio_gateway service?
   • For example, are you using a LoadBalancer, or have you pointed an Ingress to this service?
3. Is your deploykf_istio_gateway service/ingress providing its own HTTPS/TLS encapsulation?
   • If so, you don't need to provide the certificate to the istio gateway (but you will need to ensure your "outer" service is capable of talking to the "inner" service with its self-signed certificate)
   • Alternatively, you could have your "outer" service "pass through" the TLS to the deploykf_istio_gateway, and then provision a cert-manager ClusterIssuer that is capable of making valid certificates for your domain, and tell deployKF to use that issuer with the deploykf_dependencies.cert_manager.clusterIssuer values.

[tam-pham-ts]

Hi @thesuperzapper,

So I want to set up the SSL/TLS when I had an existing valid certificate from Cloudflare. but still ensure everything is work well. What should I do?
Did you set up SSL/TLS for your system? If you did, how did you do that? Please suggest us some solutions.

My current configurations as below:

  deploykf_istio_gateway:
    namespace: deploykf-istio-gateway

    gateway:
      hostname: kubeflow-mlops.mydomain.com
      ports:
        http: 80
        https: 443
      selectorLabels:
        app: deploykf-gateway
        istio: deploykf-gateway

    gatewayDeployment:
      serviceAccount:
        name: deploykf-gateway
        annotations: {}

    gatewayService:
      name: deploykf-gateway
      annotations:
        cloud.google.com/load-balancer-type: Internal
        networking.gke.io/internal-load-balancer-allow-global-access: "true"
      type: LoadBalancer

deploykf_dependencies:
  cert_manager:
    enabled: true
    namespace: cert-manager
    
    clusterIssuer:
      enabled: true
      issuerName: deploykf-gateway-issuer

Thanks.

[thesuperzapper]

@tam-pham-ts it is an understatement to say this is a complicated topic! In fact, I am currently writing a dedicated documentation page just about "exposing the deployKF gateway" which will describe configuring cert-manager (for TLS certificates) and using external-dns (for managing DNS records).

However, so you can get this done before then, here is a brief overview of the cert-manager part:

1. You need to manually create a cert-manager ClusterIssuer in your custer which is able to provision certificates for your domain
2. For almost everyone, the best option will be using LetsEncrypt to issue your certificates
3. Because deployKF uses a wildcard certificate, you MUST use the DNS01 challenge type (not the HTTP01 type)
4. How you create a LetsEncrypt ClusterIssuer with DNS01 will depend on which DNS provider you are using:
   • AWS Route 53:
     • Upstream Docs
     • To use "EKS IRSA" (rather than a SA token), you can annotate the cert-manager ServiceAccount with the deploykf_dependencies.cert_manager.controller.serviceAccount.annotations value (see these docs for more info)
   • Google CloudDNS:
     • Upstream Docs
     • To use "GKE WorkloadIdentity" (rather than a SA token), you can annotate the cert-manager ServiceAccount with the deploykf_dependencies.cert_manager.controller.serviceAccount.annotations value (see these docs for more info)
   • Azure DNS:
     • Upstream Docs
     • To use "Azure AD Workload Identity" (rather than a Service Principal token), you can label the cert-manager Pods & ServiceAccount with the TBA (will be added in the next release) values (see these docs for more info)
   • Cloudflare DNS:
     • Upstream Docs
   • Akamai Edge DNS:
     • Upstream Docs
   • Other (RFC2136):
     • Upstream Docs
5. Once you have the ClusterIssuer, you can tell deployKF to use it by setting the following values:
   • deploykf_dependencies.cert_manager.clusterIssuer.enabled to false
   • deploykf_dependencies.cert_manager.clusterIssuer.issuerName to the name of your ClusterIssuer

---

You could also avoid the need to configure a ClusterIssuer by provisioning an Ingress or Gateway (provided by your cloud provider's Kubernetes distribution) which targets the Service named deploykf-gateway (in the deploykf-istio-gateway Namespace) on the ports named http and https.

But this is very specific to which Kubernetes distribution you are using!
(NOTE: if you are a cloud vendor, please reach out to me if you want to help provide docs for your platform)

Also, if your cloud vendor’s Ingress/Gateway has a problem with self-signed certificates on the backend, please tell me!

• Right now, if you set deploykf_core.deploykf_istio_gateway.gateway.tls.enabled to false, deployKF will think the user is not accessing it via HTTP (so the links/redirects will not be correct)
• This means that you have to leave it as true, which means that the Ingress/Gateway has to have to be happy talking to the service via HTTPS (and with its self-signed certificate)
• I am hesitant to provide a value that allows the Ingress/Gateway to communicate with the Service via HTTP, as even though that traffic is cluster-internal, its bad to have it unencrypted.

[em-le-ts]

hi @thesuperzapper

We trying set up an ingress to access Service deploykf-gateway (in the deploykf-istio-gateway Namespace) on the ports named http and https. But maybe it's having self-certificate problem.

Please have a look at the configuration below for me, if there are any mistakes please let us know.

• We were provisioning Ingress Controller on GCP and It was been create Internal Network Load balancer ( L4 ).
  We can't create GCP Internal Application Loadbaalncer ( L7), because it requires a proxy-only subnet on our VPC ( conflict with our network )

This is our helm value

ingress-nginx:
  controller:
    kind: Deployment
    ingressClass: internal-nginx
    ingressClassResource:
      name: internal-nginx
      default: false
      controllerValue: "k8s.io/internal-ingress-nginx"
    service:
      externalTrafficPolicy: "Local"
      external:
        enabled: false
      internal:
        enabled: true
        annotations:
          networking.gke.io/load-balancer-type: "Internal"
        externalTrafficPolicy: "Local"
    electionID: internal-ingress-controller-leader

• Our ingress config:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: envoy-ingress
  namespace: deploykf-istio-gateway
spec:
  ingressClassName: internal-nginx
  rules:
  - host: kubeflow-mlops.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: deploykf-gateway
            port:
              number: 80
        path: /
        pathType: Prefix
      - backend:
          service:
            name: deploykf-gateway
            port:
              number: 443
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - kubeflow-mlops.mydomain.com
    secretName: mydomain-com-tls

• DeployKF Gateway

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  labels:
    app.kubernetes.io/instance: dkf-core--deploykf-istio-gateway
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: deploykf-istio-gateway
    argocd.argoproj.io/instance: dkf-core--deploykf-istio-gateway
    helm.sh/chart: deploykf-istio-gateway-1.0.0
    k8slens-edit-resource-version: v1alpha3
  name: deploykf-gateway
  namespace: deploykf-istio-gateway
spec:
  selector:
    app: deploykf-gateway
    istio: deploykf-gateway
  servers:
  - hosts:
    - '*/kubeflow-mlops.mydomain.com'
    - kubeflow-argo-workflows/argo-server.kubeflow-mlops.mydomain.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: deploykf-istio-gateway-cert
      mode: SIMPLE

• Deploykf-istio-gateway-https-redirect

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  labels:
    app.kubernetes.io/instance: dkf-core--deploykf-istio-gateway
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: deploykf-istio-gateway
    argocd.argoproj.io/instance: dkf-core--deploykf-istio-gateway
    helm.sh/chart: deploykf-istio-gateway-1.0.0
    k8slens-edit-resource-version: v1alpha3
  name: deploykf-istio-gateway-https-redirect
  namespace: deploykf-istio-gateway
spec:
  selector:
    app: deploykf-gateway
    istio: deploykf-gateway
  servers:
  - hosts:
    - deploykf-istio-gateway/kubeflow-mlops.mydomain.com
    - deploykf-istio-gateway/*.kubeflow-mlops.mydomain.com
    - '*/kubeflow-mlops.mydomain.com'
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true

• This is screenshot error:
  

Sorry to bother you, have a nice weekend.
Many thanks.

cc: @tam-pham-ts

[thesuperzapper]

@em-le-ts I have published the new Expose deployKF Gateway and configure HTTPS guide on the website, can you check if it answers your question and report back here either way?

[tam-pham-ts]

Hi @thesuperzapper,

In summary, I just want to replace root certificate with my certificate, is that possible for having valid SSL/TLS?

[thesuperzapper]

@tam-pham-ts I have published the new Expose deployKF Gateway and configure HTTPS guide on the website, can you check if it answers your question and report back here either way?

[thesuperzapper]

I have published a new Expose deployKF Gateway and configure HTTPS guide on the deployKF website, it should explain how to get the TLS certificates to be valid.