How do I embed Auth0 into kubeapps and deploy it via helm?

[tobi-at-cariad]

I would like to allow users to Authenticate with Auth0.

I have the following input at Auth0:

Name: "kubeapps-auth"
Domain: "my-auth..eu.auth0.com"
Client ID: "abcd"
Client Secret: "mysecret"

How would I deploy kubeapps with helm by using this Auth0 service?

Many thanks for your support in advance.

[antgamdia]

Hi @tobi-at-cariad,

I haven't tested it personally, but I guess you should follow this guide on how to configure the oauth2-proxy in Kubeapps and, the info you'll need from Auth0 as your OIDC provider should be something similar to: https://auth0.com/docs/get-started/applications/configure-applications-with-oidc-discovery.

Hope it helps!

[junoriosity]

Sure, this is the response there

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

I login as test@test.com, but in the RBAC I have

That makes me a bit unsure. Besides is it possible to log in even, if there is nothing in RBAC for your e-mail already and it has to be created after you already logged in into the dashboard?

[absoludity]

Sure, this is the response there

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

Great, thanks. So yes, you are authenticating fine, and Kubeapps is correctly passing the token through, pinniped is exchanging for a custom cert, but the cluster is rejecting it. We need to find out why.

I login as test@test.com, but in the RBAC I have

[snip]

That makes me a bit unsure. Besides is it possible to log in even, if there is nothing in RBAC for your e-mail already and it has to be created after you already logged in into the dashboard?

Logging in to your identity provider is unrelated to RBAC. So, you can log in via your identity provider without issue, it's just when your credential (id_token) is passed through to the cluster that we find out whether your user has access.

Is the RBAC that you've a screenshot of above a ClusterRoleBinding or a RoleBinding? I can't tell from the screenshot (I see it refers to a ClusterRole, but both can do that - actually, I assume it's a clusterrolebinding given that there's no namespaces set for each target).

So given that RBAC, I am confused - it looks correct. You should be able to verify (think we already did) that you can authenticate with that user via kubectl (kubectl auth can-i --as test@test.com get secrets -n default for example). If that's working, then the cert provided by pinniped should also work. Another debugging you could do to gather data would be to follow the normal pinniped tutorial and verify that you can authenticate via pinniped itself. The only other possibility I can think of right now is that, possibly, pinniped-proxy is failing to do the credential exchange, but no - the logs would show that.

I'll keep thinking, but if you can try the above two options in the mean-time, that'd be great.

[junoriosity]

Hi @absoludity @antgamdia,

many thanks for getting back to me. The role is a ClusterRoleBinding

and here is the output for the kubectl command

kubectl auth can-i --as test@test.com get secrets -n default
yes

If I need to provide you with something more, let me know. And many thanks for all your great support, I appreciate that very much. :)

[junoriosity]

Hi @antgamdia @absoludity , I do not know if that is of any help for the issue here, but I noticed the following:

In the whole auth0 network items we receive a 302 response. Is that correct and intended?

[absoludity]

Yes.. tho /oauth2/start URL is handled by Kubeapps' oauth2-proxy service, which redirects to your configured provider, which does its thing before redirecting back to the Kubeapps' oauth2-proxy service at /oauth2/callback which finally redirects to Kubeapps' root / (with the auth cookie set). That's all normal, the page loads with the related resources as expected, until Kubeapps then checks whether the user has perms on the k8s API server, which in your case, Kubeapps thinks it doesn't (because it gets a 401 as above). So all normal other than the known issue here of the 401.

Other avenues for more data that I can think of:

• Testing accessing the cluster via pinniped's kubectl plugin to verify it works (mentioned in my previous reply). This is important because it will verify if the cert auth that pinniped uses is working on your cluster (not sure why it wouldn't be... custom admission controllers, not sure, but whether or not it works is data worth having),
• Checking the logs of the kube-apiserver pod(s) in the kube-system namespace, particularly when you resend the request resulting in the 401 (right click on the request in your browser network tab and resend), to see if there's any error reported there when the request is made.

The data we have so far shows that the request is being passed through pinniped-proxy, which is happily using the oidc id_token to exchange it for a custom cert via pinniped, and then forward the request to the cluster. We just don't know why the cluster is rejecting that request.

[junoriosity]

@absoludity I am happy to do that, however, I cannot find it in your previous replies (but that might be due to my limited understanding). Hence, could you tell me how I can execute your suggestions here?

Also, could it be related to the fact that I am using k8s clusters on Digitalocean and pinniped is not confirmed to work fine with Digitalocean k8s clusters (see https://pinniped.dev/docs/reference/supported-clusters/)?

[antgamdia]

Mmm, good catch, I haven't thought of this impersonation proxy. In this case, the Kubeapps configuration is slightly different.
Have a look at this guide on enabling OIDC login in managed clusters.
Basically, you need to fetch both the impersonation proxy URL and CA and let Kubeapps know it should use this API server instead of the in-cluster one.

Did you already try it? By the way, thanks Michael for jumping in and providing such great insights and help!

clusters:
  - name: whatever
    apiServiceURL: https://... # impersonation proxy URL
    certificateAuthorityData: ... #  impersonation proxy CA
    pinnipedConfig:
      enabled: true

[junoriosity]

@antgamdia @absoludity Long story short: Things work well now!!! I even managed to connect several clusters with your strategy. So, for instance, I launched kubeapps in a Digitalocean cluster and I connected to kubeapps this cluster, another one from Digitalocean and one from Google. Wonderful.

There is some things that still puzzle me however.

• If the user's e is verified I can login without an issue. But if it is not verified and I try to login I receive:

Can we somehow allow users, that are not verified, to login like usual?

• I can see the catalog of available apps

but when I click on one of them I receive

However, if I click on an installed app, for instance, kubeapps, I get something sensible

Do you know how to fix that?

But honestly, I am so happy that we are finally closing in. :)

[absoludity]

@antgamdia @absoludity Long story short: Things work well now!!! I even managed to connect several clusters with your strategy. So, for instance, I launched kubeapps in a Digitalocean cluster and I connected to kubeapps this cluster, another one from Digitalocean and one from Google. Wonderful.

Excellent - so was the issue the URLs for the impersonation proxy used by pinniped? (It'd help us to know how the issue was solved).

There is some things that still puzzle me however.

* If the user's e is verified I can login without an issue. But if it is not verified and I try to login I receive:

Can we somehow allow users, that are not verified, to login like usual?

We should handle the case of an unverified email better than returning a 500, but I can't see how we can allow users with unverified email addresses, given that the email address is what's used for the RBAC username in the default setup?

* I can see the catalog of available apps

but when I click on one of them I receive

...

Do you know how to fix that?

It could be a bug on our part (I don't think the request for the available app should not have "gke" as the cluster). Could you create a specific issue for that? Thanks.

But honestly, I am so happy that we are finally closing in. :)

Glad you solved the auth issue!

[junoriosity]

@absoludity @antgamdia I can only say again, how grateful I am that you helped with all this!!! So thank you so so much! :)

Regarding the catalogue issue, I will create a new ticket.

Regarding the email verification aspect (the 500 issue), I am aware of the risks, but what would I have to do make it possible anyhow?

[tobi-at-cariad]

@absoludity @antgamdia Thanks also from my side for this very helpful input.