Skip to content
This repository has been archived by the owner on Sep 14, 2020. It is now read-only.

Authorization failure on sample operator example, maybe related to OIDC auth #349

Open
eshepelyuk opened this issue Apr 21, 2020 · 7 comments
Open

Authorization failure on sample operator example, maybe related to OIDC auth #349

eshepelyuk opened this issue Apr 21, 2020 · 7 comments
Labels
bug Something isn't working

Comments

@eshepelyuk
Copy link

Long story short

Trying to run sample operator from the docs, unable to run it. We're using OIDC for k8s cluster authorization.

Description

Using example from https://github.com/zalando-incubator/kopf/tree/master/examples/01-minimal

The exact command to reproduce the issue 
kopf run --standalone --verbose src/example.py
The full output of the command that failed 
[2020-04-21 17:42:45,962] kopf.reactor.activit [INFO    ] Initial authentication has been initiated.
[2020-04-21 17:42:45,963] kopf.activities.auth [DEBUG   ] Invoking handler 'login_via_pykube'.
[2020-04-21 17:42:46,000] kopf.activities.auth [DEBUG   ] Pykube is configured via kubeconfig file.
[2020-04-21 17:42:57,219] kopf.activities.auth [INFO    ] Handler 'login_via_pykube' succeeded.
[2020-04-21 17:42:57,219] kopf.reactor.activit [INFO    ] Initial authentication has finished.
[2020-04-21 17:42:57,358] kopf.reactor.activit [INFO    ] Re-authentication has been initiated.
[2020-04-21 17:42:57,359] kopf.activities.auth [DEBUG   ] Invoking handler 'login_via_pykube'.
[2020-04-21 17:42:57,385] kopf.activities.auth [DEBUG   ] Pykube is configured via kubeconfig file.
[2020-04-21 17:42:57,529] kopf.activities.auth [INFO    ] Handler 'login_via_pykube' succeeded.
[2020-04-21 17:42:57,529] kopf.reactor.activit [INFO    ] Re-authentication has finished.
[2020-04-21 17:42:57,530] kopf.reactor.running [ERROR   ] Root task 'watcher of kopfexamples.zalando.org' is failed: 401, message='Unauthorized', url=URL('https://api.testcluster.nas.local/apis/zalando.org/v1')
[2020-04-21 17:42:57,530] kopf.reactor.running [DEBUG   ] Root task 'poster of events' is cancelled.
[2020-04-21 17:42:57,530] kopf.reactor.running [DEBUG   ] Root task 'credentials retriever' is cancelled.
[2020-04-21 17:42:57,531] kopf.reactor.running [DEBUG   ] Root tasks are stopped: finished normally; tasks left: set()
[2020-04-21 17:42:57,531] kopf.reactor.running [DEBUG   ] Hung tasks stopping is skipped: no tasks given.
Traceback (most recent call last):
  File "/d/projects/swagger-k8s/venv/bin/kopf", line 10, in <module>
    sys.exit(main())
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/cli.py", line 32, in wrapper
    return fn(*args, **kwargs)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/cli.py", line 79, in run
    vault=__controls.vault,
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/reactor/running.py", line 111, in run
    vault=vault,
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/reactor/running.py", line 153, in operator
    await run_tasks(operator_tasks, ignored=existing_tasks)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/reactor/running.py", line 341, in run_tasks
    await _reraise(root_done | root_cancelled | hung_done | hung_cancelled)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/reactor/running.py", line 402, in _reraise
    task.result()  # can raise the regular (non-cancellation) exceptions.
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/reactor/running.py", line 418, in _root_task_checker
    await coro
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/reactor/queueing.py", line 107, in watcher
    async for event in stream:
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/clients/watching.py", line 73, in infinite_watch
    async for event in stream:
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/clients/watching.py", line 108, in streaming_watch
    async for event in stream:
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/clients/watching.py", line 125, in continuous_watch
    items, resource_version = await fetching.list_objs_rv(resource=resource, namespace=namespace)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/clients/auth.py", line 48, in wrapper
    await vault.invalidate(key, exc=e)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/structs/credentials.py", line 259, in invalidate
    raise exc
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/clients/fetching.py", line 95, in list_objs_rv
    is_namespaced = await discovery.is_namespaced(resource=resource, context=context)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/clients/auth.py", line 39, in wrapper
    return await fn(*args, **kwargs)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/clients/discovery.py", line 55, in is_namespaced
    info = await discover(resource=resource, context=context)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/clients/auth.py", line 39, in wrapper
    return await fn(*args, **kwargs)
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/kopf/clients/discovery.py", line 28, in discover
    response.raise_for_status()
  File "/d/projects/swagger-k8s/venv/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
    headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://api.testcluster.nas.local/apis/zalando.org/v1')

Environment

  • Kopf version: 0.26
  • Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"7015f71e75f670eb9e7ebd4b5749639d42e20079", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
  • Python version:
$ python --version
Python 3.7.4
  • OS/platform: Windows 10, Msys2 shell
Python packages installed 
aiohttp==3.6.2
aiojobs==0.2.2
async-timeout==3.0.1
attrs==19.3.0
certifi==2020.4.5.1
chardet==3.0.4
click==7.1.1
idna==2.9
iso8601==0.1.12
kopf==0.26
multidict==4.7.5
pip==19.0.3
pykube-ng==20.4.1
PyYAML==5.3.1
requests==2.23.0
setuptools==40.8.0
typing-extensions==3.7.4.2
urllib3==1.25.9
yarl==1.4.2
@eshepelyuk eshepelyuk added the bug Something isn't working label Apr 21, 2020
@nolar
Copy link
Contributor

nolar commented Apr 23, 2020

Thank you for reporting it with all the details, especially with the packages.

Can you please additionally pip install kubernetes — does it work then?

Kopf contains only a minimal authentication logic of using kubeconfig "as is". No token rotation or external identity engines are implemented. Instead, Kopf piggy-backs on the authentication logic of kubernetes and pykube-ng if they are installed.

kubernetes is not installed by default as a framework dependency — as it is too heavy and not needed for normal functioning in most cases (so should be pykube-ng, but it is still in the dependencies for some reason — I will take a look). OIDC seems like a complicated case for authentication (external source of tokens).

More information:

I see that pykube-ng is present, but it didn't help obviously. Maybe kubernetes has a better logic for this authentication schema.

@eshepelyuk
Copy link
Author

well, the pykube-ng itself works ! i run it from python console and retrieved k8s objects successfuly.

i'll try to install kubernetes native client and will report later

@eshepelyuk
Copy link
Author

@nolar your suggestion worked, pip install kubernetes made my code works locally.

@nolar
Copy link
Contributor

nolar commented Apr 23, 2020 •
edited

Hm. That's interesting. What I see in the logs, is that pykube's credentials were retreived and didn't work:

[2020-04-21 17:42:45,962] kopf.reactor.activit [INFO    ] Initial authentication has been initiated.
[2020-04-21 17:42:45,963] kopf.activities.auth [DEBUG   ] Invoking handler 'login_via_pykube'.
[2020-04-21 17:42:46,000] kopf.activities.auth [DEBUG   ] Pykube is configured via kubeconfig file.
[2020-04-21 17:42:57,219] kopf.activities.auth [INFO    ] Handler 'login_via_pykube' succeeded.
[2020-04-21 17:42:57,219] kopf.reactor.activit [INFO    ] Initial authentication has finished.
[2020-04-21 17:42:57,358] kopf.reactor.activit [INFO    ] Re-authentication has been initiated.
[2020-04-21 17:42:57,359] kopf.activities.auth [DEBUG   ] Invoking handler 'login_via_pykube'.
[2020-04-21 17:42:57,385] kopf.activities.auth [DEBUG   ] Pykube is configured via kubeconfig file.
[2020-04-21 17:42:57,529] kopf.activities.auth [INFO    ] Handler 'login_via_pykube' succeeded.
[2020-04-21 17:42:57,529] kopf.reactor.activit [INFO    ] Re-authentication has finished.

It is unexpected that pykube-ng works interactively. Maybe, the token extraction during auth-piggy-backing is somehow incomplete (but it should as simple as SSL certs & HTTP auth headers extraction, no magic).

When you tried it in the console, was it after the kubernetes installation and usage? Or maybe after kubectl usage? Could you also try the operator at that exact time with kubernetes uninstalled?

My hypothesis is: As far as I remember the internals of kubernetes, it gets the remote tokens via a special Google API (not K8s-related), and stores those tokens locally into ~/.kube/config. Which means, for some time after such a token rotation, the credentials will be valid even for pykube-ng — until the next expiration.

@eshepelyuk
Copy link
Author

  1. we're not using google, we're using own SSO service
  2. i tried interactive pykube-ng many times in different combinations, kubernetes was not installed, operator never worked

@styk-tv
Copy link

styk-tv commented Apr 28, 2020

Client auth seems to work ok but operator has issues accessing the api, if I had to guess, I would check/relax RBAC restrictions on what the Operator can do. Play around with Roles, RoleBinding and ClusterRole, ClusterRoleBindings. In OIDC they can bind to different attributes, anything from the token really, custom groups etc. Check under 'watcher' and make sure Operator has RBAC permission to do what you want to do.

@eshepelyuk
Copy link
Author

@styk-tv as soon I change k8s client as described abov, all is working.
So I don't think it's RBAC issue.

@kopf-archiver kopf-archiver bot mentioned this issue Aug 19, 2020
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nolar @styk-tv @eshepelyuk