RBAC access denied (403) when fetching visualization file from AWS S3 #124
Labels
Comments
bobbeeke
added
kind/bug
|
Sure. It's based on a minimal example from: |
|
I did not mention it in my earlier description but I also get 403's after clicking on the Input/Output tab. So the fetching problem is not limited to the visualization tab. |
|
Can confirm that I narrowed the issue down to Istio. (!) ONLY for temporary testing purposes in non-production: Next questions are:
The idea came from: |
|
@bobbeeke first, you should NOT run any I just tested with your specific pipeline on S3, and everything works properly, so can we try the following things so we can debug:
|
|
Did what you asked but still encounter the same issue. First I performed your steps -> Issue still existed
Logging from startup pod till after the pipeline run: Namespace: example-test Namespace: example-test When I generate the 403 responses by clicking on the visualizations tab I get these logs: Namespace: kubeflow Namespace: kubeflow And from my nginx ingress controller: |
|
Not sure if the issue is probably related to my nginx ingress setup in front of the istio-gateway but here is the definition: |
|
@bobbeeke because you said that creating the following authorization policy fixed the issue: ## !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
## WARNING DO NOT USE, DISABLES AUTH ##
## !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: allow-all
## NOTE: I assume you meant `example-test` here?
namespace: my-namespace
spec:
rules:
- {}You must have made some custom The places I would check are:
|
|
"## NOTE: I assume you meant I certainly did not mess with authorization policies except for my one-time "allow-all" test to exclude other possible causes. Also performed a full uninstall and a clean deployment. I will dive into Istio authorization and troubleshooting (little experience till now) myself to find out what is causing this in my environment. Thanks for the pointers so far, they help a lot! Current status: If I add this test policy to my profile namespace I can fetch from S3: ## !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
## WARNING DO NOT USE, MESSES WITH AUTH ##
## !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: test-policy-bob
namespace: example-test
spec:
rules:
- to:
- operation:
methods: ["GET"]
paths: ["/pipeline/artifacts/*"]
selector:
matchLabels:
app: ml-pipeline-ui-artifactSo as far as I understand a GET request on ml-pipeline-ui-artifact is blocked for some unknown reason. I still miss some knowledge right now on this topic but will try to learn and narrow it down further. I will share findings here. |
|
@bobbeeke we can probably work it out if you give the full YAML of all the |
|
Sure, here they are: ---
- apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
annotations:
argocd.argoproj.io/compare-options: ""
argocd.argoproj.io/sync-options: ""
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"security.istio.io/v1beta1","kind":"AuthorizationPolicy","metadata":{"annotations":{"argocd.argoproj.io/compare-options":"","argocd.argoproj.io/sync-options":""},"labels":{"app.kubernetes.io/instance":"kf-tools--pipelines"},"name":"ml-pipeline-visualizationserver","namespace":"example-test"},"spec":{"rules":[{"from":[{"source":{"principals":["cluster.local/ns/kubeflow/sa/ml-pipeline"]}}]}],"selector":{"matchLabels":{"app":"ml-pipeline-visualizationserver"}}}}
creationTimestamp: "2024-04-26T09:02:37Z"
generation: 14
labels:
app.kubernetes.io/instance: kf-tools--pipelines
name: ml-pipeline-visualizationserver
namespace: example-test
resourceVersion: "42385275"
uid: 3920ccd6-74f2-429d-83b6-16652b298dc6
spec:
rules:
- from:
- source:
principals:
- cluster.local/ns/kubeflow/sa/ml-pipeline
selector:
matchLabels:
app: ml-pipeline-visualizationserver
---
- apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
annotations:
role: admin
user: admin@example.com
creationTimestamp: "2024-04-26T08:58:10Z"
generation: 7
name: ns-owner-access-istio
namespace: example-test
ownerReferences:
- apiVersion: kubeflow.org/v1
blockOwnerDeletion: true
controller: true
kind: Profile
name: example-test
uid: 9e705b85-91d9-4c56-861a-a739dae2b1c2
resourceVersion: "42051914"
uid: ddae1583-7faf-44a4-b69c-80ec21f887b0
spec:
rules:
- when:
- key: request.headers[kubeflow-userid]
values:
- admin@example.com
- when:
- key: source.namespace
values:
- example-test
- to:
- operation:
paths:
- /healthz
- /metrics
- /wait-for-drain
- from:
- source:
principals:
- cluster.local/ns/kubeflow/sa/notebook-controller-service-account
to:
- operation:
methods:
- GET
paths:
- '*/api/kernels'
---
- apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"security.istio.io/v1beta1","kind":"AuthorizationPolicy","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"dkf-core--deploykf-profiles-generator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"deploykf-profiles-generator","helm.sh/chart":"deploykf-profiles-generator-1.0.0"},"name":"ns-owner-access-istio--override","namespace":"example-test"},"spec":{"action":"DENY","rules":[{"from":[{"source":{"notPrincipals":["cluster.local/ns/deploykf-istio-gateway/sa/deploykf-gateway","cluster.local/ns/kubeflow/sa/ml-pipeline-ui"]}}],"when":[{"key":"request.headers[kubeflow-userid]","values":["admin@example.com"]}]}]}}
creationTimestamp: "2024-04-26T08:58:12Z"
generation: 5
labels:
app.kubernetes.io/instance: dkf-core--deploykf-profiles-generator
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: deploykf-profiles-generator
helm.sh/chart: deploykf-profiles-generator-1.0.0
name: ns-owner-access-istio--override
namespace: example-test
resourceVersion: "42389285"
uid: cb8ea6a5-ca07-4809-9583-65a85ac7af73
spec:
action: DENY
rules:
- from:
- source:
notPrincipals:
- cluster.local/ns/deploykf-istio-gateway/sa/deploykf-gateway
- cluster.local/ns/kubeflow/sa/ml-pipeline-ui
when:
- key: request.headers[kubeflow-userid]
values:
- admin@example.com
---
- apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"security.istio.io/v1beta1","kind":"AuthorizationPolicy","metadata":{"annotations":{"role":"edit","user":"user1@example.eu"},"labels":{"app.kubernetes.io/instance":"dkf-core--deploykf-profiles-generator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"deploykf-profiles-generator","helm.sh/chart":"deploykf-profiles-generator-1.0.0"},"name":"user-user1-example-eu-clusterrole-edit","namespace":"example-test"},"spec":{"rules":[{"from":[{"source":{"principals":["cluster.local/ns/deploykf-istio-gateway/sa/deploykf-gateway","cluster.local/ns/kubeflow/sa/ml-pipeline-ui"]}}],"when":[{"key":"request.headers[kubeflow-userid]","values":["user1@example.eu"]}]}]}}
role: edit
user: user1@example.eu
creationTimestamp: "2024-04-26T08:58:12Z"
generation: 30
labels:
app.kubernetes.io/instance: dkf-core--deploykf-profiles-generator
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: deploykf-profiles-generator
helm.sh/chart: deploykf-profiles-generator-1.0.0
name: user-user1-example-eu-clusterrole-edit
namespace: example-test
resourceVersion: "42429864"
uid: 88d3c64e-7b24-4b6b-a727-60d5d1a9553b
spec:
rules:
- from:
- source:
principals:
- cluster.local/ns/deploykf-istio-gateway/sa/deploykf-gateway
- cluster.local/ns/kubeflow/sa/ml-pipeline-ui
when:
- key: request.headers[kubeflow-userid]
values:
- user1@example.eu
---
- apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"security.istio.io/v1beta1","kind":"AuthorizationPolicy","metadata":{"annotations":{"role":"edit","user":"user2@example.eu"},"labels":{"app.kubernetes.io/instance":"dkf-core--deploykf-profiles-generator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"deploykf-profiles-generator","helm.sh/chart":"deploykf-profiles-generator-1.0.0"},"name":"user-user2-example-eu-clusterrole-edit","namespace":"example-test"},"spec":{"rules":[{"from":[{"source":{"principals":["cluster.local/ns/deploykf-istio-gateway/sa/deploykf-gateway","cluster.local/ns/kubeflow/sa/ml-pipeline-ui"]}}],"when":[{"key":"request.headers[kubeflow-userid]","values":["user2@example.eu"]}]}]}}
role: edit
user: user2@example.eu
creationTimestamp: "2024-04-26T08:58:12Z"
generation: 1
labels:
app.kubernetes.io/instance: dkf-core--deploykf-profiles-generator
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: deploykf-profiles-generator
helm.sh/chart: deploykf-profiles-generator-1.0.0
name: user-user2-example-eu-clusterrole-edit
namespace: example-test
resourceVersion: "41640621"
uid: 2fadbe02-66cf-4de8-b145-3fea9a1e8b4b
spec:
rules:
- from:
- source:
principals:
- cluster.local/ns/deploykf-istio-gateway/sa/deploykf-gateway
- cluster.local/ns/kubeflow/sa/ml-pipeline-ui
when:
- key: request.headers[kubeflow-userid]
values:
- user2@example.eu
---
- apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"security.istio.io/v1beta1","kind":"AuthorizationPolicy","metadata":{"annotations":{"role":"edit","user":"user3@example.eu"},"labels":{"app.kubernetes.io/instance":"dkf-core--deploykf-profiles-generator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"deploykf-profiles-generator","helm.sh/chart":"deploykf-profiles-generator-1.0.0"},"name":"user-user3-example-eu-clusterrole-edit","namespace":"example-test"},"spec":{"rules":[{"from":[{"source":{"principals":["cluster.local/ns/deploykf-istio-gateway/sa/deploykf-gateway","cluster.local/ns/kubeflow/sa/ml-pipeline-ui"]}}],"when":[{"key":"request.headers[kubeflow-userid]","values":["user3@example.eu"]}]}]}}
role: edit
user: user3@example.eu
creationTimestamp: "2024-04-26T08:58:12Z"
generation: 1
labels:
app.kubernetes.io/instance: dkf-core--deploykf-profiles-generator
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: deploykf-profiles-generator
helm.sh/chart: deploykf-profiles-generator-1.0.0
name: user-user3-example-eu-clusterrole-edit
namespace: example-test
resourceVersion: "41640623"
uid: c5c19f93-fcd5-468c-9f49-a99d31607ffa
spec:
rules:
- from:
- source:
principals:
- cluster.local/ns/deploykf-istio-gateway/sa/deploykf-gateway
- cluster.local/ns/kubeflow/sa/ml-pipeline-ui
when:
- key: request.headers[kubeflow-userid]
values:
- user3@example.eu |
|
|
Ok, I had some time again to have a look. 1: My own existing email: user1@example.eu I also tried:
After some more trying I think I came something closer to the root of the problem but still unsure if my solution is acceptable. The ns-owner-access-istio policy (generated) looks like this: apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
annotations:
role: admin
user: admin@example.com
creationTimestamp: "2024-05-08T08:33:38Z"
generation: 2
name: ns-owner-access-istio
namespace: example-test
ownerReferences:
- apiVersion: kubeflow.org/v1
blockOwnerDeletion: true
controller: true
kind: Profile
name: example-test
uid: 9e705b85-91d9-4c56-861a-a739dae2b1c2
resourceVersion: "51547541"
uid: 95451200-168b-49a9-abce-762ef747f7d2
spec:
rules:
- when:
- key: request.headers[kubeflow-userid]
values:
- admin@example.com
- when:
- key: source.namespace
values:
- example-test
- to:
- operation:
paths:
- /healthz
- /metrics
- /wait-for-drain
- from:
- source:
principals:
- cluster.local/ns/kubeflow/sa/notebook-controller-service-account
to:
- operation:
methods:
- GET
paths:
- '*/api/kernels'I suppose now the clusterrole edit policy for my user should make sure the fetching from s3 works. apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"security.istio.io/v1beta1","kind":"AuthorizationPolicy","metadata":{"annotations":{"role":"edit","user":"user1@example.eu"},"labels":{"app.kubernetes.io/instance":"dkf-core--deploykf-profiles-generator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"deploykf-profiles-generator","helm.sh/chart":"deploykf-profiles-generator-1.0.0"},"name":"user-user1-example-eu-clusterrole-edit","namespace":"example-test"},"spec":{"rules":[{"from":[{"source":{"principals":["cluster.local/ns/deploykf-istio-gateway/sa/deploykf-gateway","cluster.local/ns/kubeflow/sa/ml-pipeline-ui"]}}],"when":[{"key":"request.headers[kubeflow-userid]","values":["user1@example.eu"]}]}]}}
role: edit
user: user1@example.eu
creationTimestamp: "2024-05-08T09:43:59Z"
generation: 7
labels:
app.kubernetes.io/instance: dkf-core--deploykf-profiles-generator
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: deploykf-profiles-generator
helm.sh/chart: deploykf-profiles-generator-1.0.0
name: user-user1-example-eu-clusterrole-edit
namespace: example-test
resourceVersion: "51657299"
uid: f0605b72-4e72-4766-b8b6-1de4193e8f2e
spec:
rules:
- from:
- source:
principals:
- cluster.local/ns/deploykf-istio-gateway/sa/deploykf-gateway
- cluster.local/ns/kubeflow/sa/ml-pipeline-ui
when:
- key: request.headers[kubeflow-userid]
values:
- user1@example.euBut somehow it seems too restrictive for the S3 fetching to work in our setup. When I add an extra policy manually allowing traffic from ml-pipeline-ui (as restrictive as possible) my problem is solved: ## !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
## WARNING DO NOT USE, MESSES WITH AUTH ##
## !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: test-work-around-policy
namespace: example-test
spec:
rules:
- when:
- key: request.headers[kubeflow-userid]
values:
- user1@example.eu
to:
- operation:
methods: ["GET"]
paths: ["/pipeline/artifacts/*"]
from:
- source:
ipBlocks: ["100.96.0.0/16"] ## covers our cluster podCIDRs
selector:
matchLabels:
app: ml-pipeline-ui-artifactI understand this policy may not be restrictive enough for best practice production but for us it is good enough for now to proceed with testing. Hopefully next releases will magically fix the issue without having to use this work around. |
|
Some extra info that might be of interest if you still want to analyze this further. If I enable istio debug logging om my ml-pipeline-ui-artifact pod and perform a request I noticed the 'x-forwarded-client-cert' header is missing. I suspect that without this header content istio is not able to validate from.source.namespaces and from.source.principals and thus denies the traffic (?). Logging 403 request: |
|
@bobbeeke did you manage to figure this out? If so, could you share how you did? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Checks
deployKF Version
v0.1.4
Kubernetes Version
Description
I try to setup a minimal pipeline which at one point should fetch visualization data from S3 (AWS) and show it in the visualization tab in the Kubeflow UI. The pipeline finishes, however the visualization tab shows: "There are no visualizations in this step."
When I inspect (network) my browser and click on this visualization tab I see this 403 (forbidden):
For some reason it fails to get permission to fetch the file from my AWS S3 bucket.
When I put the URL in my browser directly I get the same 403 and response:
The file mlpipeline-ui-metadata.tgz exists on my bucket in the right path. So my pipeline is able to write to S3 with no problem apparently. I tried opening up my bucket IAM permissions further to allow everything but this also does not seem to help.
I'm a little bit stuck here. Not sure if this is some Istio related restriction or a AWS bucket restriction I am missing.
I followed the DeployKF docs about setting up S3 connectivity as good as possible.
Some guidance as where to look for would be appreciated.
Relevant Logs
No response
deployKF Values (Optional)
The text was updated successfully, but these errors were encountered: