Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot load dashboard menu link for KF1.5 #2171

Closed
huanghy95 opened this issue Mar 15, 2022 · 5 comments
Closed

Cannot load dashboard menu link for KF1.5 #2171

huanghy95 opened this issue Mar 15, 2022 · 5 comments

Comments

@huanghy95
Copy link

Describe the bug

I installed KF1.5 and all pods seem running well, but when I visited the dashboard, the menu link cannot be loaded.

Pods in kubeflow ns:

root@MLOps-1:~/kubeflow1.5/manifests-1.5.0/hhy/multi-tenancy# kubectl get po -n kubeflow
NAME                                                         READY   STATUS    RESTARTS   AGE
admission-webhook-deployment-7df7558c67-5vg9r                1/1     Running   0          27h
cache-deployer-deployment-6f4bcc969-d47vj                    2/2     Running   0          27h
cache-server-7cc6cbbf55-jd6wq                                2/2     Running   0          27h
centraldashboard-5dd4f57bbd-2l59n                            2/2     Running   0          27h
jupyter-web-app-deployment-8d96db4cd-xmz2h                   1/1     Running   0          27h
katib-controller-58ddb4b856-fd6b7                            1/1     Running   0          27h
katib-db-manager-6df878f5b8-6992l                            1/1     Running   0          27h
katib-mysql-6dcb447c6f-6l6hm                                 1/1     Running   0          27h
katib-ui-f787b9d88-f69dp                                     1/1     Running   0          27h
kfserving-controller-manager-0                               2/2     Running   0          27h
kfserving-models-web-app-5d6cd6b5dd-84vxj                    2/2     Running   0          27h
kserve-models-web-app-6f45769bb6-99g7s                       2/2     Running   0          27h
kubeflow-pipelines-profile-controller-df66bc7f4-lmcz9        1/1     Running   0          27h
metacontroller-0                                             1/1     Running   0          27h
metadata-envoy-deployment-7b847ff6c5-lk2jc                   1/1     Running   0          27h
metadata-grpc-deployment-6f6f7776c5-7ggtv                    2/2     Running   3          27h
metadata-writer-7d57dbd567-cn24b                             2/2     Running   5          27h
minio-5b65df66c9-6qcxn                                       2/2     Running   0          27h
ml-pipeline-6bccbd7bd-qwvjr                                  2/2     Running   0          27h
ml-pipeline-persistenceagent-87b6888c4-zr82d                 2/2     Running   0          27h
ml-pipeline-scheduledworkflow-665847bb9-wld5z                2/2     Running   0          27h
ml-pipeline-ui-68cc764f66-mk2sg                              2/2     Running   0          27h
ml-pipeline-viewer-crd-68777557fb-mrz79                      2/2     Running   1          27h
ml-pipeline-visualizationserver-58ccb76855-b2wln             2/2     Running   0          27h
mysql-f7b9b7dd4-8r6x4                                        2/2     Running   0          27h
notebook-controller-deployment-5d9c6c656c-bdgn5              2/2     Running   1          27h
profiles-deployment-78ffd649f5-5lvzs                         3/3     Running   1          27h
tensorboard-controller-controller-manager-6848cb6846-7wzcc   3/3     Running   1          27h
tensorboards-web-app-deployment-7c5db448d7-424sv             1/1     Running   0          27h
training-operator-7b8cc9865d-4bqjx                           1/1     Running   0          27h
volumes-web-app-deployment-87484c848-729wt                   1/1     Running   0          27h
workflow-controller-6bf87db995-787w9                         2/2     Running   2          27h

when visiting dashboard:
2383126a-6b68-492f-a8e8-df97dd13c1bc

logs in pod centraldashboard-5dd4f57bbd-2l59n:

> kubeflow-centraldashboard@0.0.2 start /app
> npm run serve


> kubeflow-centraldashboard@0.0.2 serve /app
> node dist/server.js

Initializing Kubernetes configuration
Unable to fetch Nodes Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1502:34)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket._finishInit (_tls_wrap.js:937:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:711:12) {
  code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
}
Unable to fetch Nodes Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1502:34)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket._finishInit (_tls_wrap.js:937:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:711:12) {
  code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
}
Unable to fetch Application information: Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1502:34)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket._finishInit (_tls_wrap.js:937:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:711:12) {
  code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
}
"other" is not a supported platform for Metrics
Using Profiles service at http://profiles-kfam.kubeflow:8081/kfam
Server listening on port http://localhost:8082 (in production mode)
Unable to fetch Events for kubeflow-user-example-com: Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1502:34)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket._finishInit (_tls_wrap.js:937:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:711:12) {
  code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
}
Unable to fetch ConfigMap: Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1502:34)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket._finishInit (_tls_wrap.js:937:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:711:12) {
  code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
}
Unable to fetch Nodes Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1502:34)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket._finishInit (_tls_wrap.js:937:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:711:12) {
  code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
}
Unable to fetch Application information: Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1502:34)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket._finishInit (_tls_wrap.js:937:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:711:12) {
  code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
}
Unable to fetch Events for kubeflow-user-example-com: Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1502:34)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket._finishInit (_tls_wrap.js:937:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:711:12) {
  code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
}
Unable to fetch ConfigMap: Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1502:34)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket._finishInit (_tls_wrap.js:937:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:711:12) {
  code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
}
Unable to fetch Events for kubeflow-user-example-com: Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1502:34)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket._finishInit (_tls_wrap.js:937:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:711:12) {
  code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
}
Unable to fetch ConfigMap: Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1502:34)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket._finishInit (_tls_wrap.js:937:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:711:12) {
  code: 'DEPTH_ZERO_SELF_SIGNED_CERT'
}

but I can visit the components normally through urls, they work well.

Tested environment

  • Linux: Ubuntu 20.04
  • k8s version: 1.19
  • kubeflow version: 1.5
  • Docker Engine 20.10.19
@kubeflow-bot kubeflow-bot added this to To Do in Needs Triage Mar 15, 2022
@huanghy95 huanghy95 changed the title Cannot load dashboard menu link Cannot load dashboard menu link for KF1.5 Mar 15, 2022
@kimwnasptd
Copy link
Member

Hi @huanghy95, looking at the logs it seems that the CentralDashboard fails to communicate with KFAM.

We have an AuthorizationPolicy for only allowing the CentralDashboard to talk to KFAP (resides in Profiles Controller) https://github.com/kubeflow/kubeflow/pull/6077/files#diff-532bf0a4d11747124ef0280e12efb25d5529e3260ed624ef828d82c89ae42082

The logs report a TLS issue, so maybe there's something wrong with your Istio installation and how it handles the secure incluster communication between the two workloads?

@huanghy95
Copy link
Author

Hi @huanghy95, looking at the logs it seems that the CentralDashboard fails to communicate with KFAM.

We have an AuthorizationPolicy for only allowing the CentralDashboard to talk to KFAP (resides in Profiles Controller) https://github.com/kubeflow/kubeflow/pull/6077/files#diff-532bf0a4d11747124ef0280e12efb25d5529e3260ed624ef828d82c89ae42082

The logs report a TLS issue, so maybe there's something wrong with your Istio installation and how it handles the secure incluster communication between the two workloads?

Thanks for your help! @kimwnasptd
But the logs in istio seem normal. It seems that the only problem is the /api/dashboard-links

pods in ns istio-system:

root@MLOps-1:~# kubectl get po -n istio-system 
NAME                                     READY   STATUS    RESTARTS   AGE
authservice-0                            1/1     Running   0          3d19h
cluster-local-gateway-7796d7bc87-qdgfb   1/1     Running   0          3d19h
istio-ingressgateway-64b7899489-762gn    1/1     Running   0          3d19h
istiod-5d9bbb9cb4-b5kn4                  1/1     Running   0          3d19h

logs in authservice-0 during login:

time="2022-03-17T03:05:52Z" level=info msg="URI is whitelisted. Accepted without authorization." ip=172.20.1.4 request="/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY0NzQ4NjM1MnxFd3dBRUZSUE1tOVhWMlU0TTI1bVIyNTRWVk09fAU8Fh4dC2vVNJVf0x41v2FIH_YYwSUeyDAyIEN9cuy2"
time="2022-03-17T03:05:52Z" level=info msg="URI is whitelisted. Accepted without authorization." ip=172.20.1.4 request="/dex/auth/local?req=llzapdztc3sq2dow6v6nbemp4"
time="2022-03-17T03:07:07Z" level=info msg="URI is whitelisted. Accepted without authorization." ip=172.20.1.4 request="/dex/auth/local?req=llzapdztc3sq2dow6v6nbemp4"
time="2022-03-17T03:07:08Z" level=info msg="URI is whitelisted. Accepted without authorization." ip=172.20.1.4 request="/dex/approval?req=llzapdztc3sq2dow6v6nbemp4"
time="2022-03-17T03:07:08Z" level=info msg="Login validated with ID token, redirecting." ip=172.20.1.4 request="/login/oidc?code=i6meil2tfokhgpg4f6wtt6oiy&state=MTY0NzQ4NjM1MnxFd3dBRUZSUE1tOVhWMlU0TTI1bVIyNTRWVk09fAU8Fh4dC2vVNJVf0x41v2FIH_YYwSUeyDAyIEN9cuy2"

logs in istio-ingressgateway during login:

[2022-03-17T03:05:52.396Z] "GET / HTTP/1.1" 302 - ext_authz_denied - "-" 0 269 2 1 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "e635f8aa-3148-9871-a191-93cd694a9f79" "33.33.33.191:30556" "-" outbound|80||centraldashboard.kubeflow.svc.cluster.local - 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:05:52.584Z] "GET /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY0NzQ4NjM1MnxFd3dBRUZSUE1tOVhWMlU0TTI1bVIyNTRWVk09fAU8Fh4dC2vVNJVf0x41v2FIH_YYwSUeyDAyIEN9cuy2 HTTP/1.1" 302 - via_upstream - "-" 0 68 9 8 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "d6cab13e-65d6-9a54-8435-3ce06dce6cab" "33.33.33.191:30556" "172.20.0.5:5556" outbound|5556||dex.auth.svc.cluster.local 172.20.1.4:51274 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:05:52.652Z] "GET /dex/auth/local?req=llzapdztc3sq2dow6v6nbemp4 HTTP/1.1" 200 - via_upstream - "-" 0 1497 10 10 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "ced07a3d-3a97-9287-b5be-17ad61101cb2" "33.33.33.191:30556" "172.20.0.5:5556" outbound|5556||dex.auth.svc.cluster.local 172.20.1.4:51274 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:07.829Z] "POST /dex/auth/local?req=llzapdztc3sq2dow6v6nbemp4 HTTP/1.1" 303 - via_upstream - "-" 42 0 377 375 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "c4418a01-c636-9c0a-9aab-a94a04c38324" "33.33.33.191:30556" "172.20.0.5:5556" outbound|5556||dex.auth.svc.cluster.local 172.20.1.4:51274 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:08.216Z] "GET /dex/approval?req=llzapdztc3sq2dow6v6nbemp4 HTTP/1.1" 303 - via_upstream - "-" 0 176 12 11 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "31c27878-e1fc-9da3-be83-5b217eec9574" "33.33.33.191:30556" "172.20.0.5:5556" outbound|5556||dex.auth.svc.cluster.local 172.20.1.4:51274 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:08.337Z] "GET /login/oidc?code=i6meil2tfokhgpg4f6wtt6oiy&state=MTY0NzQ4NjM1MnxFd3dBRUZSUE1tOVhWMlU0TTI1bVIyNTRWVk09fAU8Fh4dC2vVNJVf0x41v2FIH_YYwSUeyDAyIEN9cuy2 HTTP/1.1" 302 - ext_authz_denied - "-" 0 24 363 362 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "6d9450ce-0ab2-97cd-a0ce-95d203cc11f9" "33.33.33.191:30556" "-" outbound|80||centraldashboard.kubeflow.svc.cluster.local - 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:08.727Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 1433 4 3 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "ddf83835-4172-9668-ba75-f3f973e1b086" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45286 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:08.798Z] "GET /webcomponentsjs/webcomponents-loader.js HTTP/1.1" 304 - via_upstream - "-" 0 0 2 1 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "b7e42cef-a7ab-959c-8642-be29c82b62c7" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45286 172.20.1.4:8080 172.20.0.0:45529 - -
[2022-03-17T03:07:08.797Z] "GET /app.css HTTP/1.1" 304 - via_upstream - "-" 0 0 2 1 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "5811a850-61f8-9ef3-9c26-ed44a23b603f" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45288 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:08.816Z] "GET /vendor.bundle.js HTTP/1.1" 304 - via_upstream - "-" 0 0 1 0 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "6644dc66-641d-9cb1-9040-bc548bf0b57a" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45288 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:08.816Z] "GET /webcomponentsjs/custom-elements-es5-adapter.js HTTP/1.1" 304 - via_upstream - "-" 0 0 1 0 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "a1d33a16-8492-910d-b47b-62275a55b1b8" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45286 172.20.1.4:8080 172.20.0.0:38893 - -
[2022-03-17T03:07:08.870Z] "GET /app.bundle.js HTTP/1.1" 304 - via_upstream - "-" 0 0 1 0 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "184c3b3a-12b5-9bf3-be50-235a9add2458" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45286 172.20.1.4:8080 172.20.0.0:38893 - -
[2022-03-17T03:07:08.870Z] "GET /dashboard_lib.bundle.js HTTP/1.1" 304 - via_upstream - "-" 0 0 1 1 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "64871eaf-e5be-9562-9de3-d5cc24eb30f0" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45288 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:09.240Z] "GET /api/dashboard-links HTTP/1.1" 500 - via_upstream - "-" 0 43 12 11 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "47389b8e-bb62-984a-94fa-ce8664b7a996" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45286 172.20.1.4:8080 172.20.0.0:38893 - -
[2022-03-17T03:07:09.236Z] "GET /api/workgroup/exists HTTP/1.1" 304 - via_upstream - "-" 0 0 20 19 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "cb65cdf9-9cae-96db-80de-cf6743c14ccb" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45288 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:09.240Z] "GET /pipeline/apis/v1beta1/pipelines?page_size=5&sort_by=created_at%20desc HTTP/1.1" 200 - via_upstream - "-" 0 4168 23 22 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "a13c5e19-6149-9f0e-b305-7c527664b641" "33.33.33.191:30556" "172.20.3.7:3000" outbound|80||ml-pipeline-ui.kubeflow.svc.cluster.local 172.20.1.4:45414 172.20.1.4:8080 172.20.0.0:45529 - -
[2022-03-17T03:07:09.262Z] "GET /pipeline/apis/v1beta1/runs?page_size=5&sort_by=created_at%20desc&resource_reference_key.type=NAMESPACE&resource_reference_key.id=undefined HTTP/1.1" 403 - via_upstream - "-" 0 1365 14 13 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "9e54a890-eda4-9284-969d-80d8be729a2e" "33.33.33.191:30556" "172.20.3.7:3000" outbound|80||ml-pipeline-ui.kubeflow.svc.cluster.local 172.20.1.4:45414 172.20.1.4:8080 172.20.0.0:18751 - -
[2022-03-17T03:07:09.340Z] "GET /api/workgroup/env-info HTTP/1.1" 304 - via_upstream - "-" 0 0 10 9 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "7359e0ad-2a83-95a8-949f-fe792f65a58a" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45288 172.20.1.4:8080 172.20.0.0:18751 - -
[2022-03-17T03:07:09.380Z] "GET /api/activities/kubeflow-user-example-com HTTP/1.1" 304 - via_upstream - "-" 0 0 9 8 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "41e4e395-743a-9eeb-b5ed-b3200d820307" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45288 172.20.1.4:8080 172.20.0.0:18751 - -
[2022-03-17T03:07:09.400Z] "GET /api/workgroup/get-contributors/kubeflow-user-example-com HTTP/1.1" 304 - via_upstream - "-" 0 0 5 4 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "8dc9fc96-8aec-9b2e-a197-0b9afc91c04d" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45288 172.20.1.4:8080 172.20.0.0:38893 - -
[2022-03-17T03:07:09.400Z] "GET /pipeline/apis/v1beta1/runs?page_size=5&sort_by=created_at%20desc&resource_reference_key.type=NAMESPACE&resource_reference_key.id=kubeflow-user-example-com HTTP/1.1" 200 - via_upstream - "-" 0 71621 29 25 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "163a73c2-e034-9e68-961c-388b0df2ce41" "33.33.33.191:30556" "172.20.3.7:3000" outbound|80||ml-pipeline-ui.kubeflow.svc.cluster.local 172.20.1.4:45414 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:09.400Z] "GET /jupyter/api/namespaces/kubeflow-user-example-com/notebooks HTTP/1.1" 200 - via_upstream - "-" 0 1273 67 66 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "2c3734e2-d4ef-9b9c-8cd3-d8ed9c2caa4c" "33.33.33.191:30556" "172.20.0.16:5000" outbound|80||jupyter-web-app-service.kubeflow.svc.cluster.local 172.20.1.4:57738 172.20.1.4:8080 172.20.0.0:45529 - -
[2022-03-17T03:07:09.623Z] "GET /assets/favicon.ico HTTP/1.1" 304 - via_upstream - "-" 0 0 1 0 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "165edfba-457d-9806-866d-e541ac028630" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45288 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:17.844Z] "GET /app.css HTTP/1.1" 304 - via_upstream - "-" 0 0 3 1 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "f06c8525-bb61-93ec-9766-644e97e58a01" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45480 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:18.120Z] "GET /vendor.bundle.js.map HTTP/1.1" 200 - via_upstream - "-" 0 1248099 19 2 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "090a5f94-1611-9930-adbf-7325b84c9669" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45480 172.20.1.4:8080 172.20.0.0:43671 - -
[2022-03-17T03:07:18.139Z] "GET /app.bundle.js.map HTTP/1.1" 200 - via_upstream - "-" 0 114988 4 2 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "83fdb8f9-bdac-9383-bfae-67b146f218c0" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45480 172.20.1.4:8080 172.20.0.0:45529 - -
[2022-03-17T03:07:18.140Z] "GET /dashboard_lib.bundle.js.map HTTP/1.1" 200 - via_upstream - "-" 0 2896 4 3 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "724bcbda-467d-93a1-bf83-5b3dbb09a4b8" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45492 172.20.1.4:8080 172.20.0.0:38893 - -
[2022-03-17T03:07:18.140Z] "GET /app.css.map HTTP/1.1" 200 - via_upstream - "-" 0 36880 5 4 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "b305c7b2-f64c-9d40-aca4-e974b5a24956" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45490 172.20.1.4:8080 172.20.0.0:18751 - -

the abnormal log might be :

[2022-03-17T03:07:09.240Z] "GET /api/dashboard-links HTTP/1.1" 500 - via_upstream - "-" 0 43 12 11 "172.20.0.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" "47389b8e-bb62-984a-94fa-ce8664b7a996" "33.33.33.191:30556" "172.20.1.17:8082" outbound|80||centraldashboard.kubeflow.svc.cluster.local 172.20.1.4:45286 172.20.1.4:8080 172.20.0.0:38893 - -

@huanghy95
Copy link
Author

Add NODE_TLS_REJECT_UNAUTHORIZED = "0" as the env of centraldashboard pods can "fix" this problem.
can also see request/request#418
I will close this issue

Needs Triage automation moved this from To Do to Closed Apr 5, 2022
@kubeflow-bot kubeflow-bot removed this from Closed in Needs Triage Apr 5, 2022
@jaffe-fly
Copy link

kubeflow 1.6.1 also have this problem,and no NODE_TLS_REJECT_UNAUTHORIZED env var ,so how to modify

@jaffe-fly
Copy link

jaffe-fly commented Nov 4, 2022 •
edited

kubeflow 1.6.1 also have this problem,and no NODE_TLS_REJECT_UNAUTHORIZED env var ,so how to modify

apps/centraldashboard/upstream/base/deployment.yaml here to add env

After entering, some ports keep reporting errors，so above is a bad Solution

@diankasileymane diankasileymane mentioned this issue Jul 27, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kimwnasptd @jaffe-fly @huanghy95