Problem deploying ORC8R on Minikube

[bruneves2002]

Hello, I deployed the orchestrator on minikube and I get problem in some pods CrashLoopBackOff.
It is the first time deployed orc8r on minikube.
I am not sure there are any more network configuration missing, I could not find any reference.

I am not sure the problem is at nginx, and I am not find how to fix it.
I need some help.

[tapasmishra]

It would be helpful if you could provide the output of the "describe" command for the pods that are currently in a "CrashLoopBackOff" state.

[bruneves2002]

Yes, here are the logs of the describe pods:

bcoelho@jgtvsr04:~$ kubectl -n orc8r describe pods nms-nginx-proxy-5bc7cc4d9d-h5rg5
Name: nms-nginx-proxy-5bc7cc4d9d-h5rg5
Namespace: orc8r
Priority: 0
Service Account: orc8r
Node: minikube/192.168.49.2
Start Time: Tue, 05 Sep 2023 02:04:25 +0000
Labels: app.kubernetes.io/component=nginx
app.kubernetes.io/instance=orc8r
app.kubernetes.io/name=nms
pod-template-hash=5bc7cc4d9d
release_group=orc8r
Annotations: chart-version: 0.1.12
release-name: orc8r
Status: Running
IP: 10.244.0.19
IPs:
IP: 10.244.0.19
Controlled By: ReplicaSet/nms-nginx-proxy-5bc7cc4d9d
Containers:
nms-nginx:
Container ID: docker://485fbc52e08e59e2a7aa1a86bf4c2681c5d7d725124b5ddfb3f83b8d3778a27b
Image: nginx:latest
Image ID: docker-pullable://nginx@sha256:104c7c5c54f2685f0f46f3be607ce60da7085da3eaa5ad22d3d9f01594295e9c
Port: 443/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 06 Sep 2023 11:24:50 +0000
Finished: Wed, 06 Sep 2023 11:24:50 +0000
Ready: False
Restart Count: 395
Environment:
Mounts:
/etc/nginx/conf.d/nginx_proxy_ssl.conf from nginx-proxy-etc (rw,path="nginx_proxy_ssl.conf")
/etc/nginx/conf.d/nms_nginx.key.pem from orc8r-secrets-certs (ro,path="nms_nginx.key.pem")
/etc/nginx/conf.d/nms_nginx.pem from orc8r-secrets-certs (ro,path="nms_nginx.pem")
/var/run/secrets/kubernetes.io/serviceaccount from orc8r-token-zljdr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
nginx-proxy-etc:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: nginx-proxy-etc
Optional: false
orc8r-secrets-certs:
Type: Secret (a volume populated by a Secret)
SecretName: orc8r-secrets-certs
Optional: false
orc8r-token-zljdr:
Type: Secret (a volume populated by a Secret)
SecretName: orc8r-token-zljdr
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Pulled 6m15s (x394 over 33h) kubelet Container image "nginx:latest" already present on machine
Warning BackOff 78s (x9238 over 33h) kubelet Back-off restarting failed container
bcoelho@jgtvsr04:~$

bcoelho@jgtvsr04:~$ kubectl -n orc8r describe pods orc8r-bootstrapper-b995c97b7-r7rdv
Name: orc8r-bootstrapper-b995c97b7-r7rdv
Namespace: orc8r
Priority: 0
Service Account: default
Node: minikube/192.168.49.2
Start Time: Tue, 05 Sep 2023 02:04:26 +0000
Labels: app.kubernetes.io/component=bootstrapper
app.kubernetes.io/instance=orc8r
app.kubernetes.io/name=orc8r
pod-template-hash=b995c97b7
Annotations: chart-version: 1.8.0
release-name: orc8r
Status: Running
IP: 10.244.0.17
IPs:
IP: 10.244.0.17
Controlled By: ReplicaSet/orc8r-bootstrapper-b995c97b7
Containers:
bootstrapper:
Container ID: docker://492f8c5f0c19570017f214c99c64fa7126b7b50277a281958df50a0176786bff
Image: registry.hub.docker.com/bruneves2002/controller:latest
Image ID: docker-pullable://registry.hub.docker.com/bruneves2002/controller@sha256:89c316d90e6637f6f0cebdb9ba03b7d72cfb0b2731b0be2443e91b466fcd2b15
Ports: 9088/TCP, 9188/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/usr/bin/envdir
Args:
/var/opt/magma/envdir
/var/opt/magma/bin/bootstrapper
-cak=/var/opt/magma/certs/bootstrapper.key
-logtostderr=true
-v=0
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Wed, 06 Sep 2023 11:29:44 +0000
Finished: Wed, 06 Sep 2023 11:29:44 +0000
Ready: False
Restart Count: 396
Liveness: tcp-socket :9088 delay=10s timeout=1s period=30s #success=1 #failure=3
Readiness: tcp-socket :9088 delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
DATABASE_SOURCE: <set to the key 'postgres.connstr' in secret 'orc8r-controller'> Optional: false
SQL_DRIVER: postgres
SQL_DIALECT: psql
SERVICE_HOSTNAME: (v1:status.podIP)
SERVICE_REGISTRY_MODE: k8s
HELM_RELEASE_NAME: orc8r
SERVICE_REGISTRY_NAMESPACE: orc8r
HELM_VERSION_TAG: 1.8.0
VERSION_TAG: latest
ORC8R_DOMAIN_NAME: magma.test
Mounts:
/var/opt/magma/certs from certs (ro)
/var/opt/magma/configs/orc8r from orc8r-secrets-configs-orc8r-orc8r (ro)
/var/opt/magma/envdir from envdir (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-jq7cb (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
certs:
Type: Secret (a volume populated by a Secret)
SecretName: orc8r-secrets-certs
Optional: false
envdir:
Type: Secret (a volume populated by a Secret)
SecretName: orc8r-secrets-envdir
Optional: false
orc8r-secrets-configs-orc8r-orc8r:
Type: Secret (a volume populated by a Secret)
SecretName: orc8r-secrets-configs-orc8r
Optional: false
default-token-jq7cb:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-jq7cb
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Pulled 5m46s (x395 over 33h) kubelet Container image "registry.hub.docker.com/bruneves2002/controller:latest" already present on machine
Warning BackOff 46s (x9454 over 33h) kubelet Back-off restarting failed container
bcoelho@jgtvsr04:~$

bcoelho@jgtvsr04:~$ kubectl -n orc8r describe pods orc8r-certifier-5d89cf665c-zjrhg
Name: orc8r-certifier-5d89cf665c-zjrhg
Namespace: orc8r
Priority: 0
Service Account: default
Node: minikube/192.168.49.2
Start Time: Tue, 05 Sep 2023 02:04:26 +0000
Labels: app.kubernetes.io/component=certifier
app.kubernetes.io/instance=orc8r
app.kubernetes.io/name=orc8r
pod-template-hash=5d89cf665c
Annotations: chart-version: 1.8.0
release-name: orc8r
Status: Running
IP: 10.244.0.20
IPs:
IP: 10.244.0.20
Controlled By: ReplicaSet/orc8r-certifier-5d89cf665c
Containers:
certifier:
Container ID: docker://545950fb09bb74f054c6e88bf38b03118352b87d32abdd196f86562bb80c9c57
Image: registry.hub.docker.com/bruneves2002/controller:latest
Image ID: docker-pullable://registry.hub.docker.com/bruneves2002/controller@sha256:89c316d90e6637f6f0cebdb9ba03b7d72cfb0b2731b0be2443e91b466fcd2b15
Ports: 9086/TCP, 9186/TCP, 10089/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/usr/bin/envdir
Args:
/var/opt/magma/envdir
/var/opt/magma/bin/certifier
-cac=/var/opt/magma/certs/certifier.pem
-cak=/var/opt/magma/certs/certifier.key
-vpnc=/var/opt/magma/certs/vpn_ca.crt
-vpnk=/var/opt/magma/certs/vpn_ca.key
-run_echo_server=true
-logtostderr=true
-v=0
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Wed, 06 Sep 2023 11:30:12 +0000
Finished: Wed, 06 Sep 2023 11:30:12 +0000
Ready: False
Restart Count: 396
Liveness: tcp-socket :9086 delay=10s timeout=1s period=30s #success=1 #failure=3
Readiness: tcp-socket :9086 delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
DATABASE_SOURCE: <set to the key 'postgres.connstr' in secret 'orc8r-controller'> Optional: false
SQL_DRIVER: postgres
SQL_DIALECT: psql
SERVICE_HOSTNAME: (v1:status.podIP)
SERVICE_REGISTRY_MODE: k8s
HELM_RELEASE_NAME: orc8r
SERVICE_REGISTRY_NAMESPACE: orc8r
HELM_VERSION_TAG: 1.8.0
VERSION_TAG: latest
ORC8R_DOMAIN_NAME: magma.test
Mounts:
/var/opt/magma/certs from certs (ro)
/var/opt/magma/configs/orc8r from orc8r-secrets-configs-orc8r-orc8r (ro)
/var/opt/magma/envdir from envdir (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-jq7cb (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
certs:
Type: Secret (a volume populated by a Secret)
SecretName: orc8r-secrets-certs
Optional: false
envdir:
Type: Secret (a volume populated by a Secret)
SecretName: orc8r-secrets-envdir
Optional: false
orc8r-secrets-configs-orc8r-orc8r:
Type: Secret (a volume populated by a Secret)
SecretName: orc8r-secrets-configs-orc8r
Optional: false
default-token-jq7cb:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-jq7cb
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Pulled 32m (x390 over 33h) kubelet Container image "registry.hub.docker.com/bruneves2002/controller:latest" already present on machine
Warning BackOff 2m24s (x9462 over 33h) kubelet Back-off restarting failed container
bcoelho@jgtvsr04:~$

bcoelho@jgtvsr04:~$ kubectl -n orc8r describe pods orc8r-nginx-866f74b7c8-l297m
Name: orc8r-nginx-866f74b7c8-l297m
Namespace: orc8r
Priority: 0
Service Account: default
Node: minikube/192.168.49.2
Start Time: Tue, 05 Sep 2023 02:04:25 +0000
Labels: app.kubernetes.io/component=nginx-proxy
app.kubernetes.io/instance=orc8r
app.kubernetes.io/name=orc8r
pod-template-hash=866f74b7c8
Annotations: chart-version: 1.8.0
release-name: orc8r
Status: Running
IP: 10.244.0.13
IPs:
IP: 10.244.0.13
Controlled By: ReplicaSet/orc8r-nginx-866f74b7c8
Containers:
orc8r-nginx:
Container ID: docker://c1c541915ff92c154be688baaefc7b4ccb6dbc108c983e3a757db8b54b67a542
Image: registry.hub.docker.com/bruneves2002/nginx:latest
Image ID: docker-pullable://registry.hub.docker.com/bruneves2002/nginx@sha256:91bba350979781bc83910d381ae6d934b1ed80dd9d964a698521f626672fa05f
Ports: 8443/TCP, 8444/TCP, 9443/TCP, 80/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 06 Sep 2023 11:28:56 +0000
Finished: Wed, 06 Sep 2023 11:28:56 +0000
Ready: False
Restart Count: 396
Liveness: tcp-socket :health delay=10s timeout=1s period=30s #success=1 #failure=3
Readiness: tcp-socket :health delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
PROXY_BACKENDS: orc8r.svc.cluster.local
CONTROLLER_HOSTNAME: controller.magma.test
RESOLVER: kube-dns.kube-system.svc.cluster.local valid=10s
SERVICE_REGISTRY_MODE: k8s
SSL_CERTIFICATE: /var/opt/magma/certs/controller.crt
SSL_CERTIFICATE_KEY: /var/opt/magma/certs/controller.key
SSL_CLIENT_CERTIFICATE: /var/opt/magma/certs/certifier.pem
Mounts:
/var/opt/magma/certs from certs (ro)
/var/opt/magma/envdir from envdir (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-jq7cb (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
certs:
Type: Secret (a volume populated by a Secret)
SecretName: orc8r-secrets-certs
Optional: false
envdir:
Type: Secret (a volume populated by a Secret)
SecretName: orc8r-secrets-envdir
Optional: false
default-token-jq7cb:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-jq7cb
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Warning BackOff 3m7s (x9442 over 33h) kubelet Back-off restarting failed container
bcoelho@jgtvsr04:~$

Tell me if you need more logs.
Thank you in advance!

[jordanvrtanoski]

@bruneves2002, Can you please get the logs from the failing containers as well as if there are any other error messages on from the minikube. @ShubhamTatvamasi can you please check this item?

[bruneves2002]

Hello, this is my first time deplying orc8r on minikube, I am a new one in magma. We decided to create a new cluster from zero. @jordanvrtanoski and @ShubhamTatvamasi could you please help me what steps I have to folllow. I don't thik so it is so much clear for me https://magma.github.io/magma/docs/orc8r/dev_minikube . Do you have some additional instructions? We want to deploy magma 1.8.0, and I have installed docker v24.0.6, kubectl v1.28.2 and minikube v1.27.4. Thank you in adcance.

[bruneves2002]

Hello @jordanvrtanoski and @ShubhamTatvamasi I just deployed new cluster but I have same problema. I was following the instructions on the official magma docs page for v1.8.0 https://magma.github.io/magma/docs/orc8r/dev_minikube . I am no able to find the error is the cause of the problem. I tried to install charts from the official chart repositories (Stable: https://linuxfoundation.jfrog.io/artifactory/magma-helm-prod/) I need help thanks!
magma orc8r - deploy on minicube (problem crashloopbackoff - pods status).pdf

[bruneves2002]

I faced different errors when trying kubernetes v1.27. I didn't find how to fix it. May anybody help the issue?

[panyogesh]

Is the issue still observed.

[lucasgonze]

Blocked on #15357. Watch that ticket to get a solution.

This may be fixed in the 1.9 release. Need to test once the branch is cut.