after local install some images are tried to be pulled from registry

[vsaase]

I followed the instructions for the local install, which finished without errors.
Now with the command
microk8s.kubectl get pods --all-namespaces
I get the following output:

NAMESPACE     NAME                                    READY   STATUS                  RESTARTS   AGE
base          landingpage-bc44bdccc-2rdbc             1/1     Running                 0          33m
flow          airflow-fc8658bfd-2lcjt                 0/2     Init:0/2                0          33m
flow          ctp-5774599964-qjjp2                    0/1     ImagePullBackOff        0          33m
flow          postgres-airflow-b6f5d88d9-4tvxt        0/1     ImagePullBackOff        0          33m
kube-system   coredns-588fd544bf-vhqzx                1/1     Running                 1          18h
kube-system   error-pages-6c76f58767-tpfvb            1/1     Running                 0          33m
kube-system   keycloak-6d58c4749-pdjz7                0/1     Init:0/1                0          33m
kube-system   kube-helm-deployment-7c549dccc4-2nj24   0/1     Init:ImagePullBackOff   0          33m
kube-system   kube-state-metrics-658b4868bd-bvpkg     1/1     Running                 0          33m
kube-system   kubernetes-dashboard-7d4df94679-lhlp2   1/1     Running                 0          33m
kube-system   louketo-74667d46ff-8vjj9                0/1     Init:0/1                0          33m
kube-system   postgres-keycloak-8898ddfc6-7btd8       0/1     ImagePullBackOff        0          33m
kube-system   preinstall-extensions-init-7s6l2        0/2     Init:0/2                0          33m
kube-system   traefik-796f6f9564-9dr2q                1/1     Running                 0          33m
kube-system   update-extensions-init-74xw6            0/1     Init:0/1                0          33m
meta          elastic-meta-de-885cbb85d-65t74         1/1     Running                 0          33m
meta          init-meta-7t8jx                         0/1     Completed               0          33m
meta          kibana-meta-de-65c4ff78-jsnhc           1/1     Running                 0          33m
monitoring    alertmanager-c44dd9669-fdtc5            1/1     Running                 0          33m
monitoring    grafana-5f4c5f5ffd-6xrhr                1/1     Running                 0          33m
monitoring    prometheus-f58db5bff-bgvgk              1/1     Running                 0          33m
store         dcm4chee-7bddb8776c-mcm9z               1/1     Running                 0          33m
store         dicom-init-ksbtw                        0/1     Init:0/1                0          33m
store         ldap-d95dbcc74-wff5m                    1/1     Running                 0          33m
store         minio-deployment-76d8b4d49-gxnqm        0/1     Init:0/1                0          33m
store         minio-init-7k2xf                        0/1     Init:0/1                0          33m
store         ohif-c7b7d8b76-688fb                    0/1     ImagePullBackOff        0          33m
store         postgres-dcm4che-f85d85f56-48t6w        1/1     Running                 0          33m

For example with kubectl describe pod --namespace flow ctp-5774599964-qjjp2 I see:
Failed to pull image "local/ctp:0.1.2-vdev": rpc error: code = Unknown desc = failed to resolve image "docker.io/local/ctp:0.1.2-vdev": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
This image is already available locally, so shouldn't be pulled.

[fedorov]

I have the same problem with 0.1.2-november. I build those images locally, so I should have them. I also have them in the remote registry, they were successfully pushed.

cc: @nadya-58

[fedorov]

I really am operating at the level of extremely limited understanding and keywords, but...

Running kubectl describe pod, I realized I have a different error - "Temporary failure of name resolution". Then I tried the following:

$ nslookup kubernetes.default
Server: 127.0.0.1
Address: 127.0.0.53#53

** server can't find kubernetes.default: NXDOMAIN **

Which brought me to this issue: kubernetes/kubernetes#66924, and I am now trying the build script after updating busybox Dockerfule to use the latest stable release.

[fedorov]

Switching to 1.33 for busybox did not address the failure of name resolution problem for me.

[klauskades]

Dear @fedorov and @vsaase can you please provide us the output of microk8s ctr images ls and docker images. In case you are low on space it might be the case that images are "replaced" on microk8s and therefore missing. That is why I want to check it the images are really there in the microk8s storage. Sorry for the late reply and best Klaus

[fedorov]

Correction: I do not have the same issue as reported by @vsaase. Mine fail with different issues:

Let me know if I should create a new ticket instead of investigating this here.

@klauskades here are the items you requested: https://gist.github.com/fedorov/dde4720673c6133f468074ac02e86dca