minikube v1.19.0 issues with ingress-nginx validation webhook.

[bhundven]

Minikube v1.19.0 updated ingress-nginx to v0.44.0. For what ever reason (still trying to figure it out), I get the following error while following logs on the awx-operator container:

(item=tower_ingress) => {\"ansible_loop_var\": \"item\", \"changed\": false, \"error\": 500, \"item\": \"tower_ingress\", \"msg\": \"Failed to apply object: b'{\\\"kind\\\":\\\"Status\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{},\\\"status\\\":\\\"Failure\\\",\\\"message\\\":\\\"Internal error occurred: failed calling webhook \\\\\\\\\\\"validate.nginx.ingress.kubernetes.io\\\\\\\\\\\": an error on the server (\\\\\\\\\\\"\\\\\\\\\\\") has prevented the request from succeeding\\\",\\\"reason\\\":\\\"InternalError\\\",\\\"details\\\":{\\\"causes\\\":[{\\\"message\\\":\\\"failed calling webhook \\\\\\\\\\\"validate.nginx.ingress.kubernetes.io\\\\\\\\\\\": an error on the server (\\\\\\\\\\\"\\\\\\\\\\\") has prevented the request from succeeding\\\"}]},\\\"code\\\":500}\\\\n'\", \"reason\": \"Internal Server Error\", \"status\": 500}

This may be related to: kubernetes/ingress-nginx#5401

I was able to work-around the issue by running:

$ kubectl get validatingwebhookconfigurations
$ kubectl delete -A ValidatingWebhookConfiguration [configuration-name]

where in my case with minikube, the [configuration-name] is ingress-nginx-admission.

If I move back to minikube v1.18.1, everything works fine with both 0.8.0 and devel of awx-operator.

To reproduce (macos):

$ minikube config set vm-driver hyperkit
$ minikube start --cpus=4 --memory=8192
$ minikube addons configure ingress
$ minikube addons enable ingress
$ minikube addons enable metrics-server
$ minikube addons enable dashboard
$ kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/devel/deploy/awx-operator.yaml
$ kubectl apply -f awx-deploy.yml

awx-deploy.yml:

---
apiVersion: v1
kind: Namespace
metadata:
  name: awx
  labels:
    name: awx
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
  namespace: awx
  labels:
    name: awx
spec:
  tower_ingress_type: Ingress
  tower_hostname: awx-service.awx.svc.cluster.local
---
apiVersion: v1
kind: Secret
metadata:
  name: awx-admin-password
  namespace: awx
  labels:
    name: awx
stringData:
  password: changeme

Works: Minikube v1.18.1
Fails: Minikube v1.19.0

[mooky31]

Same issue here, thanks @bhundven for the workaround

[laurence-syree]

Same issue here as well, Thanks a bunch @bhundven spent an hour looking for the problem and you probably saved me another 2.

[abcqwertz]

Same issue here, works fine with minikube v1.18.1 and devel of awx-operator.

[UnderGrounder96]

And I thought I was doing something wrong, everything worked fine last week.
Thank you for the workaround, I hope the Ansible team solves this issue ASAP. 👍🏽

[TheBigBear]

Trying to go back to 1.18.1 under Mac OS using brew , but brew removed switch and minikube is not versioned as per brew's understanding of versioning. So how do you guys go back to qinikube 1.18.1 to work mourned this? brew now only supports latest supported versioned version of a software pkg ...

[medyagh]

sorry for this issue, we are aware of this bug, we introduced kubernetes/minikube#10879 that unfortunately was not detected by our integration tests, applogize for all the inconvenience, this will be high priority to be fixed in the version coming next please track kubernetes/minikube#11121 for the updates

[govargo]

Hi awx-operator developers and users,
Sorry for this issue. We will add the workaround by the PR kubernetes/minikube#11189 so that ingress resource can be created.
And we will add integration test pattern to check if the ingress resource can be created.

It seems that the awx-operator uses old ingress version(extensions/v1beta1), but I think kubernetes/minikube#11189 will resolve this issue at this version(I already tested in my local).

awx-operator/roles/installer/templates/tower_ingress.yaml.j2

Lines 3 to 5 in ee9800c

	apiVersion: extensions/v1beta1
	kind: Ingress
	metadata:

Please track kubernetes/minikube#11121 and kubernetes/minikube#11189 for the updates.
Thank you,

[shanemcd]

Hi @govargo - thank you for taking the time to help us out.

FWIW, I tried to update our Ingress object to use networking.k8s.io/v1, but am still seeing the same error. I will try to test the patch you linked, if I can figure out how to build minikube myself. Otherwise I'll just wait until the next version is released.

[shanemcd]

@govargo The installer now gets past the part where it was failing, but when I try to get the service URL from minikube I see:

$ minikube service awx-service --url
😿  service default/awx-service has no node port

[shanemcd]

Ignore my last comment. It seems like something else has broken on our end. Thanks for your help @govargo

[shanemcd]

Just to condense everything down: this will be fixed in the next version of minikube.

[govargo]

Hi shanemcd

I will try to test the patch you linked, if I can figure out how to build minikube myself.

If you want to try patched minikube, you can do by following methods.

Method1: Build latest application
$ mkdir -p $GOPATH/src/k8s.io/
$ cd $GOPATH/src/k8s.io/
$ git clone https://github.com/kubernetes/minikube.git
$ cd minikube
# output file will be generated to ./out directory(e.g. out/minikube)
$ make

Method2: Use built binary
# built binaries are put in object storage
# http://storage.googleapis.com/minikube-builds/11189/minikube-linux-amd64
# http://storage.googleapis.com/minikube-builds/11189/minikube-darwin-amd64
# http://storage.googleapis.com/minikube-builds/11189/minikube-windows-amd64.exe

# in case of macOS, use minikube-darwin-amd64
$ curl -LO http://storage.googleapis.com/minikube-builds/11189/minikube-darwin-amd64
$ chmod +x minikube-darwin-amd64
$ sudo mv minikube-darwin-amd64 /usr/local/bin/minikube

Just to condense everything down: this will be fixed in the next version of minikube.

Yes, the next version release will fix this issue.

Thank you.

[rromanchuk]

on a fresh t4g.medium with Ubuntu 20.04.2 LTS both minikube_1.18.1-0_arm64.deb and minikube_1.20.0.beta.0-0_arm64.deb i'm getting immediate crash standard_init_linux.go:219: exec user process caused: exec format error after minikube kubectl apply, i'm probably just trying to run the wrong/confused arch or something.

Very vanilla reproduction

$ sudo apt update
$ sudo apt upgrade -y
$ curl -fsSL get.docker.com -o get-docker.sh && sh get-docker.sh
$ sudo usermod -aG docker ubuntu ; newgrp docker
$ curl -LO https://github.com/kubernetes/minikube/releases/download/v1.20.0-beta.0/minikube_1.20.0.beta.0-0_arm64.deb
$ sudo dpkg -i minikube_1.20.0.beta.0-0_arm64.deb 
$ minikube start
😄  minikube v1.20.0-beta.0 on Ubuntu 20.04 (arm64)
✨  Automatically selected the docker driver. Other choices: none, ssh
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.20.2 preload ...
    > preloaded-images-k8s-v10-v1...: 514.95 MiB / 514.95 MiB  100.00% 126.56 M
    > gcr.io/k8s-minikube/kicbase...: 327.13 MiB / 327.13 MiB  100.00% 48.85 Mi
    > gcr.io/k8s-minikube/kicbase...: 327.13 MiB / 327.13 MiB  100.00% 16.23 Mi
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ minikube kubectl -- apply -f https://raw.githubusercontent.com/ansible/awx-operator/0.9.0/deploy/awx-operator.yaml
$ minikube kubectl get pods
awx-operator-5595d6fc57-tlw8p   0/1     CrashLoopBackOff   1          15s

[govargo]

The minikube v1.20.0 was released.
https://github.com/kubernetes/minikube/releases/tag/v1.20.0

This version includes the fix of this ingress(networking.k8s.io/v1) issue.
Thank you.

[bhundven]

I'm still trying to get #245 in which will allow for setting the apiVersion of the ingress, but for now it's still using extensions/v1beta1 and I run into other issues.