Unable to find an execution environment

[anetwalker]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Recently I deployed AWX 23.5.1 on a Kubernetes cluster. Initially, everything went well, and I can log in to the AWX management web interface. I created an Organization, an inventory, a host group, and some hosts. The issue arises when I try to run a command from the inventory using the ping module; I get an immediate error with the following message:

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 605, in run
res = receptor_job.run()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 318, in run
res = self._run_internal(receptor_ctl)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 335, in _run_internal
work_submit_kw = dict(worktype=self.work_type, params=self.receptor_params, signwork=self.sign_work)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 475, in receptor_params
spec_yaml = yaml.dump(self.pod_definition, explicit_start=True)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py", line 517, in pod_definition
default_pod_spec = get_default_pod_spec()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/execution_environments.py", line 35, in get_default_pod_spec
raise RuntimeError("Unable to find an execution environment.")
RuntimeError: Unable to find an execution environment.

AWX Operator version

23.5.1

AWX version

23.5.1

Kubernetes platform

other (please specify in additional information)

Kubernetes/Platform version

VMWare Tanzu

Modifications

yes

Steps to reproduce

I've changed the URLs of the images in the deployments to a local registry where I had previously uploaded them

Expected results

I expect the execution of this or another command works.

Actual results

Unable to find an execution environment

Additional information

No response

Operator Logs

No response

[jessicamack]

Hello @anetwalker, can you provide your AWX spec file (at least the part related to your custom EE urls) and the full operator logs? Did you notice any particular tasks failing in your operator logs related to EEs? In the UI do you see EEs listed?

[anetwalker]

Hi @jessicamack , thanks for your support.

At the moment, I have the operator stopped because when I start it, it overwrites the changes I've made in deployment.apps/ansible-awx-task and reverts back to the original URLs:

Before start the Operator:

[root@myhost ~]# kubectl get all -n awx
NAME READY STATUS RESTARTS AGE
pod/ansible-awx-postgres-13-0 1/1 Running 0 45h
pod/ansible-awx-task-5bd4fdb589-4nd8g 4/4 Running 0 39h
pod/ansible-awx-web-689788457b-8zcdq 3/3 Running 0 45h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ansible-awx-postgres-13 ClusterIP None 5432/TCP 42d
service/ansible-awx-service NodePort 192.96.112.114 80:31940/TCP 42d
service/awx-operator-controller-manager-metrics-service ClusterIP 192.68.46.4 8443/TCP 47d
service/external-awx-service LoadBalancer 192.68.144.237 192.168.158.81 8052:32528/TCP 33d

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ansible-awx-task 1/1 1 1 7d20h
deployment.apps/ansible-awx-web 1/1 1 1 42d
deployment.apps/awx-operator-controller-manager 0/0 0 0 47d

NAME DESIRED CURRENT READY AGE
replicaset.apps/ansible-awx-task-5bd4fdb589 1 1 1 46h
replicaset.apps/ansible-awx-task-7bf8589c7f 0 0 0 46h
replicaset.apps/ansible-awx-task-dc4767fdc 0 0 0 46h
replicaset.apps/ansible-awx-web-689788457b 1 1 1 46h
replicaset.apps/ansible-awx-web-69f7b67797 0 0 0 34d
replicaset.apps/awx-operator-controller-manager-79b5967d5b 0 0 0 43d

NAME READY AGE
statefulset.apps/ansible-awx-postgres-13 1/1 42d

After start the Operator

[root@myhost]# kubectl get all -n awx
NAME READY STATUS RESTARTS AGE
pod/ansible-awx-postgres-13-0 1/1 Running 0 46h
pod/ansible-awx-task-5bd4fdb589-4nd8g 4/4 Running 0 40h
pod/ansible-awx-task-dc4767fdc-jlq7s 0/4 Init:ErrImagePull 0 16m
pod/ansible-awx-web-689788457b-8zcdq 3/3 Running 0 45h
pod/awx-operator-controller-manager-79b5967d5b-cnv7h 2/2 Running 0 17m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ansible-awx-postgres-13 ClusterIP None 5432/TCP 42d
service/ansible-awx-service NodePort 192.168.112.114 80:31940/TCP 42d
service/awx-operator-controller-manager-metrics-service ClusterIP 192.168.46.4 8443/TCP 47d
service/external-awx-service LoadBalancer 192.168.144.237 192.168.158.81 8052:32528/TCP 33d

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ansible-awx-task 1/1 1 1 7d21h
deployment.apps/ansible-awx-web 1/1 1 1 42d
deployment.apps/awx-operator-controller-manager 1/1 1 1 47d

NAME DESIRED CURRENT READY AGE
replicaset.apps/ansible-awx-task-5bd4fdb589 1 1 1 47h
replicaset.apps/ansible-awx-task-7bf8589c7f 0 0 0 47h
replicaset.apps/ansible-awx-task-dc4767fdc 1 1 0 47h
replicaset.apps/ansible-awx-web-689788457b 1 1 1 47h
replicaset.apps/ansible-awx-web-69f7b67797 0 0 0 34d
replicaset.apps/awx-operator-controller-manager-79b5967d5b 1 1 1 43d

NAME READY AGE
statefulset.apps/ansible-awx-postgres-13 1/1 42d

I attach the description of ansible-awx-task.describe.after before and after starting the Operator. I also attach the log of the awx-manager container from the operator.
ansible-awx-task.describe.after.txt
ansible-awx-task.describe.before.txt
awx-manager.log