Internal Server Error after deployment with new awx-operator version 0.7.0

[lingenavd]

Yesterday (22nd of march 2021) I deployed AWX fine with awx operator version 0.6.0.
Today I got the new awx-operator version 0.7.0 and all pods start, but the awx-web is full of errors.
the ingress url to AWX now gives me the only line: Internal Server Error

what I also noticed that yesterday I only had 3 containers in the awx deployment, now I have 4. awx-ee is now also the 4th container in the POD.

logging of awx-web:

Traceback (most recent call last):
File "/usr/bin/awx-manage", line 8, in
sys.exit(manage())
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/init.py", line 154, in manage
execute_from_command_line(sys.argv)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py", line 381, in execute_from_command_line
utility.execute()
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py", line 357, in execute
django.setup()
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/init.py", line 19, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/utils/log.py", line 76, in configure_logging
logging_config_func(logging_settings)
File "/usr/lib64/python3.6/logging/config.py", line 802, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib64/python3.6/logging/config.py", line 573, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'console': Unable to add filter 'guid': 'guid'
2021-03-23 22:05:53,677 INFO exited: wsbroadcast (exit status 1; not expected)
2021-03-23 22:05:53,677 INFO exited: wsbroadcast (exit status 1; not expected)
2021-03-23 22:05:54,753 INFO spawned: 'wsbroadcast' with pid 4180
2021-03-23 22:05:54,753 INFO spawned: 'wsbroadcast' with pid 4180
Traceback (most recent call last):
File "/usr/lib64/python3.6/logging/config.py", line 692, in add_filters
filterer.addFilter(self.config['filters'][f])
File "/usr/lib64/python3.6/logging/config.py", line 326, in getitem
value = dict.getitem(self, key)
KeyError: 'guid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.6/logging/config.py", line 565, in configure
handler = self.configure_handler(handlers[name])
File "/usr/lib64/python3.6/logging/config.py", line 753, in configure_handler
self.add_filters(result, filters)
File "/usr/lib64/python3.6/logging/config.py", line 694, in add_filters
raise ValueError('Unable to add filter %r: %s' % (f, e))
ValueError: Unable to add filter 'guid': 'guid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/bin/daphne", line 8, in
sys.exit(CommandLineInterface.entrypoint())
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/daphne/cli.py", line 191, in entrypoint
cls().run(sys.argv[1:])
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/daphne/cli.py", line 252, in run
application = import_by_path(args.application)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/daphne/utils.py", line 12, in import_by_path
target = importlib.import_module(module_path)
File "/var/lib/awx/venv/awx/lib64/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/asgi.py", line 35, in
django.setup()
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/init.py", line 19, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/utils/log.py", line 76, in configure_logging
logging_config_func(logging_settings)
File "/usr/lib64/python3.6/logging/config.py", line 802, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib64/python3.6/logging/config.py", line 573, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'console': Unable to add filter 'guid': 'guid'
2021-03-23 22:05:55,974 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-03-23 22:05:55,974 INFO success: wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-03-23 22:05:55,974 INFO exited: daphne (exit status 1; not expected)
2021-03-23 22:05:55,974 INFO exited: daphne (exit status 1; not expected)

[shanemcd]

Did you supply your own tower_image? You'll either need to use the latest, or use an older version of this operator.

[lingenavd]

Tower image was the vanilla version 15.0. (I did not confige it in my yaml)
I will give it a go with the latest tower image today.
Thanx for the feedback.
Grtz,
Andre

[lingenavd]

Sorry to say it does not make a difference :-(
Clean K8s cluster.
Only line I get on the URL to AWX is: Internal Server Error

kubectl -n awx get pods
NAME READY STATUS RESTARTS AGE
awx-968dd9cc8-85l7n 4/4 Running 0 35m
awx-postgres-0 1/1 Running 0 35m

my-awx.yml:
#######################################################

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
namespace: awx
name: awx
spec:
tower_admin_user: admin
#tower_admin_password_secret:

tower_image: ansible/awx:latest
tower_image_pull_policy: IfNotPresent

tower_web_resource_requirements:
requests:
cpu: 1000m
memory: 2Gi
limits:
cpu: 2000m
memory: 3Gi

tower_task_resource_requirements:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 1000m
memory: 2Gi

tower_postgres_resource_requirements:
requests:
memory: 1Gi
storage: 8Gi
limits:
memory: 2Gi
storage: 16Gi
tower_postgres_storage_class: manual
#######################################################

All containers except awx-web and awx_task are fine.
see attached zip file for all container logging.
awx-postgres.zip

[lingenavd]

I tested it again with new clean k8s cluster.
I changed the following line in the awx-operator.yaml after download:
cat awx-operator.yaml | grep 0.
image: "quay.io/ansible/awx-operator:0.6.0"
I used the same: my-awx.yml

kubectl -n awx get pods
NAME READY STATUS RESTARTS AGE
awx-8b89d69f-xd259 3/3 Running 0 19m
awx-postgres-0 1/1 Running 0 19m

Now, I don't have the awx-ee container in the POD for AWX.

BUT the MOST important thing is that AWX was completely deployed without issues. :-)
I am now logged into AWX with the new AWX images, looks much better, compliments for that :-)

There is definitely something wrong with the awx-operator 0.7.0 and prob. the combination with awx-ee?

Only issue I have with this awx-operator that it does not create the secret "awx-admin-password".
So you cannot login when AWX is ready up and running.
But that is easly fixed doing the following:

kubectl -n awx exec -it <AWX POD name> --container awx-web -- bash
In the container just create you own superuser account:
awx-manage createsuperuser --username <username>

grtz,
Andre

[Zokormazo]

Wich registry is resolving that tower_image: ansible/awx:latest ? I would say that your k8s is pulling that from dockerhub, where latest is 17.1.0 and not 18.0.0

Try using fqdn: tower_image: quay.io/ansible/awx:latest

[lingenavd]

I just checked, the installed version is indeed 17.1.0

Grtz,
Andre

[lingenavd]

Final test from me, because it works now :-)

again with new clean k8s cluster.
I used the 0.7.0 for the awx-operator
I changed the line in my-awx.yml to get the image from quay.io/ansible/awx:latest
This resulted in version 18.0, so @Zokormazo was right that my k8s cluster got the latest version from dockerhub.

kubectl -n awx get pods
NAME READY STATUS RESTARTS AGE
awx-889f8c78-4bj5l 4/4 Running 0 26m
awx-postgres-0 1/1 Running 0 27m

the awx-ee container is now also deployed in the AWX deployment pod.

AWX was completely deployed without any issues. :-)
Also the secret "awx-admin-password" is created with the awx-operator version 0.7.0 and this works like a charm.

Conclusion:
Everything works as designed if you have the correct combination for the awx-operator and the AWX image version.
So:
- awx-operator version 0.7.0 (lastest) ONLY works successfully in combination with AWX 18.0.0 (latest from quay.io/ansible/awx:latest)
- awx-operator version 0.6.0 in combination with the AWX versions I tested (AWX 15.0.0 and 17.1.0) works fine with the workaround for the AWX admin password.

Maybe this could be mentioned in the install docs more clearly.

Issue can be closed :-)

grtz,
Andre

[balonik]

I agree with @lingenavd here. If version of awx-operator requires specific version of AWX it should be written somewhere.