feat: implement extra_settings_files #1836

kurokobo · 2024-04-22T07:31:47Z

SUMMARY

This PR add a new paramerter extra_settings_files for AWX CR that allows users to add extra settings through python (*.py) files from ConfigMaps or Secrets.

This is helpful when users are in following situations that are difficult to solve by existing extra_settings:

The extra_settings is very long and hard to manage
Want to pass dictionaries or lists which written in readable multiline codes
Doesn't want to consider quoted-quotes in extra_settings
Want to manage extra_settings as sepalated multiple files
Want to pass values from Secrets

Design:

Users can add extra settings by mounting *.py files from ConfigMaps or Secrets
This feature mounts specified *.py files under /etc/tower/conf.d in following pods/jobs/cronjobs which already have settngs.py
- The task and web pods
- The migration job
- The metrics-utility-gather cronjobs and metrics-utility-report cronjobs
Users can specify multiple ConfigMaps and Secrets
Users can include multiple files in a single ConfigMap or Secret

Restrictions:

The keys in ConfigMaps or Secrets MUST be the name of python files and MUST end with ,py
The keys in ConfigMaps or Secrets MUST consists of alphanumeric characters, -, _ or .
- https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/config-map-v1/
- https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/secret-v1/
The keys in ConfigMaps or Secrets are converted to the following strings, which MUST not exceed 63 characters
- Keys in ConfigMaps: <instance name>-<KEY>-configmap
- Keys in Secrets: <instance name>-<KEY>-secret
- This is used as the name of the volume
  - The name of the volume can contain only lowercase alphanumeric characters or -, so any _ and . in keys are replacetd with - then the keys are converted to lowercase
  - https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/volume/
  - https://kubernetes.io/docs/concepts/overview/working-with-objects/names/

ISSUE TYPE

New or Enhanced Feature

ADDITIONAL INFORMATION

Tested by deploying custom Operator:

IMG=registry.example.com/ansible/awx:extra_settings_files \
DEFAULT_AWX_VERSION=24.2.0 \
NAMESPACE=awx \
make docker-build docker-push deploy

Example demo ConfigMaps and Secrets:

Since this is just for demonstration purposes, included settings and its value have no meaning.

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: awx
  name: demo-configmap-01
data:
  democonfigmap01_01.py: |
    UI_NEXT = False
    AUTOMATION_ANALYTICS_GATHER_INTERVAL = 1800
  democonfigmap01_02.py: |
    REMOTE_HOST_HEADERS = [
      "HTTP_X_FORWARDED_FOR",
      "REMOTE_ADDR",
      "REMOTE_HOST",
    ]
---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: awx
  name: demo-configmap-02
data:
  democonfigmap02_01.py: |
    AUTOMATION_ANALYTICS_URL = "https://awesome.demo.example.com"
    GALAXY_TASK_ENV = {
      "ANSIBLE_FORCE_COLOR": "false",
      "GIT_SSH_COMMAND": "ssh -o StrictHostKeyChecking=no",
      "EXTRA_SETTINGS_FROM_CONFIGMAP": "hello awx community",
    }
---
apiVersion: v1
kind: Secret
metadata:
  namespace: awx
  name: demo-secret-01
stringData:
  demosecret01_01.py: |
    TOWER_URL_BASE = "https://extra.settings.from.secret.example.com"
    ACTIVITY_STREAM_ENABLED = False
  demosecret01_02.py: |
    AWX_ISOLATION_SHOW_PATHS = [
      "/etc/pki/ca-trust:/etc/pki/ca-trust:O",
      "/usr/share/pki:/usr/share/pki:O",
      "/tmp/awx/demo:/tmp/awx/demo:0",
    ]
---
apiVersion: v1
kind: Secret
metadata:
  namespace: awx
  name: demo-secret-02
stringData:
  demosecret02_01.py: |
    SUBSCRIPTIONS_PASSWORD = "my-subscription-password123!"
    REDHAT_PASSWORD = "my-redhat-password123!"

Example CR:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  namespace: awx
  name: awx
spec:
  service_type: nodeport
  nodeport_port: 30080

  extra_settings_files:
    configmaps:
      - name: demo-configmap-01
        key: democonfigmap01_01.py
      - name: demo-configmap-01
        key: democonfigmap01_02.py
      - name: demo-configmap-02
        key: democonfigmap02_01.py
    secrets:
      - name: demo-secret-01
        key: demosecret01_01.py
      - name: demo-secret-01
        key: demosecret01_02.py
      - name: demo-secret-02
        key: demosecret02_01.py

Results

$ kubectl -n awx get deployment/awx-task -o json | jq -r  '.spec.template.spec.volumes[].name' | grep demo
awx-democonfigmap01-01-py-configmap
awx-democonfigmap01-02-py-configmap
awx-democonfigmap02-01-py-configmap
awx-demosecret01-01-py-secret
awx-demosecret01-02-py-secret
awx-demosecret02-01-py-secret

$ kubectl -n awx exec deployment/awx-task -- ls -l /etc/tower/conf.d
total 36
-rw-r--r--. 1 root root 613 Apr 22 06:20 credentials.py
-rw-r--r--. 1 root root  60 Apr 22 06:20 democonfigmap01_01.py
-rw-r--r--. 1 root root  86 Apr 22 06:20 democonfigmap01_02.py
-rw-r--r--. 1 root root 232 Apr 22 06:20 democonfigmap02_01.py
-rw-r--r--. 1 root root  98 Apr 22 06:20 demosecret01_01.py
-rw-r--r--. 1 root root 146 Apr 22 06:20 demosecret01_02.py
-rw-r--r--. 1 root root  99 Apr 22 06:20 demosecret02_01.py
-rw-r--r--. 1 root root 262 Apr 22 06:20 execution_environments.py
-rw-r--r--. 1 root root  91 Apr 22 06:20 ldap.py

$ kubectl -n awx exec deployment/awx-task -- bash -c 'find /etc/tower/conf.d -name 'demo*.py' | xargs grep -H .'
/etc/tower/conf.d/democonfigmap01_02.py:REMOTE_HOST_HEADERS = [
/etc/tower/conf.d/democonfigmap01_02.py:  "HTTP_X_FORWARDED_FOR",
/etc/tower/conf.d/democonfigmap01_02.py:  "REMOTE_ADDR",
/etc/tower/conf.d/democonfigmap01_02.py:  "REMOTE_HOST",
/etc/tower/conf.d/democonfigmap01_02.py:]
/etc/tower/conf.d/democonfigmap02_01.py:AUTOMATION_ANALYTICS_URL = "https://awesome.demo.example.com"
/etc/tower/conf.d/democonfigmap02_01.py:GALAXY_TASK_ENV = {
/etc/tower/conf.d/democonfigmap02_01.py:  "ANSIBLE_FORCE_COLOR": "false",
/etc/tower/conf.d/democonfigmap02_01.py:  "GIT_SSH_COMMAND": "ssh -o StrictHostKeyChecking=no",
/etc/tower/conf.d/democonfigmap02_01.py:  "EXTRA_SETTINGS_FROM_CONFIGMAP": "hello awx community",
/etc/tower/conf.d/democonfigmap02_01.py:}
/etc/tower/conf.d/democonfigmap01_01.py:UI_NEXT = False
/etc/tower/conf.d/democonfigmap01_01.py:AUTOMATION_ANALYTICS_GATHER_INTERVAL = 1800
/etc/tower/conf.d/demosecret02_01.py:SUBSCRIPTIONS_PASSWORD = "my-subscription-password123!"
/etc/tower/conf.d/demosecret02_01.py:REDHAT_PASSWORD = "my-redhat-password123!"
/etc/tower/conf.d/demosecret01_02.py:AWX_ISOLATION_SHOW_PATHS = [
/etc/tower/conf.d/demosecret01_02.py:  "/etc/pki/ca-trust:/etc/pki/ca-trust:O",
/etc/tower/conf.d/demosecret01_02.py:  "/usr/share/pki:/usr/share/pki:O",
/etc/tower/conf.d/demosecret01_02.py:  "/tmp/awx/demo:/tmp/awx/demo:0",
/etc/tower/conf.d/demosecret01_02.py:]
/etc/tower/conf.d/demosecret01_01.py:TOWER_URL_BASE = "https://extra.settings.from.secret.example.com"
/etc/tower/conf.d/demosecret01_01.py:ACTIVITY_STREAM_ENABLED = False

We can see from the UI that the settings have been changed. This is an excerpt:

To ensure there is no side-effect, I have also confirmed the minimal AWX CR without extra_settings_files can be deployed:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  namespace: awx
  name: awx-demo
spec:
  service_type: nodeport
  nodeport_port: 30081

TODO

Documentation

kurokobo · 2024-04-22T07:36:42Z

@rooftopcellist @TheRealHaoLiu
I have implemented the idea for now. I would like to have your opinion 😃
Sorry for this but I can't test changes for metrics-utility things on my side since I don't have any environment that metrics-utility connects to.

kurokobo · 2024-04-24T13:36:42Z

@rooftopcellist @TheRealHaoLiu
Updated docs:

Add documentation about extra_settings_files and modify exsiting extra_settings documentation
Remove examples that mounts *.py files by extra_volumes and *__volume_mounts

Now I've marked this PR as ready for review. I am not sure if the parameter names and dictionary structure are ideally designed, but I think it allows for flexibility of use.

Any feedbacks are welcome! Of course feel free to reject this if this is not required for you. Thanks😃

kurokobo · 2024-04-24T13:59:16Z

Documentation previews

fosterseth · 2024-05-15T19:01:39Z

if the same setting is set in both extra_settings an in one of the .py files, how does that interaction work?

Also, the API blocks changing any setting if that setting is in extra_settings in the AWX spec, and I'm wondering if that same logic should apply to settings in the .py files

kurokobo · 2024-05-16T12:19:24Z

@fosterseth
Hi, thanks for the comment.

if the same setting is set in both extra_settings an in one of the .py files, how does that interaction work?

Including any *.py files are by django-split-settings in production.py in AWX, so,

Include settings.py which contains all settings from extra_settings first,
Then include any /etc/tower/conf.d/*.py from extra_settings_files which overrides any settings insettings.py

Note that if two or more /etc/tower/conf.d/*.py files contain the same key, it would be difficult to predict which would be adopted due to the following behavior:

Note that files are included in the order that glob returns them, probably in the same order as what ls -U would list them. The files are NOT in alphabetical order.
https://github.com/wemake-services/django-split-settings

Also, the API blocks changing any setting if that setting is in extra_settings in the AWX spec, and I'm wondering if that same logic should apply to settings in the .py files

Yes, AFAIK the same logic will be applied for any /etc/tower/conf.d/*.py files, since both settings.py and *.py are included by above single include() in production.py, they should all be read-only as well.

kurokobo · 2024-05-16T12:40:08Z

Conflict resolved and some notes are added that @fosterseth pointed out 😃

rooftopcellist · 2024-05-22T18:36:12Z

@kurokobo we have reviewed this PR and are ready to merge it, but it looks like there are some conflicts. Could you please rebase? I think that will resolve the CI failure too.

kurokobo · 2024-05-23T12:29:38Z

@rooftopcellist
Thanks, resolved conflicts and rebased!

kurokobo · 2024-05-23T13:21:27Z

Hmm 🤔
Something was happened in init-receptor?

2024-05-23T12:58:33.5143827Z     "msg": "Unable to retrieve log from Pod due to: container \"example-awx-ee\" in pod \"example-awx-task-65cfbfbd68-t5c5t\" is waiting to start: PodInitializing"
2024-05-23T12:58:33.5146601Z }
2024-05-23T12:58:33.5150069Z 
2024-05-23T12:58:33.5156040Z PLAY RECAP *********************************************************************
2024-05-23T12:58:33.5159721Z localhost                  : ok=42   changed=9    unreachable=0    failed=1    skipped=2    rescued=2    ignored=0kuro@kuro-dev01:~/work/kurokobo/private-odc-moderator/dify/docker$

2024-05-23T12:58:10.0399069Z                 "initContainerStatuses": [
2024-05-23T12:58:10.0401984Z                     {
2024-05-23T12:58:10.0405443Z                         "containerID": "containerd://757fa0dd390cc8246372831c691312be17ca18a33b4ff183433488d73ef1897f",
2024-05-23T12:58:10.0407640Z                         "image": "quay.io/ansible/awx:24.4.0",
2024-05-23T12:58:10.0411161Z                         "imageID": "quay.io/ansible/awx@sha256:62df9e77fef859f172c51c3e564d8a3ac03a5b32d4f4c398501eee65b9967633",
2024-05-23T12:58:10.0413551Z                         "lastState": {},
2024-05-23T12:58:10.0416360Z                         "name": "init-database",
2024-05-23T12:58:10.0419126Z                         "ready": true,
2024-05-23T12:58:10.0421940Z                         "restartCount": 0,
2024-05-23T12:58:10.0424703Z                         "started": false,
2024-05-23T12:58:10.0427582Z                         "state": {
2024-05-23T12:58:10.0430498Z                             "terminated": {
2024-05-23T12:58:10.0434083Z                                 "containerID": "containerd://757fa0dd390cc8246372831c691312be17ca18a33b4ff183433488d73ef1897f",
2024-05-23T12:58:10.0435949Z                                 "exitCode": 0,
2024-05-23T12:58:10.0439132Z                                 "finishedAt": "2024-05-23T12:57:12Z",
2024-05-23T12:58:10.0441800Z                                 "reason": "Completed",
2024-05-23T12:58:10.0444292Z                                 "startedAt": "2024-05-23T12:54:20Z"
2024-05-23T12:58:10.0446722Z                             }
2024-05-23T12:58:10.0449269Z                         }
2024-05-23T12:58:10.0452118Z                     },
2024-05-23T12:58:10.0454833Z                     {
2024-05-23T12:58:10.0458148Z                         "image": "quay.io/ansible/awx-ee:24.4.0",
2024-05-23T12:58:10.0460556Z                         "imageID": "",
2024-05-23T12:58:10.0463573Z                         "lastState": {},
2024-05-23T12:58:10.0466700Z                         "name": "init-receptor",
2024-05-23T12:58:10.0469385Z                         "ready": false,
2024-05-23T12:58:10.0472413Z                         "restartCount": 0,
2024-05-23T12:58:10.0475477Z                         "started": false,
2024-05-23T12:58:10.0478455Z                         "state": {
2024-05-23T12:58:10.0481321Z                             "waiting": {
2024-05-23T12:58:10.0484336Z                                 "reason": "PodInitializing"
2024-05-23T12:58:10.0487156Z                             }
2024-05-23T12:58:10.0489835Z                         }
2024-05-23T12:58:10.0492564Z                     }
2024-05-23T12:58:10.0495279Z                 ],

TheRealHaoLiu · 2024-05-23T13:43:45Z

darn my latest debug artifact gathering for the molecule test does not gather log from init container... >.<
https://github.com/ansible/awx-operator/blob/devel/molecule/default/verify.yml/#L44-L49

kurokobo · 2024-05-23T14:03:27Z

Since it succeeded in #1867, I can't imagine it well, but this PR may be the cause of the CI failure.

Before digging into this deeper, could you please try to re-run CI by closing / reopening this PR?

kurokobo · 2024-05-23T14:32:13Z

Failed in very early stage 😞

Run ansible-galaxy collection install -r molecule/requirements.yml
Starting galaxy collection install process
Process install dependency map
ERROR! Error when getting collection version metadata for community.general:9.0.0 from default (https://galaxy.ansible.com/api/) (HTTP Code: 530, Message:  Code: Unknown)
Error: Process completed with exit code 1.

TheRealHaoLiu · 2024-05-23T15:46:50Z

i rerun it again and it's successful... we been observing flakiness in our CI lately and i been trying to track it down (so far no success...)

kurokobo · 2024-05-24T01:18:43Z

@TheRealHaoLiu @rooftopcellist
Thanks for working on this, I don't think there is a developer out there who has not been plagued by flakiness😞
Resources for GHA are not very plentiful, so it would be difficult to reduce the occurrence to zero. It might be better to automate retries.

github-actions bot added the community label Apr 22, 2024

kurokobo marked this pull request as ready for review April 24, 2024 13:30

kurokobo force-pushed the extra_settings_files branch from 12a25b9 to cb75ae5 Compare April 24, 2024 13:37

kurokobo force-pushed the extra_settings_files branch from 084ea66 to fa072c4 Compare May 16, 2024 12:39

kurokobo added 6 commits May 23, 2024 21:27

feat: implement extra_settings_files

31794d9

fix: reduce duplicated code blocks by templates

50bb071

docs: update docs for extra settings

439a67a

docs: simplify the commands

5400382

docs: add notes for duplicated keys in setting files

e3a37a2

docs: fix headings

89013e4

kurokobo force-pushed the extra_settings_files branch from 759c408 to 89013e4 Compare May 23, 2024 12:27

rooftopcellist enabled auto-merge (squash) May 23, 2024 17:40

rooftopcellist approved these changes May 23, 2024

View reviewed changes

rooftopcellist merged commit 56df327 into ansible:devel May 23, 2024
7 checks passed

kurokobo deleted the extra_settings_files branch May 24, 2024 01:07

kurokobo mentioned this pull request May 24, 2024

feat: implement extra_settings_files ansible/galaxy-operator#125

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement extra_settings_files #1836

feat: implement extra_settings_files #1836

kurokobo commented Apr 22, 2024 •

edited

kurokobo commented Apr 22, 2024

kurokobo commented Apr 24, 2024

kurokobo commented Apr 24, 2024 •

edited

fosterseth commented May 15, 2024 •

edited

kurokobo commented May 16, 2024

kurokobo commented May 16, 2024

rooftopcellist commented May 22, 2024

kurokobo commented May 23, 2024

kurokobo commented May 23, 2024

TheRealHaoLiu commented May 23, 2024

kurokobo commented May 23, 2024

kurokobo commented May 23, 2024

TheRealHaoLiu commented May 23, 2024

kurokobo commented May 24, 2024

feat: implement extra_settings_files #1836

feat: implement extra_settings_files #1836

Conversation

kurokobo commented Apr 22, 2024 • edited

SUMMARY

ISSUE TYPE

ADDITIONAL INFORMATION

TODO

kurokobo commented Apr 22, 2024

kurokobo commented Apr 24, 2024

kurokobo commented Apr 24, 2024 • edited

fosterseth commented May 15, 2024 • edited

kurokobo commented May 16, 2024

kurokobo commented May 16, 2024

rooftopcellist commented May 22, 2024

kurokobo commented May 23, 2024

kurokobo commented May 23, 2024

TheRealHaoLiu commented May 23, 2024

kurokobo commented May 23, 2024

kurokobo commented May 23, 2024

TheRealHaoLiu commented May 23, 2024

kurokobo commented May 24, 2024

kurokobo commented Apr 22, 2024 •

edited

kurokobo commented Apr 24, 2024 •

edited

fosterseth commented May 15, 2024 •

edited