Managed postgres - /var/lib/postgresql/data: permission denied #483
Comments
|
Have a look at this. |
|
I'm am following the exact instructions @Marwel linked above but I'm getting the exact same error (mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied). I've been banging my head for 4 days w/ no success on installing AWX. |
|
I'll give a try this afternoon using |
|
Hello guys, I've deployed a $ kubectl get nodes
NAME STATUS ROLES AGE VERSION
storm.tatu.home Ready control-plane,master 3m39s v1.21.3+k3s1
$ kubectl get pods -A 23:01:09
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-5ff76fc89d-4d7bn 1/1 Running 0 9m51s
kube-system metrics-server-86cbb8457f-9fkt2 1/1 Running 0 9m51s
kube-system coredns-7448499f4d-9t87w 1/1 Running 0 9m51s
kube-system helm-install-traefik-crd-mlrtg 0/1 Completed 0 9m51s
kube-system helm-install-traefik-v5n5s 0/1 Completed 1 9m51s
kube-system svclb-traefik-c9cgh 2/2 Running 0 9m28s
kube-system traefik-97b44b794-6dz4g 1/1 Running 0 9m28s
Then I generated the latest kubectl apply -f deploy/awx-operator.yaml 23:08:40
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created
clusterrole.rbac.authorization.k8s.io/awx-operator created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator created
serviceaccount/awx-operator created
deployment.apps/awx-operator createdThe operator started as expected: kubectl get pods -A -w 23:07:32
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-5ff76fc89d-4d7bn 1/1 Running 0 11m
kube-system metrics-server-86cbb8457f-9fkt2 1/1 Running 0 11m
kube-system coredns-7448499f4d-9t87w 1/1 Running 0 11m
kube-system helm-install-traefik-crd-mlrtg 0/1 Completed 0 11m
kube-system helm-install-traefik-v5n5s 0/1 Completed 1 11m
kube-system svclb-traefik-c9cgh 2/2 Running 0 10m
kube-system traefik-97b44b794-6dz4g 1/1 Running 0 10m
default awx-operator-88b886454-9pq7w 0/1 ContainerCreating 0 15s
default awx-operator-88b886454-9pq7w 1/1 Running 0 16sSo now to troubleshooting, I'm using a similar AWX spec provided earlier on as follows below. As you can see, I have to extend it so I could create the PVC ---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: awx-projects-claim
namespace: default
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-path
resources:
requests:
storage: 2Gi
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
spec:
ingress_type: Ingress
route_tls_termination_mechanism: edge
hostname: localhost
postgres_storage_requirements:
requests:
storage: 3Gi
projects_persistence: true
projects_existing_claim: awx-projects-claim
web_resource_requirements:
requests:
cpu: 250m
memory: 2Gi
limits:
cpu: 750m
memory: 4Gi
task_resource_requirements:
requests:
cpu: 250m
memory: 1Gi
limits:
cpu: 500m
memory: 2Gi
ee_resource_requirements:
requests:
cpu: 250m
memory: 1Gi
limits:
cpu: 500m
memory: 2Gi$ kubectl apply -f pg-k3s.yml 23:13:08
persistentvolumeclaim/awx-projects-claim created
awx.awx.ansible.com/awx created
# still pending because POD has not started yet
$ kubectl get pvc 23:14:05
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
awx-projects-claim Pending local-path 23s
postgres-awx-postgres-0 Bound pvc-3b9e6563-9085-4d79-90ba-fa6c88431c6c 3Gi RWO local-path 4sThen looking at the pod, I got it to crash So basically the $ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
postgres-awx-postgres-0 Bound pvc-3b9e6563-9085-4d79-90ba-fa6c88431c6c 3Gi RWO local-path 2m37s
awx-projects-claim Bound pvc-85b1b705-43b3-42a6-a96b-1e79943e99d5 2Gi RWO local-path 2m56s
$ kubectl get pods 23:17:32
NAME READY STATUS RESTARTS AGE
awx-operator-88b886454-9pq7w 1/1 Running 0 8m47s
awx-76bdfc954c-jxvll 4/4 Running 0 3m21s
awx-postgres-0 0/1 Error 5 3m30sThen looking at the container, yes, I got the same error on $ kubectl logs awx-postgres-0 23:26:52
mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission deniedI'm working on it. |
|
So basically we will need to leverage on a `initContainer approach to fix the permission so the database can be created. This snippet will do the job: diff --git a/roles/installer/tasks/database_configuration.yml b/roles/installer/tasks/database_configuration.yml
index 2e99be5..470530a 100644
--- a/roles/installer/tasks/database_configuration.yml
+++ b/roles/installer/tasks/database_configuration.yml
@@ -80,8 +80,9 @@
- block:
- name: Create Database if no database is specified
k8s:
- apply: true
+ apply: yes
definition: "{{ lookup('template', 'postgres.yaml.j2') }}"
+ wait: yes
register: create_statefulset_result
rescue:
diff --git a/roles/installer/templates/postgres.yaml.j2 b/roles/installer/templates/postgres.yaml.j2
index d17ee12..f87c842 100644
--- a/roles/installer/templates/postgres.yaml.j2
+++ b/roles/installer/templates/postgres.yaml.j2
@@ -37,10 +37,27 @@ spec:
imagePullSecrets:
- name: {{ image_pull_secret }}
{% endif %}
+ initContainers:
+ - name: init-chmod-data
+ image: '{{ postgres_image }}:{{ postgres_image_version }}'
+ imagePullPolicy: '{{ image_pull_policy }}'
+ command:
+ - /bin/sh
+ - -c
+ - |
+ if [ ! -f {{ postgres_data_path }}/PG_VERSION ]; then
+ chown postgres:root {{ postgres_data_path | dirname }}
+ fi
+ volumeMounts:
+ - name: postgres
+ mountPath: '{{ postgres_data_path | dirname }}'
+ subPath: '{{ postgres_data_path | dirname | basename }}'
containers:
- image: '{{ postgres_image }}:{{ postgres_image_version }}'
imagePullPolicy: '{{ image_pull_policy }}'
name: postgres
+ securityContext:
+ fsGroup: 999
env:
# For postgres_image based on rhel8/postgresql-12
- name: POSTGRESQL_DATABASE
It does result in a working state once the patch is applied: $ ubectl get pods -w 00:38:58
NAME READY STATUS RESTARTS AGE
awx-operator-5bc776b4d4-d9ww2 1/1 Running 0 4m41s
awx-postgres-0 1/1 Running 0 4m3s
awx-d67898cd9-k6jrc 4/4 Running 0 3m48s
$ kubectl iexec awx-postgres-0 /bin/bash 00:57:00
root@awx-postgres-0:/# namei -xmolv /var/lib/postgresql/data/pgdata/
f: /var/lib/postgresql/data/pgdata/
Drwxr-xr-x root root /
drwxr-xr-x root root var
drwxr-xr-x root root lib
drwxr-xr-x postgres postgres postgresql
Drwx------ postgres root data
drwx------ postgres root pgdata
I'll create a PR for it. Thanks for reporting the issue @flisak-robert and @scott-vick |
|
Is there any temporary solution before the update? |
Don't know if it suits your needs, but I just ran postgres in a docker container and pointed awx to use that postgres instance instead. Don't forget to include |
|
A workaround that I have found is to create a PV: The path will be given 777 permission through chmod. |
let's say one would declare |
|
Created a PV with storageClassName and used that SC name as above, but getting following error in Postgres pod log. |
|
Funny, I see deltas in behaviour on the 0.13 vs the 0.12; DB working, but other pods not starting.
|
|
did someone manage to solve this problem? |
|
I eventually deployed with the dev branch of awx operator. |
|
I solved this with First, I tried only with But I succeeded by adding |
I am trying to install AWX using the awx-operator running on k3s and awx-postgres pod fails with the message:
mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission deniedHere is my
awx.yml:What am I doing wrong here?
The text was updated successfully, but these errors were encountered: