Crash without logs

[tamis-laan]

I'm using the following kubernetes deployment for nominatim:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nominatim
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nominatim
  template:
    metadata:
      labels:
        app: nominatim
    spec:
      containers:
        - name: nominatim
          image: mediagis/nominatim:4.2
          ports:
            - containerPort: 8080
          env:
            - name: PBF_URL
              value: https://download.geofabrik.de/europe/monaco-latest.osm.pbf
            - name: REPLICATION_URL
              value: https://download.geofabrik.de/europe/monaco-updates/
          volumeMounts:
            - name: nominatim-db
              mountPath: /var/lib/postgresql/14/main
            - name: nominatim-other
              mountPath: /nominatim
          readinessProbe:
            httpGet:
              path: /search
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /search
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 15
      volumes:
        - name: nominatim-db
          persistentVolumeClaim:
            claimName: nominatim-db
        - name: nominatim-other
          persistentVolumeClaim:
            claimName: nominatim-other

---
apiVersion: v1
kind: Service
metadata:
  name: nominatim
spec:
  type: ClusterIP
  selector:
    app: nominatim
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nominatim-db
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 40Gi

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nominatim-other
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 40Gi

Everything work up to the point where it says warming database caches, at that point the pod goes dead.

2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 9

2023-07-19 14:35:32: Starting rank 10 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 10

2023-07-19 14:35:32: Starting rank 11 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 11

2023-07-19 14:35:32: Starting rank 12 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 12

2023-07-19 14:35:32: Starting rank 13 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 13

2023-07-19 14:35:32: Starting rank 14 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 14

2023-07-19 14:35:32: Starting rank 15 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 15

2023-07-19 14:35:32: Starting rank 16 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 16

2023-07-19 14:35:32: Starting rank 17 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 17

2023-07-19 14:35:32: Starting rank 18 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 18

2023-07-19 14:35:32: Starting rank 19 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 19

2023-07-19 14:35:32: Starting rank 20 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 20

2023-07-19 14:35:32: Starting rank 21 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 21

2023-07-19 14:35:32: Starting rank 22 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 22

2023-07-19 14:35:32: Starting rank 23 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 23

2023-07-19 14:35:32: Starting rank 24 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 24

2023-07-19 14:35:32: Starting rank 25 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 25

2023-07-19 14:35:32: Starting rank 26 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 26

2023-07-19 14:35:32: Starting rank 27 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 27

2023-07-19 14:35:32: Starting rank 28 (using batch size 1)
2023-07-19 14:35:32: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 28

2023-07-19 14:35:32: Starting rank 29 (using batch size 1)
2023-07-19 14:35:33: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 29

2023-07-19 14:35:33: Starting rank 30 (using batch size 20)
2023-07-19 14:35:33: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 30

2023-07-19 14:35:33: Starting rank 0 (using batch size 1)
2023-07-19 14:35:33: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 0

2023-07-19 14:35:33: Starting interpolation lines (location_property_osmline) (using batch size 20)
2023-07-19 14:35:33: Done 0/0 in 0 @ 0.000 per second - FINISHED interpolation lines (location_property_osmline)

+ sudo -E -u nominatim nominatim admin --check-database
2023-07-19 14:35:33: Using project directory: /nominatim
2023-07-19 14:35:33: Checking database
Checking database connection ... OK
Checking for placex table ... OK
Checking for placex content ... OK
Checking that tokenizer works ... OK
Checking for wikipedia/wikidata data ... WARNING

Wikipedia/Wikidata importance tables missing.
Quality of search results may be degraded. Reverse geocoding is unaffected.
See https://nominatim.org/release-docs/latest/admin/Import/#wikipediawikidata-rankings

Checking indexing status ... OK
Checking that database indexes are complete ... OK
Checking that all database indexes are valid ... OK
Checking TIGER external data table. ... not applicable
+ '[' https://download.geofabrik.de/europe/monaco-updates/ '!=' '' ']'
+ sudo -E -u nominatim nominatim replication --init
2023-07-19 14:35:39: Using project directory: /nominatim
2023-07-19 14:35:40: Initialising replication updates
2023-07-19 14:35:40: Updates initialised at sequence 5599936 (2023-06-01 09:57:41+00:00)
2023-07-19 14:35:40: Create functions
+ '[' '' = true ']'
+ '[' '' = true ']'
+ sudo -E -u nominatim nominatim admin --warm
2023-07-19 14:35:41: Using project directory: /nominatim
2023-07-19 14:35:41: Warming database caches
Warm reverse: ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Stream closed EOF for default/nominatim-85465f6d7d-jbh9j (nominatim)

It looks like removing the readiness probe and liveness probe allows for the container to run properly. This is confusing because kubernetes does not kill or restart pods based on these probes. So I'm thinking hitting the /search route before nominatim finished initializing somehow brakes things.

BTW it would be nice if the initialization process could be split from serving the API so we can put the initialization process into a seperate init container.