Clarification on Update Process

[SenorKarlos]

I've ran various versions of docker nominatim over the years and am never quite sure if I've gotten this right. I prefer compose but that shouldn't make a difference I think.

on my last install (4.2) I put UPDATE_MODE: continuous in my environment, and I think I saw some successful looking log readouts on startup, but don't recall seeing anything about updates afterwards, and am in process of formatting that server after far too long and going up to 4.3.

My points of confusion come from these:

• in the General Config part of the readme, you show the replication options with a link to the official site. The docker options seem like it would be fully automated, but the only 'automatic' mode seems to be continuous, which the official site has a big red marker on and mentions dropping support/setting schedules.
• then in the Updating Database section, it mentions no automation at all and talks about running docker exec commands on the container.

Basically, which is it, and what is the best recommended way per y'all and nominatim itself? :) Thanks hopefully in advance!

[mtmail]

in process of formatting that server after far too long and going up to 4.3.

Nominatim 4.4 was released last week, you might want to wait

[SenorKarlos]

Hrm eh, good to know. How long after a release is the docker update here usually? I need to get on task basically right away, I have the instance to not abuse the official site with my service lol

[SenorKarlos]

Also my 4.2 decided to randomly quit at some point, db won't start. I've been down for other reasons so didn't notice until last night LOL

[mtmail]

How long after a release is the docker update here usually?

Some weeks I think. Best if you open an github issue asking for a new new release. (I'm working on Nominatim only, not docker, but often enough I can help answer Nominatim questions)

the only 'automatic' mode seems to be continuous

Yes. Continuous can crash though so check the /var/log/replication.log In a production system you could regularly check the /status?format=json for the data date. It's usually enough to have a cronjob starting (docker exec) the 'catch up' mode once per day (or once per hour as long as you make sure only one process runs). Continous mode was created for minutely updates which is rarely used with docker containers.

[SenorKarlos]

That makes sense and also explains it being out of date. I'll go this route :) I've set my user to run docker without sudo so takes root cron out as well :) (Ubuntu user lol)

Thanks much!! And I guess I'll go with 4.3 for now :) gotta get a move on :D

[markdavieswlt]

I would also like to know
when I navigate to http://myipaddress:8080/status?format=json
I see data_updated date as no where near current

[mtmail]

Then run the update https://github.com/mediagis/nominatim-docker/blob/master/4.3/README.md#updating-the-database I'd use and check the time it takes to run --once to get an estimate how long the full update will take. If for example updating one day takes 4 hours and you need to update 3 months the it can be easier to reinstall Nominatim instead of waiting 15 days. Or of course just let it run in the background.

[SenorKarlos]

wound up waiting for 4.4 after all... and of course, it can't just work for me right LMAO

/nominatim$ docker exec -it nominatim sudo -u nominatim NOMINATIM_REPLICATION_MAX_DIFF=5000 nominatim replication --catch-up --threads 16
2024-03-31 15:26:14: Using project directory: /app
Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 12, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 228, in nominatim
    return get_set_parser().run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 121, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/replication.py", line 199, in run
    self._update(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/replication.py", line 147, in _update
    tokenizer = tokenizer_factory.get_tokenizer_for_db(args.config)
  File "/usr/local/lib/nominatim/lib-python/nominatim/tokenizer/factory.py", line 88, in get_tokenizer_for_db
    basedir.mkdir()
  File "/usr/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/app/tokenizer'

trying to test what will go in my cron, and yup... gotta be an error LOL

[SenorKarlos]

I hodgepodged the sparse examples provided with the update doc for the standard install, and not sure what is wrong with it

[SenorKarlos]

I just ran --init sucessfully, but then when trying catch up again, with or without the MAX_DIFF, still gives the same permission error.

the hell lol? :D

[SenorKarlos]

hopefully someone can enlighten me, should be a standard install.

senorkarlos@tileserver:~/nominatim$ docker exec -it nominatim sudo -u nominatim nominatim replication --init
2024-03-31 15:36:50: Using project directory: /app
2024-03-31 15:36:50: Initialising replication updates
2024-03-31 15:36:53: Updates initialised at sequence 6023637 (2024-03-28 18:20:51+00:00)
2024-03-31 15:36:53: Create functions
senorkarlos@tileserver:~/nominatim$ docker exec -it nominatim sudo -u nominatim NOMINATIM_REPLICATION_MAX_DIFF=5000 nominatim replication --catch-up --threads 16
2024-03-31 15:37:02: Using project directory: /app
Traceback (most recent call last):
... same as above
senorkarlos@tileserver:~/nominatim$ docker exec -it nominatim sudo -u nominatim nominatim replication --check-for-updates
2024-03-31 15:54:26: Using project directory: /app
2024-03-31 15:54:27: New data available (6023637 => 6027717).
senorkarlos@tileserver:~/nominatim$ docker exec -it nominatim sudo -u nominatim nominatim replication --once
2024-03-31 15:55:11: Using project directory: /app
Traceback (most recent call last):
... same as above

So it can init and check but can't access the database? O.o

[SenorKarlos]

Decided to restart it and log the opening, hopefully it helps, just gonna move on for now. Y'all need a discord 😂

nominatim  | + tailpid=0
nominatim  | + replicationpid=0
nominatim  | + trap stopServices SIGTERM TERM INT
nominatim  | + /app/config.sh
nominatim  | + id nominatim
nominatim  | + useradd -m -p qaIACxO6wMR3 nominatim
nominatim  | + IMPORT_FINISHED=/var/lib/postgresql/14/main/import-finished
nominatim  | + '[' '!' -f /var/lib/postgresql/14/main/import-finished ']'
nominatim  | + chown -R nominatim:nominatim /nominatim
nominatim  | + service postgresql start
nominatim  |  * Starting PostgreSQL 14 database server
nominatim  |    ...done.
nominatim  | + cd /nominatim
nominatim  | + sudo -E -u nominatim nominatim refresh --website --functions
nominatim  | 2024-03-31 16:01:50: Using project directory: /nominatim
nominatim  | 2024-03-31 16:01:50: Create functions
nominatim  | 2024-03-31 16:01:56: Setting up website directory at /nominatim/website
nominatim  | + service apache2 start
nominatim  |  * Starting Apache httpd web server apache2
nominatim  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.21.0.2. Set the 'ServerName' directive globally to suppress this message
nominatim  |  *
nominatim  | + '[' https://download.geofabrik.de/north-america-updates/ '!=' '' ']'
nominatim  | + '[' '' '!=' true ']'
nominatim  | + sudo -E -u nominatim nominatim replication --project-dir /nominatim --init
nominatim  | 2024-03-31 16:01:58: Using project directory: /nominatim
nominatim  | 2024-03-31 16:01:58: Initialising replication updates
nominatim  | 2024-03-31 16:02:02: Updates initialised at sequence 4014 (2024-03-28 18:20:51+00:00)
nominatim  | 2024-03-31 16:02:02: Create functions
nominatim  | + '[' '' == continuous ']'
nominatim  | + '[' '' == once ']'
nominatim  | + '[' '' == catch-up ']'
nominatim  | + echo 'skipping replication'
nominatim  | skipping replication
nominatim  | + tailpid=106
nominatim  | + export NOMINATIM_QUERY_TIMEOUT=600
nominatim  | + NOMINATIM_QUERY_TIMEOUT=600
nominatim  | + export NOMINATIM_REQUEST_TIMEOUT=3600
nominatim  | + NOMINATIM_REQUEST_TIMEOUT=3600
nominatim  | + tail -Fv /var/log/postgresql/postgresql-14-main.log /var/log/apache2/access.log /var/log/apache2/error.log /var/log/replication.log
nominatim  | Warm database caches for search and reverse queries
nominatim  | + '[' '' = true ']'
nominatim  | + echo 'Warm database caches for search and reverse queries'
nominatim  | + sudo -H -E -u nominatim nominatim admin --warm
nominatim  | tail: cannot open '/var/log/replication.log' for reading: No such file or directory
nominatim  | ==> /var/log/postgresql/postgresql-14-main.log <==
nominatim  | 2024-03-31 16:01:47.547 UTC [36] LOG:  starting PostgreSQL 14.11 (Ubuntu 14.11-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
nominatim  | 2024-03-31 16:01:47.547 UTC [36] LOG:  listening on IPv4 address "0.0.0.0", port 5432
nominatim  | 2024-03-31 16:01:47.547 UTC [36] LOG:  listening on IPv6 address "::", port 5432
nominatim  | 2024-03-31 16:01:47.549 UTC [36] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
nominatim  | 2024-03-31 16:01:47.554 UTC [37] LOG:  database system was shut down at 2024-03-31 16:01:34 UTC
nominatim  | 2024-03-31 16:01:47.560 UTC [36] LOG:  database system is ready to accept connections
nominatim  |
nominatim  | ==> /var/log/apache2/access.log <==
nominatim  |
nominatim  | ==> /var/log/apache2/error.log <==
nominatim  | [Sun Mar 31 16:01:56.990860 2024] [mpm_prefork:notice] [pid 81] AH00163: Apache/2.4.52 (Ubuntu) configured -- resuming normal operations
nominatim  | [Sun Mar 31 16:01:56.990933 2024] [core:notice] [pid 81] AH00094: Command line: '/usr/sbin/apache2'
nominatim  | 2024-03-31 16:02:03: Using project directory: /nominatim
nominatim  | 2024-03-31 16:02:03: Warming database caches
nominatim  |
nominatim  | ==> /var/log/apache2/access.log <==
^Ccanceled
senorkarlos@tileserver:~/nominatim$ docker exec -it nominatim sudo -u nominatim nominatim replication --check-for-updates
2024-03-31 16:03:01: Using project directory: /app
2024-03-31 16:03:02: New data available (4014 => 6027725).
senorkarlos@tileserver:~/nominatim$ docker exec -it nominatim sudo -u nominatim nominatim replication --once
2024-03-31 16:04:24: Using project directory: /app
Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 12, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 228, in nominatim
    return get_set_parser().run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 121, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/replication.py", line 199, in run
    self._update(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/replication.py", line 147, in _update
    tokenizer = tokenizer_factory.get_tokenizer_for_db(args.config)
  File "/usr/local/lib/nominatim/lib-python/nominatim/tokenizer/factory.py", line 88, in get_tokenizer_for_db
    basedir.mkdir()
  File "/usr/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/app/tokenizer'
senorkarlos@tileserver:~/nominatim$ docker exec -it nominatim sudo -E -u nominatim nominatim replication --once
2024-03-31 16:04:43: Using project directory: /app
Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 12, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 228, in nominatim
    return get_set_parser().run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 121, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/replication.py", line 199, in run
    self._update(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/replication.py", line 147, in _update
    tokenizer = tokenizer_factory.get_tokenizer_for_db(args.config)
  File "/usr/local/lib/nominatim/lib-python/nominatim/tokenizer/factory.py", line 88, in get_tokenizer_for_db
    basedir.mkdir()
  File "/usr/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/app/tokenizer'

So now it has done it's own init for sure, set a completely different start value, and I still cannot run the command. Patch needed?? Something I did??

my yml

version: "3.7"

services:
  nominatim:
    container_name: nominatim
    image: mediagis/nominatim:4.4
    logging:
      driver: "json-file"
      options:
        max-size: "250m"
    restart: always
    ports:
      - '7070:8080'
    environment:
      PBF_URL: https://download.geofabrik.de/north-america-latest.osm.pbf
      REPLICATION_URL: https://download.geofabrik.de/north-america-updates/
      USER_AGENT: My/UserAgentRedacted
      IMPORT_WIKIPEDIA: 'true'
      IMPORT_US_POSTCODES: 'true'
      THREADS: 16
    volumes:
      - ./nominatim-data:/var/lib/postgresql/14/main
      - ./nominatim-flatnode:/nominatim/flatnode
    shm_size: '16gb'
networks:
  default:
    name: nominatim

[SenorKarlos]

Un-freakin believable

no offense and I say with as much respect as I can, as the project is amazing... but....

WHY THE HELL DOES YOUR DOCKER EXEC -IT EXAMPLE NOT MENTION --project-dir /nominatim!!!!!!!!

Saw that in someone elses command while browsing open issues.... it's started which is better than the other tries.

PLEASE FOR THE LOVE OF GOD clean up the update information. Just take the mode out of the environment, clean up the example and provide some more docker command examples, make it less confusing and provide the command that will actually work

[mtmail]

WHY THE HELL
PLEASE FOR THE LOVE OF GOD

Nobody works full-time on nominatim-docker. I provided answers as well as I could on my holiday. Help in form of PRs are welcome.

Y'all need a discord

I respectfully disagree.