problem importing North America

[ashley-mort]

Any ideas on why this is dying? Is it a hardware limitation? I'm just on my desktop that has 300GB free and 16GB RAM.

docker run -it -e PBF_PATH=/nominatim/data/north-america-latest.osm.pbf -p 8080:8080 -v C:\nominatimData:/nominatim/data --name nominatim mediagis/nominatim:4.2

+ tailpid=0
+ replicationpid=0
+ trap stopServices SIGTERM TERM INT
+ /app/config.sh
+ id nominatim
+ useradd -m -p qaIACxO6wMR3 nominatim
+ IMPORT_FINISHED=/var/lib/postgresql/14/main/import-finished
+ '[' '!' -f /var/lib/postgresql/14/main/import-finished ']'
+ /app/init.sh
+ OSMFILE=/nominatim/data.osm.pbf
+ CURL='curl -L -A mediagis/nominatim-docker:4.2.0 --fail-with-body'
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional Wikipedia importance import'
Skipping optional Wikipedia importance import
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional GB postcode import'
Skipping optional GB postcode import
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional US postcode import'
Skipping optional US postcode import
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional Tiger addresses import'
Skipping optional Tiger addresses import
+ '[' '' '!=' '' ']'
+ '[' /nominatim/data/north-america-latest.osm.pbf '!=' '' ']'
+ echo Reading OSM extract from /nominatim/data/north-america-latest.osm.pbf
Reading OSM extract from /nominatim/data/north-america-latest.osm.pbf
+ OSMFILE=/nominatim/data/north-america-latest.osm.pbf
+ '[' '!' -f /var/lib/postgresql/14/main/PG_VERSION ']'
+ cp /etc/postgresql/14/main/conf.d/postgres-import.conf.disabled /etc/postgresql/14/main/conf.d/postgres-import.conf
+ sudo service postgresql start
 * Starting PostgreSQL 14 database server                                                                                                                                        [ OK ]
+ sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='\''nominatim'\'''
+ grep -q 1
+ sudo -E -u postgres createuser -s nominatim
+ sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='\''www-data'\'''
+ grep -q 1
+ sudo -E -u postgres createuser -SDR www-data
+ sudo -E -u postgres psql postgres -tAc 'ALTER USER nominatim WITH ENCRYPTED PASSWORD '\''qaIACxO6wMR3'\'''
ALTER ROLE
+ sudo -E -u postgres psql postgres -tAc 'ALTER USER "www-data" WITH ENCRYPTED PASSWORD '\''qaIACxO6wMR3'\'''
ALTER ROLE
+ sudo -E -u postgres psql postgres -c 'DROP DATABASE IF EXISTS nominatim'
NOTICE:  database "nominatim" does not exist, skipping
DROP DATABASE
+ chown -R nominatim:nominatim /nominatim
+ cd /nominatim
+ '[' '' = true ']'
+ sudo -E -u nominatim nominatim import --osm-file /nominatim/data/north-america-latest.osm.pbf --threads 16
2023-03-17 18:17:55: Using project directory: /nominatim
2023-03-17 18:17:57: Creating database
2023-03-17 18:17:57: Setting up country tables
2023-03-17 18:17:58: Importing OSM data file
2023-03-17 18:17:58  osm2pgsql version 1.7.1
2023-03-17 18:17:58  Database version: 14.5 (Ubuntu 14.5-0ubuntu0.22.04.1)
2023-03-17 18:17:58  PostGIS version: 3.2
2023-03-17 18:17:58  Parsing gazetteer style file '/usr/local/etc/nominatim/import-full.style'.
NOTICE:  table "place" does not exist, skipping
Processing: Node(553320k 1246.2k/s) Way(0k 0.00k/s) Relation(0 0.0/s)Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 14, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim
    return parser.run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 94, in run
    database_import.import_osm_data(files,
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 113, in import_osm_data
    run_osm2pgsql(options)
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 159, in run_osm2pgsql
    subprocess.run(cmd, cwd=options.get('cwd', '.'),
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--log-progress', 'true', '--number-processes', '1', '--cache', '6209', '--style', '/usr/local/etc/nominatim/import-full.style', '--output', 'gazetteer', '--create', '--with-forward-dependencies', 'false', '/nominatim/data/north-america-latest.osm.pbf']' died with <Signals.SIGKILL: 9>.

[mtmail]

https://github.com/mediagis/nominatim-docker/tree/master/4.2#flatnode-files "This is advised for bigger imports (Europe, North America etc.)"
Monitor the RAM usage on your machine. It's hard to remote-debug if your system ran out of memory.

[noelstieglitz]

For future readers, if you see a similar error after the fact (and can't watch current memory, you can run this:
sudo dmesg -eT
and look for something like this:
Out of memory: Killed process 1234 (osm2pgsql)...