Importing the entire planet #265

aviNrich · 2021-09-24T06:14:17Z

aviNrich
Sep 24, 2021

Hi.
Is it possible to import the entire world to the database?
If so, can you share the correct docker commands?

Thanks

leonardehrenfried · 2021-09-24T07:18:16Z

leonardehrenfried
Sep 24, 2021
Collaborator

Yes, it's possible but will require a machine with 64GB of memory: https://nominatim.org/release-docs/latest/admin/Installation/#hardware

The original planet file from https://planet.openstreetmap.org/ is throttled so you want to use a mirror from: https://wiki.openstreetmap.org/wiki/Planet.osm#Planet.osm_mirrors

Choose a mirror near your server for the best download speed.

If you're in Europe you could use the German mirror. I haven't tested it myself but i think it's this command:

docker run -it \
  -e PBF_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-latest.osm.pbf \
  -e REPLICATION_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day/ \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:3.7

You might need to also play around with the tuning parameters documented in the wiki.

Please report back if it works.

0 replies

aviNrich · 2021-09-27T06:46:43Z

aviNrich
Sep 27, 2021
Author

Thank you.

Can you please estimate the load time 4 cores + 64GB?
Once imported, can I reduce the memory?

Thanks

0 replies

mtmail · 2021-09-27T10:46:44Z

mtmail
Sep 27, 2021

Prepare for 5-10 days of importing. It's easier to estimate once the first couple of ranks are imported.

Disc speed is also a factor, use the fastest discs (or network mounted storage) you can find.

After the import you can lower the RAM.

1 reply

yikakabu Nov 4, 2022

After the import, the server running normally with ocuppied about 10 GB memory. But the memory usage starts to increase from a certain point, and now takes up 30GB.
What happened about the sudden growth? And How much memory should supply to daily use.

MichaelKorn · 2022-06-24T17:03:05Z

MichaelKorn
Jun 24, 2022

Thanks for the information here! Some recent insights:
I did the import on GCP with n2-highmem-16 (16 vCPUs, 128 GB) and SSD Disk. It took about 62 hours (+download time for pbf file).
Directly after the import 958GB were occupied on the nominatim disk (including 64GB for pbf file) (separate boot disk).

After the imported I scaled down to n2-custom-6-49152 (6 vCPUs, 48 GB) and "balanced persistent disk", it is running fine! It seems a smaller machine could also be fine, PostgreSQL is doing a great job.

The downloaded pbf file was a little bit outdated (June 11th) and daily updates ware applied (until today, June 24th). After the updates the nominatim disk is with 976GB occupied.

1 reply

leonardehrenfried Jun 25, 2022
Collaborator

Great update, thanks!

pgassmann · 2022-09-10T15:34:16Z

pgassmann
Sep 10, 2022

I started Nominatim 4.1 with planet import on a Hetzner Root Server AX101 with AMD Ryzen™ 9 5950X (16 cores) and 128 GB DDR4 ECC

It took around 2 days to import everything. plus ~1 night to get up-to-date with nominatim replication.

It uses now 428G on ZFS (zfs with lz4 compression)

zfsdata/container  referenced            428G                   -
zfsdata/container  compressratio         2.12x                  -

1 reply

leonardehrenfried Sep 10, 2022
Collaborator

Thanks for that!

Did you use the defaut command line options or did you customise it a bit?

mtmail · 2022-10-03T09:49:14Z

mtmail Oct 3, 2022

Open a new question for your specific setup

meyerovb · 2023-03-09T20:02:03Z

meyerovb Mar 9, 2023

Why would you run an rds? You're already paying for an ec2 to host the software, why not just run postgres on it? U use rds ur now paying data transfer fee (2 cents per gb but still) and the server spec for the postgres would probably rival the ec2, doubling your cost. Unless I'm guessing wrong here.

darkBuddha · 2022-10-26T10:27:43Z

darkBuddha
Oct 26, 2022

Import of Germany is possible on a M1 MacBook, volume size is 99 GB after import. It takes 8-12 hours, I used 12 GB RAM for the podman VM.

0 replies

yikakabu · 2022-11-04T08:31:55Z

yikakabu
Nov 4, 2022

After the import, the server running normally with ocuppied about 10 GB memory. But the memory usage starts to increase from a certain point, and now takes up 30GB.
What happened about the sudden growth? And How much memory should supply to daily use.

0 replies

meyerovb · 2023-03-09T20:19:14Z

meyerovb
Mar 9, 2023

Would using a 128gb ram 32 core cpu make it substantially faster for the initial load than 64gb ram 16 cores?

1 reply

mtmail Mar 9, 2023

duplicate of #428

nikitadtastewise · 2023-04-28T13:05:17Z

nikitadtastewise
Apr 28, 2023

Upload OSM data North America 12.3 GB + US Postcode.

Infrastructure configuration for initialization:

AWS RDS Postgres 14.6, db.r6g.xlarge (4vCPU, 32GB, arm), 300GB, 3000 IOPS, 125Mb/s throughput, volume autoscaling. Default AWS Postgres DB parameters.
EKS Job pod request 3vCPU + 6GB memory.
Enable flatnode feature to speed up process.

Total time initialize: 37 hours.
Total db size: 182 GB.

0 replies

ligoo · 2023-06-18T09:57:41Z

ligoo
Jun 18, 2023

The whole planet import on:

AMD Ryzen 7 7700, 8 cores, 16 threads
64 GB DDR5 RAM
2x1 TB Gen4 NVMe SSDs

Total time: ~38 hours.
Total DB size: 884G + 82G flatnode = 966G

The import was done with the IMPORT_WIKIPEDIA, the IMPORT_US_POSTCODES, IMPORT_GB_POSTCODES and IMPORT_TIGER_ADDRESSES options to true and shm_size to 16gb.

0 replies

sungwoo-lifeoverflow · 2023-10-13T00:26:27Z

sungwoo-lifeoverflow
Oct 13, 2023

The whole planet import ( `IMPORT_STYLE`=admin )

Docker image : mediagis/nominatim:4.3
Machine : aws ec2 m5.2xlarge ( 32GB RAM & vCPU 8 ) & EBS 1TB

At EC2, before run docker

make folder mkdir -p /nominatim-data/share (for docker mount)
cd /nominatim-data/share & manual download planet-latest.osm.pbf (71.x GB) ~1.5hr (delete after init)
mkdir -p /nominatim-data/postgresql (for docker mount)
mkdir -p /nominatim-data/flatnode (for docker mount)

"init command (init.sh)"

#!/bin/sh

HOST_CACHE_DIR_PATH="/nominatim-data"
NOMINATIM_PASSWORD="nominatim_password"

docker run -it \
  --shm-size=8g \
  -e THREADS=8 \
  -e PBF_PATH=/nominatim/share/planet-latest.osm.pbf \
  -e REPLICATION_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day \
  -e NOMINATIM_PASSWORD=$NOMINATIM_PASSWORD \
  -e IMPORT_STYLE=admin \
  -v $HOST_CACHE_DIR_PATH/postgresql:/var/lib/postgresql/14/main \
  -v $HOST_CACHE_DIR_PATH/flatnode:/nominatim/flatnode \
  -v $HOST_CACHE_DIR_PATH/share:/nominatim/share \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:4.3

run init script & simple logs (~24 hours)

2023-10-11 09:51:59: (run init.sh)
2023-10-12 05:01:42 - rank 20
2023-10-12 06:27:57: Done 2285867 in 7330 @ 311.833 per second - rank 20 ETA (seconds): 2301.67
2023-10-12 06:48:54: Done 2681964 in 8586 @ 312.332 per second - rank 20 ETA (seconds): 1029.80
2023-10-12 09:09:29: Done 1344800 in 1445 @ 930.395 per second - rank 0 ETA (seconds): 465.01
2023-10-12 09:31:34: Done 125874/125874 in 60 @ 2074.746 per second - FINISHED postcodes (location_postcode)
2023-10-12 09:33:26: Post-process tables
2023-10-12 09:58:27: Setting up website directory at /nominatim/website

Final Size ( DB+flatnode = 321G + 84G = 405G )

# Terminal
[... nominatim-data] sudo du -d 1 -h
84G     ./flatnode
72G     ./share ---> (planet-latest.osm.pbf, not necessary)
321G    ./postgresql
476G    .

3 replies

leonardehrenfried Oct 13, 2023
Collaborator

Thanks for the report. These are valuable for others wanting to import the entire planet.

willb0 Oct 22, 2023

bless up bro ty

Curiosity007 Nov 19, 2023

The whole planet import ( `IMPORT_STYLE`=admin )

Docker image : mediagis/nominatim:4.3
Machine : aws ec2 m5.2xlarge ( 32GB RAM & vCPU 8 ) & EBS 1TB

At EC2, before run docker

make folder mkdir -p /nominatim-data/share (for docker mount)
cd /nominatim-data/share & manual download planet-latest.osm.pbf (71.x GB) ~1.5hr (delete after init)
mkdir -p /nominatim-data/postgresql (for docker mount)
mkdir -p /nominatim-data/flatnode (for docker mount)

"init command (init.sh)"

#!/bin/sh

HOST_CACHE_DIR_PATH="/nominatim-data"
NOMINATIM_PASSWORD="nominatim_password"

docker run -it \
  --shm-size=8g \
  -e THREADS=8 \
  -e PBF_PATH=/nominatim/share/planet-latest.osm.pbf \
  -e REPLICATION_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day \
  -e NOMINATIM_PASSWORD=$NOMINATIM_PASSWORD \
  -e IMPORT_STYLE=admin \
  -v $HOST_CACHE_DIR_PATH/postgresql:/var/lib/postgresql/14/main \
  -v $HOST_CACHE_DIR_PATH/flatnode:/nominatim/flatnode \
  -v $HOST_CACHE_DIR_PATH/share:/nominatim/share \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:4.3

run init script & simple logs (~24 hours)

2023-10-11 09:51:59: (run init.sh)
2023-10-12 05:01:42 - rank 20
2023-10-12 06:27:57: Done 2285867 in 7330 @ 311.833 per second - rank 20 ETA (seconds): 2301.67
2023-10-12 06:48:54: Done 2681964 in 8586 @ 312.332 per second - rank 20 ETA (seconds): 1029.80
2023-10-12 09:09:29: Done 1344800 in 1445 @ 930.395 per second - rank 0 ETA (seconds): 465.01
2023-10-12 09:31:34: Done 125874/125874 in 60 @ 2074.746 per second - FINISHED postcodes (location_postcode)
2023-10-12 09:33:26: Post-process tables
2023-10-12 09:58:27: Setting up website directory at /nominatim/website

Final Size ( DB+flatnode = 321G + 84G = 405G )

# Terminal
[... nominatim-data] sudo du -d 1 -h
84G     ./flatnode
72G     ./share ---> (planet-latest.osm.pbf, not necessary)
321G    ./postgresql
476G    .

Is there a way to export the entire docker structure or the folders inside the docker, and replicate in local Ubuntu box?

I would require ec2 to create the initial configuration, but once it is done, I can comfortably run it in my local machine. Is there a way to do this?

mikeresnik · 2023-11-25T14:06:23Z

mikeresnik
Nov 25, 2023

I ran into out of memory / out of space issues with the simple m5.2xlarge ec2 instance at 1TB.
Currently am trying a m5.4xlarge (64gb) with 1.5TB storage since I remember seeing that 64gb - 128gb the minimum recommended.
Hoping to make this into an AMI and spin up on machines with lower RAM once running.

I am also importing US postcodes, GB postcodes, and TIGER data which adds to the data size some.

1 reply

leonardehrenfried Nov 26, 2023
Collaborator

Thanks for the report. They are useful because both nominatim and the data evolves over time and hardware requirements change.

lnbs97 · 2024-03-09T12:20:51Z

lnbs97
Mar 9, 2024

For reference: importing Germany took me 10hrs and reverse geocoding 55k coordinate pairs took 16mins.

Container Size: 108GB
System: AMD Ryzen 5 3600, 32GB RAM, 1TB NVMe SSD

0 replies

nikitadtastewise · 2024-03-11T07:20:15Z

nikitadtastewise
Mar 11, 2024

Import multiple regions version 4.3.0:

North America 14.6 GB + US and GB Postcode + TIGER data
Australia and Oceania 1.2 GB
France 4.5 GB
Germany 4.2 GB
India 1.46 GB
Spain 1.2 GB
United Kingdom 1.78 GB

Hardware:

RDS Postgres 14.10, db.r6g.xlarge 4vCPU, 32GB, 550GB
EC2 r5.xlarge 4vCPU, 32GB, 200GB root, 150GB flatnode file

ETA

Import multiple regions + US and GB postcodes extra ~ 3 days 18 hours
Import tigger data ~ 9 hours 15 mins

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Importing the entire planet #265

{{title}}

Replies: 15 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

This comment was marked as off-topic.

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

The whole planet import ( `IMPORT_STYLE`=admin )

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Importing the entire planet #265

Replies: 15 comments · 10 replies

leonardehrenfried Sep 24, 2021 Collaborator

aviNrich Sep 27, 2021 Author

leonardehrenfried Jun 25, 2022 Collaborator

leonardehrenfried Sep 10, 2022 Collaborator

This comment was marked as off-topic.

The whole planet import ( IMPORT_STYLE=admin )

leonardehrenfried Oct 13, 2023 Collaborator

The whole planet import ( IMPORT_STYLE=admin )

leonardehrenfried Nov 26, 2023 Collaborator

Replies: 15 comments 10 replies

leonardehrenfried
Sep 24, 2021
Collaborator

aviNrich
Sep 27, 2021
Author

leonardehrenfried Jun 25, 2022
Collaborator

leonardehrenfried Sep 10, 2022
Collaborator

The whole planet import ( `IMPORT_STYLE`=admin )

leonardehrenfried Oct 13, 2023
Collaborator

The whole planet import ( `IMPORT_STYLE`=admin )

leonardehrenfried Nov 26, 2023
Collaborator