No space left device.

[gmalenko]

Describe the bug
When I run my docker command. The process seems to proceed fine but will eventually return the message

systemd-journald: Failed to create new system journal: No space left on device.
systemd-journald: Failed to create new user journal: No space left on device

To Reproduce
This is the command I'm using

docker run -it --rm --shm-size=29g \
  -e PBF_URL=https://download.geofabrik.de/north-america/us-latest.osm.pbf \
  -e REPLICATION_URL=https://download.geofabrik.de/north-america/us-updates/ \
  -e IMPORT_WIKIPEDIA=false \
  -v /ssdstorage2/nominatimdata/postgresdata:/var/lib/postgresql/12/main \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:4.2

Expected behavior
Expecting it to not fail!

Screenshots & Logs
If applicable, add screenshots & logs to help explain your problem.

Desktop / Server (please complete the following information):

• OS & Version: Ubuntu 20.04.2 LTS
• Docker Version: 20.10.22, build 3a2c30b
• Nominatim Version: 4.2

Additional context
ssdstorage2 has plenty of space. Its 1 tb in size and this would have been the only thing on the drive.
I only get this in versions 4.2 and 4.1. However, it seems to be working with version 4.0 and below (I was running 3.7 for a while)

[alvasilvao]

Hi, I am having the same problem. I managed to install Monaco without problems, but France and Europe gave me this problem. I am now installing the Alpes region of France, it hasn't finished but I can see that it has already used more than 10GB on my VM disk.

[gsg-git]

I think you should use the flatnode file:

-v nominatim-flatnode:/nominatim/flatnode

[lublak]

@gsg-git i tried this but this doesn't help:

sudo docker run -it \
  -v nominatim-flatnode:/nominatim/flatnode \
  -e PBF_URL=https://download.geofabrik.de/europe/germany-latest.osm.pbf \
  -e REPLICATION_URL=https://download.geofabrik.de/europe/germany-updates/ \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:4.3

023-12-10 23:32:23: Done 69229 in 36 @ 1920.503 per second - rank 25 ETA (seconds): 9.85
2023-12-10 23:32:24: Done 71149 in 37 @ 1913.688 per second - rank 25 ETA (seconds): 8.88
2023-12-10 23:32:25: Done 73062 in 38 @ 1912.173 per second - rank 25 ETA (seconds): 7.89
2023-12-10 23:32:26: Done 74974 in 39 @ 1909.916 per second - rank 25 ETA (seconds): 6.89
2023-12-10 23:32:27: Done 76883 in 40 @ 1916.580 per second - rank 25 ETA (seconds): 5.87
2023-12-10 23:32:28: Done 78799 in 41 @ 1915.197 per second - rank 25 ETA (seconds): 4.88
2023-12-10 23:32:29: Done 80714 in 42 @ 1914.132 per second - rank 25 ETA (seconds): 3.88
2023-12-10 23:32:30: Done 82628 in 43 @ 1915.103 per second - rank 25 ETA (seconds): 2.88
2023-12-10 23:32:31: Done 84543 in 44 @ 1917.137 per second - rank 25 ETA (seconds): 1.88
2023-12-10 23:32:32: Done 86460 in 45 @ 1912.156 per second - rank 25 ETA (seconds): 0.88
2023-12-10 23:32:33: Done 88141/88141 in 46 @ 1912.823 per second - FINISHED rank 25

2023-12-10 23:32:41: Starting indexing rank (26 to 30) using 16 threads
2023-12-10 23:32:41: Starting rank 26 (using batch size 1)
Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 12, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 225, in nominatim
    return get_set_parser().run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 121, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 134, in run
    indexer.index_full(analyse=not args.index_noanalyse)
  File "/usr/local/lib/nominatim/lib-python/nominatim/indexer/indexer.py", line 140, in index_full
    if self.index_by_rank(26, 30) > 1000:
  File "/usr/local/lib/nominatim/lib-python/nominatim/indexer/indexer.py", line 174, in index_by_rank
    total += self._index(runners.RankRunner(rank, analyzer), 20 if rank == 30 else 1)
  File "/usr/local/lib/nominatim/lib-python/nominatim/indexer/indexer.py", line 210, in _index
    total_tuples = cur.scalar(runner.sql_count_objects())
  File "/usr/local/lib/nominatim/lib-python/nominatim/db/connection.py", line 53, in scalar
    self.execute(sql, args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/db/connection.py", line 36, in execute
    super().execute(query, args)
  File "/usr/lib/python3/dist-packages/psycopg2/extras.py", line 146, in execute
    return super().execute(query, vars)
psycopg2.errors.DiskFull: could not resize shared memory segment "/PostgreSQL.571434360" to 50438144 bytes: No space left on device

[alvasilvao]

I have just tried again but for the whole planet and with a new computer (196GB RAM and 4TB) and after 1,5 days and more than 2TB later I got this message, pretty much the same you received.

@gsg-git any suggestions?

I used this:
docker run -it
-v nominatim-flatnode:/nominatim/flatnode
-e PBF_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-latest.osm.pbf
-e REPLICATION_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day/
-p 8080:8080
--name nominatim
mediagis/nominatim:4.3

[tranquocnhat]

My docker-compose

I also meet problem

And still plenty of storage for flatnode

[tranquocnhat]

I also format many type filesystem and rerun but no affect btrfs, xfs

[alvasilvao]

I have just tried again (started last 1,5 days ago) and it stopped again after RANK 26.
What can I try?

[leonardehrenfried]

Here is a thread giving you an idea what hardware you need to import the entire planet and how long it takes: #265

[leonardehrenfried]

It seems like it's normal for an import to take several days.

[alvasilvao]

Hi, thanks for your answer. I am runningn a PC with 2,5 TB of free disk and 192 GB of ram. That is why I am surprised that I don't manage to get over RANK 26.
Am I doing anything wrong? Maybe a set up on Docker or on my prompt?

[gmalenko]

I'm guessing that there's no actual resolution to the original problem yet?

[leonardehrenfried]

If you get the error in the title you're running out of either RAM or disk. I would triple check that you're assigning enough resources to the container. For example, docker on windows runs in a VM which by default limits RAM and disk.

On other OSs there might be other restrictions and limits.

[gmalenko]

If you get the error in the title you're running out of either RAM or disk. I would triple check that you're assigning enough resources to the container. For example, docker on windows runs in a VM which by default limits RAM and disk.

On other OSs there might be other restrictions and limits.

I'm a few months removed now from the issue. But there was plenty of ram and plenty of hard drive space. In fact, the previous version of nominatim works just fine (and is still running today). Nothing was different between the two containers except the version.

[tranquocnhat]

@gmalenko what previous version nominatim you use?

[gmalenko]

@gmalenko what previous version nominatim you use?

@tranquocnhat 3.7 at the moment.

[leonardehrenfried]

For all issues where I believe that people are running into resource problems I've added a new label: https://github.com/mediagis/nominatim-docker/issues?q=label%3A%22resource+problem%22

Maybe other issues give you a clue what the problem is.

[gsg-git]

I set nominatim on its own server without virtualisation.
4TB M2 drive and 64GB RAM with ubuntu and i used:

docker run -it \ --shm-size=2g \ -e THREADS=5 \ -v nominatim-flatnode:/nominatim/flatnode \ -e POSTGRES_SHARED_BUFFERS=2GB \ -e POSTGRES_MAINTAINENCE_WORK_MEM=1GB \ -e POSTGRES_AUTOVACUUM_WORK_MEM=2GB \ -e POSTGRES_WORK_MEM=50MB \ -e POSTGRES_EFFECTIVE_CACHE_SIZE=4GB \ -e POSTGRES_SYNCHRONOUS_COMMIT=off \ -e POSTGRES_MAX_WAL_SIZE=1GB \ -e POSTGRES_CHECKPOINT_TIMEOUT=10min \ -e POSTGRES_CHECKPOINT_COMPLETITION_TARGET=0.9 \ -e PBF_URL=http://download.geofabrik.de/africa/rwanda-latest.osm.pbf \ -e REPLICATION_URL=http://download.geofabrik.de/africa/rwanda-updates/ \ -e IMPORT_STYLE=admin \ -e IMPORT_WIKIPEDIA=false \ -p 8080:8080 \ --name nominatim42 \ mediagis/nominatim:4.2

I would try to import a smaller country to check if the script is running until the end.
And if you want to use nominatim long therm you should mind to keep the Database not in the same container or locally and the flantnode file too, because updating docker nominatim means reating a new container with the new version. So you have just set new "nominatimverionconainer" flatnode and DB path of the old version and youre done normally.