How long is building the initial graph supposed to take? #129

Dunge · 2023-12-20T03:04:33Z

Dunge
Dec 20, 2023

I started the docker with a 13.6gb north-america-latest.osm.pbf file. I'm using Docker on Windows with WSL as the backend which is limited to 5gb of memory.

In the first hour I got a few warnings about the graph, seems normal. But now It's been running for 7 hours and I haven't had anything progressing since the last 6. I'm not even sure if it's processing anymore. It have the RAM filled up, but barely using any CPU.

2023-12-19 15:19:00 
2023-12-19 15:19:00 INFO: Running container with user valhalla UID 59999 and GID 59999.
2023-12-19 15:19:00 
2023-12-19 15:19:00 find: '/custom_files/transit_tiles': No such file or directory
2023-12-19 15:19:00 WARNING: Hash not found for: /custom_files/north-america-latest.osm.pbf!
2023-12-19 15:19:00 WARNING: No routing tiles found at /custom_files/valhalla_tiles.tar or /custom_files/valhalla_tiles, starting a new build
2023-12-19 15:19:00 INFO: Running build tiles with: /custom_files/valhalla.json /custom_files/north-america-latest.osm.pbf
2023-12-19 15:19:00 
2023-12-19 15:19:00 ============================
2023-12-19 15:19:00 = Build the initial graph. =
2023-12-19 15:19:00 ============================
2023-12-19 15:19:00 2023/12/19 20:19:00.212219 [INFO] Running valhalla_build_tiles with 16 thread(s).
2023-12-19 15:19:00 2023/12/19 20:19:00.212272 [INFO] Start stage = initialize End stage = build
2023-12-19 15:19:00 2023/12/19 20:19:00.215442 [INFO] Parsing files for ways: /custom_files/north-america-latest.osm.pbf
2023-12-19 15:19:00 2023/12/19 20:19:00.243542 [INFO] Parsing ways...
2023-12-19 15:29:00 2023/12/19 20:29:00.841381 [INFO] invalid_argument thrown for condition (Jul 1- Oct 1)
2023-12-19 15:34:24 2023/12/19 20:34:24.206601 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:24 2023/12/19 20:34:24.206999 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:24 2023/12/19 20:34:24.207455 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:24 2023/12/19 20:34:24.208313 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.085669 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.086286 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.086694 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.087238 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.087731 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.088193 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.088663 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.089347 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.089767 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.090201 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.090708 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:34:37 2023/12/19 20:34:37.091125 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:35:58 2023/12/19 20:35:58.927536 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.436604 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.437023 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.450281 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.451319 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.455878 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.456303 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.456720 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.462230 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.463239 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.463670 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.464148 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.470703 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.471102 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.471614 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.472921 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.473339 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.473790 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.479644 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.480029 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:36:32 2023/12/19 20:36:32.481715 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:40:18 2023/12/19 20:40:18.438381 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:42:33 2023/12/19 20:42:33.016968 [INFO] invalid_argument thrown for condition (Jun 15-Labor Day)
2023-12-19 15:43:31 2023/12/19 20:43:31.160677 [INFO] invalid_argument thrown for condition (Jun 15-Labor Day)
2023-12-19 15:44:08 2023/12/19 20:44:08.822931 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 15:54:23 2023/12/19 20:54:23.953123 [WARN] Error during parsing of `layer` tag on the way 769745481: stoi
2023-12-19 16:03:21 2023/12/19 21:03:21.657695 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:03:21 2023/12/19 21:03:21.658516 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:03:21 2023/12/19 21:03:21.658948 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:03:45 2023/12/19 21:03:45.676607 [WARN] Error during parsing of `layer` tag on the way 987800239: stoi
2023-12-19 16:05:13 2023/12/19 21:05:13.956021 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:05:14 2023/12/19 21:05:14.905183 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:05:14 2023/12/19 21:05:14.906041 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:05:14 2023/12/19 21:05:14.906795 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:05:14 2023/12/19 21:05:14.907572 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:05:14 2023/12/19 21:05:14.908378 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:05:14 2023/12/19 21:05:14.909130 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:05:14 2023/12/19 21:05:14.909918 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:05:14 2023/12/19 21:05:14.928173 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:05:14 2023/12/19 21:05:14.928579 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:07:46 2023/12/19 21:07:46.532974 [INFO] invalid_argument thrown for condition (Apr-Sep 20:00-06:00, Oct-Mar 18:00-06:00)
2023-12-19 16:11:05 2023/12/19 21:11:05.303264 [WARN] Error during parsing of `layer` tag on the way 1229020034: stoi

Answered by nilsnolde

Dec 20, 2023

From the project description it led me to believe this was lightweight and didn't require that much memory and could be run on low-end devices, doesn't seem to be the case for building.

Correct, that's not the case for building which is anything else but light-weight. That only goes for runtime.

5 GB is quite low for NA and I wouldn't be surprised if it fails for you in a later stage. valhalla_build_tiles -h will list them for you. You just finished the parsing phase (I think), subsequent stuff will mostly run multi-threaded.

Maybe some general guidelines on what to expect in term of time, disk space, and memory/cpu requirements could be added to the project homepage...

Sure, someone …

View full answer

Dunge · 2023-12-20T18:20:40Z

Dunge
Dec 20, 2023
Author

22 hours in the process, after a 12 hours pause with no output I finally got some output. How many steps are left?
From the project description it led me to believe this was lightweight and didn't require that much memory and could be run on low-end devices, doesn't seem to be the case for building.

I just hope my Winx64 WSL built data will be compatible on a Linux x64 docker.

Maybe some general guidelines on what to expect in term of time, disk space, and memory/cpu requirements could be added to the project homepage...

2023-12-20 04:35:07 2023/12/20 09:35:07.958168 [INFO] Added 195301 culdesac roundabouts from 264461 candidates.
2023-12-20 04:35:07 2023/12/20 09:35:07.962471 [INFO] Finished with 50398259 routable ways containing 517499688 nodes
2023-12-20 04:35:18 2023/12/20 09:35:18.378189 [INFO] Sorting osm access tags by way id...
2023-12-20 04:37:08 2023/12/20 09:37:08.609233 [INFO] Finished
2023-12-20 04:37:11 2023/12/20 09:37:11.001874 [INFO] Write OSMData to temp files
2023-12-20 04:37:12 2023/12/20 09:37:12.260931 [INFO] Done
2023-12-20 04:37:12 2023/12/20 09:37:12.266786 [INFO] Parsing files for relations: /custom_files/north-america-latest.osm.pbf
2023-12-20 04:37:12 2023/12/20 09:37:12.274742 [INFO] Parsing relations...
2023-12-20 05:04:13 2023/12/20 10:04:13.361252 [INFO] invalid_argument thrown for condition  Jul 1- Aug 31 off
2023-12-20 05:04:20 2023/12/20 10:04:20.377031 [INFO] Finished with 414780 simple restrictions
2023-12-20 05:04:20 2023/12/20 10:04:20.377067 [INFO] Finished with 0 lane connections
2023-12-20 05:04:20 2023/12/20 10:04:20.548279 [INFO] Sorting complex restrictions by from id...
2023-12-20 05:04:21 2023/12/20 10:04:21.631135 [INFO] Sorting complex restrictions by to id...
2023-12-20 05:04:22 2023/12/20 10:04:22.524673 [INFO] Finished
2023-12-20 05:04:22 2023/12/20 10:04:22.525476 [INFO] Write OSMData to temp files
2023-12-20 05:04:23 2023/12/20 10:04:23.661299 [INFO] Done
2023-12-20 05:04:23 2023/12/20 10:04:23.662431 [INFO] Parsing files for nodes: /custom_files/north-america-latest.osm.pbf
2023-12-20 05:04:23 2023/12/20 10:04:23.662986 [INFO] Sorting osm way node references by node id...

6 replies

nilsnolde Dec 20, 2023
Maintainer

From the project description it led me to believe this was lightweight and didn't require that much memory and could be run on low-end devices, doesn't seem to be the case for building.

Correct, that's not the case for building which is anything else but light-weight. That only goes for runtime.

5 GB is quite low for NA and I wouldn't be surprised if it fails for you in a later stage. valhalla_build_tiles -h will list them for you. You just finished the parsing phase (I think), subsequent stuff will mostly run multi-threaded.

Maybe some general guidelines on what to expect in term of time, disk space, and memory/cpu requirements could be added to the project homepage...

Sure, someone can add that, doesn't need to be us, it's all open. It varies with CPU, memory and a whole lot of option in the config, so it's not straight-forward.

Answer selected by nilsnolde

Dunge Dec 20, 2023
Author

Yeah, that 5GB limit was something I previously set for another usage. Had I known I would have unlocked it before starting this.

But now I'm hesitating if it will be shorter to just kill it and starting over with my full 32gb, or just let it finish. Let's say I wasn't expecting this process to span over days.

valhalla_build_tiles -h doesn't seem to list the steps:

$ valhalla_build_tiles -h
valhalla_build_tiles 3.4.0

a program that creates the route graph
from one or multiple osm.pbf extract(s)

Usage:
  valhalla_build_tiles [OPTION...] OSM PBF file(s)

  -h, --help               Print this help message.
  -v, --version            Print the version of this software.
  -c, --config arg         Path to the configuration file
  -i, --inline-config arg  Inline JSON config
  -s, --start arg          Starting stage of the build pipeline (default:
                           initialize)
  -e, --end arg            End stage of the build pipeline (default: cleanup)
  -j, --concurrency arg    Number of threads to use. Defaults to all threads.

Of course everything varies, but maybe something like "have as much or double memory of the file you try to parse" and "the output will be 10x its size" (currently my 13gb files generated 91gb of files and it's not over).

nilsnolde Dec 20, 2023
Maintainer

16 GB is more than enough for NA to build. In any case what really matter is the CPU clock and how many there are. RAM is just for some caching mostly (above the very minimum at least).

Right, we should print them on the normal -h. It'll print if you put some bogus input for --start or --end.

Dunge Dec 20, 2023
Author

My docker CPU usage is 0.04%, its practically idling. I have a high-end Intel i7, shouldn't be a problem.

I found a sample of full output ran on a small country here: https://water-gis.com/en/setups/valhalla/setup-valhalla/
From what I can see, on his full 22min process, I'm only one minute in. So assuming proportions remains the same, it would take a month before my process complete? 😆 . Let's hope after sorting the nodes it will start doing CPU work as you mentioned and speed up.

nilsnolde Dec 20, 2023
Maintainer

I can tell you that the public instance has 16 threads (not the best CPUs) and builds the planet in 1.5 days.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How long is building the initial graph supposed to take? #129

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How long is building the initial graph supposed to take? #129

Dunge Dec 20, 2023

Replies: 1 comment · 6 replies

Dunge Dec 20, 2023 Author

nilsnolde Dec 20, 2023 Maintainer

Dunge Dec 20, 2023 Author

nilsnolde Dec 20, 2023 Maintainer

Dunge Dec 20, 2023 Author

nilsnolde Dec 20, 2023 Maintainer

Dunge
Dec 20, 2023

Replies: 1 comment 6 replies

Dunge
Dec 20, 2023
Author

nilsnolde Dec 20, 2023
Maintainer

Dunge Dec 20, 2023
Author

nilsnolde Dec 20, 2023
Maintainer

Dunge Dec 20, 2023
Author

nilsnolde Dec 20, 2023
Maintainer