sc functionality on large networks

[jucardwell]

Looking for tips on utilizing dodgr sc functionality with OSM networks that are too large to extract using osmdata. Is there functionality to translate an sf output from osmextract to sc format within dodgr? Or is the best practice to make many small requests to osmdata? For context, I am extracting a road network for an entire US state

[mpadge]

Thanks for asking @jucardwell, Short answer: You can't do it directly in R. The overpass server behind osmdata won't deliver data that large, and all other options (sf, osmextract, ...) read through GDAL which strips away all of the information needed for routing. The good news: You can do it with osmium-tool. My general procedure for that is:

1. Download OSM data in pbf format from geofabrik
2. Cut to desired are using osmium extract and generate a pbf output
3. Filter to desired way types using osmium tags-filter and generate a .osm output (by simply specifying ".osm" as the suffix on the object file after -o).

You can then use osmdata to read that in and convert to SC format, and then to construct a network with dodgr.

Your challenge will likely be selecting an appropriate sub-set of key-value pairs. I presume you're likely interested in routing motorized vehicles (but correct me if i'm wrong!), so guess you'll only need "highway", and I'd initially recommend trying some minimal set like `highway=motorway|motorway_link|trunk|trunk_link|primary|primary_link". See how that goes, and expand to lesser road categories if you can/want.

Feel free to ask any other questions, or to close the issue if that helps you. Thanks

[jucardwell]

Got it, thanks. I think I have the file extracted, filtered, and in .osm format but receiving a bad request error when using command osmdata_sc(). Is that the correct command for reading in?

[mpadge]

You need to pass the .osm file as the "doc" parameter: x <- osmdata_sc(doc = "mydata.osm"). (The query parameter is optional.) It should then make no external calls at all, so you should never see anything like a bad request error.

[jucardwell]

Hi @mpadge, thanks for all the help so far.

I've been able to construct an sc network for my area of interest and weight the graph. However, I'm facing fatal error/ R abort when executing dodgr_times(). Figured it could be a computational limit on my local machine, so I also attempted running it on a computing cluster and received a segfault error (*** caught segfault ***
address 0xfffffffdf871f020, cause 'memory not mapped'). I've also tried disabling parallel processing and received the same error. I'm wondering if you can think of an approach that would limit memory usage (for instance, iterating through each routing request), or if you think it is the pure size of the sc network that causes the error. For context, I am routing from approx 7,000 locations to approx 100 locations and dodgr_times() executes successfully on the same size network in sf format.

Thanks for any thoughts

[mpadge]

@jucardwell I'm going to presume this is now dormant and close, but feel free to ask any further questions.