Render ref from route relationship #596
Comments
|
Actually it is not entirely clear to me what is the recommended tagging. Perhaps tagging ref's on routes is incorrect tagging? |
|
The same holds for name's on relations. |
no, recommended are refs on routes, and eventually on road ways as well, but the latter just for mapnik ;-) |
|
I'd like to put in a vote to have this implemented ASAP. OSM recommended tagging for a road route with a ref= is to use relations, e.g. http://www.openstreetmap.org/way/21548058 and NOT to put this information on individual ways. Since Mapnik does not render road route relations at all, map editors don't consider a route 'done' until all the information is duplicated on individual ways, meaning it all has to be cleaned up later - and the longer this is unimplemented (and the more editors we get, especially say, in the US where OSM is growing), the worse this gets. This is a case of users modifying OSM's database to work around a missing feature in a tool (here, support of road route relations). A quick and dirty way of implementing this, presuming that relations can be read to render, would be to recursively get all road route relations on a way if the way is tagged with highway={motorway,trunk,primary,secondary,tertiary,unclassified, or residential}. De-dup any identical ref names and drop them - those could come from duplication on the way/relation/super-relations. Then, render them one per line. So, for example...this example render would be three lines. I realize this introduces the additional problem of 'making the render look pretty', but as it is, people are doing concurrency in other 'wrong' ways, like 'WI 71;WI 99' or 'WI 71/99' which usually don't fit in the shield anyway. We'd be trading one poor render for another, but encouraging the correct OSM editing behavior. |
No, recommended is to have both, which isn't ideal, but that's what's recommended. |
I suspect that if this were implemented in openstreetmap-carto, the OSM recommendation would be changed to reflect that. That is, to prefer route relations over tagging ways when a route spans "significantly more than one" way. I believe tagging both ways and relations is done because the most common renderers, (but most especially openstreetmap-carto) doesn't render the refs that appear on the route relations at all. If it did, the OSM project could revisit that best practice. Until 'ref' on relations is supported in rendering, telling the userbase 'Please put refs on route relations only' would appear to them to be "removing data from the rendered map" at http://www.openstreetmap.org/ . |
|
Copied my comment from #508 (comment) as per @skylerbunny's request: FWIW, we spent a ton of time augmenting route relations (Interstates and US Highways mainly) in the U.S. with appropriate network and ref tags, so that we wouldn't have to rely on the brittle ref-on-ways tagging convention for determining the road designation. This is by no means finished - many state level networks remain to be done - but we have some tools in place in the U.S. to help guide this work, like the relation pages http://184.73.220.107/relationpages/ |
|
I would be hugely in favor of shield tagging based on route network / ref being rendered on osm.org - this would encourage what I consider proper tagging of network information and stop folks from duplicating this information on the ways, which is brittle and hard to maintain. |
|
Insight from the Telenav perspective as a large scale user of OSM data: we ingest both refs on ways and network / ref on route relations to determine road network designations, but we'd much rather just use the relations for getting that data, and we're seeing lots of inconsistencies in the ref information on the ways, both between way and parent relation info and within the ways. |
|
Given the current scarcity of route relations, we'll need to continue rendering ref tags.
What I'm wondering is, how, while avoiding duplication. The rendering tables don't have relation membership information, so I think we have to use postgis. Thoughts? |
|
Well, the deeper technical implementation aside, assuming the existence of a tag can be compared between a way and a relation, I'd suggest this:
For long term maintainability of routes, we need to actively discourage 'ref on ways'. Doing this supports it merely for backwards compatibility but puts thrust behind using relations. |
This is a fairly big assumption without membership information.
This would fail in the region I showed. It would also fail on the other side of the tunnel. The only route relations for roads I saw were for the E road network. |
|
On Thu, Jun 26, 2014 at 6:08 PM, Paul Norman notifications@github.com
This is however how we chose to do it for Scout. Take the relation network Martijn van Exel |
|
Could use some European feedback, as I didn't see any route relations for country motorway systems in the area I looked, just cross-Europe networks. Is that the general way it's tagged, or are there variations by country within Europe? |
|
It seems like many European mappers have been hesitant to have route refs duplicated on relations and their member ways. To be sure, route relations are being used, but the I think that’s because most countries in Europe distinguish route networks by alphabetic prefixes. There are secondary differences in signage; however, if an “M” is enough to get the point across, I would totally understand if a road map omits the symbol in Meanwhile, in the U.S, we’ve tried for years to shoehorn route network names into the eight character allotment, but it’s hopeless when there’s a concurrency. To give but one example, Interstate 71 and Interstate 75 (highways of roughly equal importance to the region) share a major stretch of road in Kentucky. Alas, |
|
Highway refs are tagged on ways in Great Britain. No way can have more than one ref (apart from the unsigned and generally unused E routes) - i.e. no "multiplexing" or "concurrency" - and each signposted ref is, in theory, unique within GB. Route relations would therefore be needless duplication, and significant extra work for the mapper. If openstreetmap-carto gets support for road route relations for the US guys then that's great, but there is no intention (and nor should there be) to stop tagging refs on ways in GB and similar countries. |
I think you're mixing up issues - this issue is about using relations instead of ref on ways, not symbols.
There's no reason we can't handle wider refs that I see. |
|
In Germany, there are routes for motorways and some primaries ("Bundesstraßen"). I haven't seen many relations for secondary and tertiary highways ("Kreis-"/"Landesstraßen"). Note that some road relations are only parent relations for the TMC-Relations (Example: http://www.openstreetmap.org/relation/2166961). Usually there's only one ref per road, except of some rare cases on motorways (Example on the A 8/A 81 here: http://www.openstreetmap.org/#map=14/48.7536/9.0367) |
This here is the big problem I'm seeing with implementing this. Taking a section of the M1 as an example, we have One option is to slice up the way refs on This would also reveal a particular class of mistake, where refs disagree between route relations and their ways. As an example, the Rodovia Washington Luiz has BR040 on some ways and BR-040 on the relation. Does anyone have any other suggestions that will work in the M1 case I gave? |
|
Perhaps the test could be whether the relation’s entire If the map adopts a deduplication algorithm, entry-level editors like iD would need to present a consolidated ref field using the same algorithm. Otherwise, an inexperienced mapper would be left wondering about phantom refs. |
The documentation you linked doesn't have any examples of road route relations. The examples I found in Germany, France, England, Brazil, Spain, Russia, Italy and Mexico all have |
|
2014-06-27 2:08 GMT+02:00 Paul Norman notifications@github.com:
In my area there are lots of route-relations so I thought they were common, not sure how many highway ways are in the average route=road relation, the Regardless of those numbers I agree with Martijn that rendering refs for |
|
@pnorman The obvious solution for the M1/E13 issue is that the E13 relation in the UK shouldn't be tagged with ref=, but instead admin_ref= or some similar tag used for unsigned, administrative-only numbers. |
|
2014-06-27 12:22 GMT+02:00 Paul Norman notifications@github.com:
In Germany there are not many overlapping highway-road-routes, but only as |
unsigned_ref. My concern is, are there roads with ref tags where they are in road route relations with a different ref and the ref on the way would not normally have a route relation created. |
|
2014-06-27 12:52 GMT+02:00 Richard Fairhurst notifications@github.com:
@richard: the common ref tag for "E" in continental Europe is "int_ref", |
|
I looked at what we've got in the database for rendering route relations
|
|
Yes, but if you want to arrange multiple shields in a certain way it is still fairly awkward to do so based on the current individual ways plus separate route geometries structure generated by osm2pgsql. Hence my general remark regarding the best approach. |
|
Please note that I did my osm-shields for my own use - the proposal to adopt it for the main rendering chain is not mine! I shared it primarily because I could not find a worked example of how to use the GroupSymbolizer to render marker clusters for concurrent routes in multiple networks, and encountered enough difficulty trying that a project to share the results seemed to be in order. With that said, don't be quick to discount the possibility that this approach could be reworked in such a way as to scale to the main map. minutely updates The current implementation works entirely without either parsing the OSM changeset or modifying osm2pgsql. It's actually pretty well suited to the 'daily update from GeoFabrik' workflow. Minutely updates are a different workflow. 'Incompatible with minutely updates,' as @imagico puts it, is far too strong a statement; 'minutely updates need a little more coding, ideally with some help from osm2pgsql' is more accurate. To me, it's quite obvious how to handle them, and I give three possible integration strategies on the project wiki. The integration alteratives depend more on social issues than technical ones; if osm2pgsql cannot be modified to maintain the auxiliary tables, then fall back on reparsing the changes delivered by osmosis, or use database triggers to keep them in sync. I haven't troubled to develop the minutely workflow, because I want to reassure myself that it won't be a complete waste of time. Given that I'm not all that well connected in the community, I strongly suspect that I may not be able to negotiate the social and political hurdles of developing to the 'big' rendering systems, or that the approach will fail to scale in some way that I have not anticipated. I haven't the resources personally to host a full-planet, minutely-updating system, so testing would be a problem. The technical problem is easy compared with getting access to the facilities needed to test a solution, and getting such a solution integrated into the standard tools. The latter issues are what worry me. With regards to the other remarks from @pnorman, relayed by @1ec5: slim tables @imagico statement that the implementation 'relies on slim tables' is also rather too strong a statement. It relies on the slim tables to perform its updates - just as osm2pgsql does. The slim tables are not used at all at render time. What is used at render time is three new tables. One has a row for each route=road relation, and maintains the 'network' and 'ref' tags. A second has a row for each way that is a member of one or more route=road relations, and holds simply the relation ID, the way ID, and the role. The third is a directory that maps network/ref pairs to the file names for the corresponding graphics. Having something in the database to identify where to find the graphic elements is essential. Alas, US road networks are anything but a 'small fixed set' and so need a database to track them. The other two tables could, I suppose, be replaced with a modification to osm2pgsql that would copy the network and ref information down to the individual items in the line table. This would, I suspect, prove less performant (because of replicated data) than the JOIN operation needed to go from 'I am rendering this set of ways' to 'this is the set of route markers that I need to place'. Given that a schema change to support the graphic elements would be required anyway, I decided to go with the JOIN. Having the tables available for joining allows for queries to go in both directions: 'what is the set of ways that I will encounter if I follow County Road 34?' and 'what is the set of route markers that I may expect on way 2718281828?' In order to have the smallest possible footprint, I confine myself to relations with 'type=route', 'route=road', and 'network' and 'ref' both present. Obviously, approach to splitting linestrings The osm2pgsql approach of reassembling route relations back into long linestrings is fundamentally not going to work with route concurrencies, and in fact I totally ignore it and work with the original ways plus the auxiliary tables. The problem with the approach is that the linestrings need to represent distinct sets of relations, not individual relations. In the diagram below, the red and blue lines represent distinct routes. In the section where the routes are concurrent, there's really no practicable way for a renderer to tell that if all it has is the separate red and blue lines. It really needs five linestrings (or three multilinestrings) in this case: the bits that belong only to the red route (two linestrings or one multilinestring), the bits that belong only to the blue route (again, two linestrings or one multilinestring), and the segment that is shared by both (one linestring).
The implementation as coded defers splitting and merging linestrings to render time, when there will always be a set of manageable size (because it consists only of the ways that will be rendered on one supertile). It uses the powerful PostGIS 'ST_LineMerge' operation to group ways that belong to the same sets of routes together and coalesce them where possible. About the only thing that it runs afoul of is that Mapnik winds up trying to place a marker on each carriageway of a motorway, even at low zoom levels, leading to a proliferation of markers. This appears to be a deviation from what the Mapnik documentation promises, and I've entered a query at mapnik/mapnik/issues/3978 about how best to solve the problem. handling both the tagging practice of putting the ref on the way and putting it in the relation Since I was focusing on a North American map, and the practice of placing route references on relations has become near-universal here, I simply didn't bother coding up the alternative. As long as you don't need to have both styles of tagging supported on the same way, it's easy to implement. Existing tagging practice in the US will mean that the ways with the tag on the way will not get shaped markers, but rather use something like today's OSM style. It's too long a leap to expect the renderer to take a ref that looks like 'CR 34', figure out from administrative boundaries what county the 'CR' refers to, and look up the appropriate marker. performance and readability The performance appears adequate. On the one largish (about a third of the United States) map that I render, the rendering remains I/O bound - it's dominated by the fact that on my machine, owing to fairly limited SSD space, rendering a supertile takes at least two seeks on magnetic disks, one to retrieve a hillshading image and another to retrieve raster-encoded land cover data. It is, of course, impossible to offer assurances regarding performance in the fielded environment without testing in a very similar setup to the production system. I do believe that what performance problems we will encounter are surmountable. As far as readability goes, given that the assembly of route concurrences is done in the database, we are dealing with some fairly complex SQL here. I can, of course, comment it much more extensively, but there is complexity there that simply cannot be eliminated while continuing to satisfy all the other constraints. From Mapnik's point of view, the complexity is reduced to a simple call on a stored procedure. Within the stored procedure, this wiki page provides an overview of what is happening. Of course, I plan to put that discussion in abbreviated form in the commentary in the stored procedures themselves. |
|
I think you misunderstood my statements - i did not mean to pass judgement on the suitability of your approach for what you are trying to accomplish (for that i have not looked even remotely close enough at your code). I just tried to answer the question by @1ec5 regarding the suitability of your code for this style based on the requirements previously stated in this issue. And i disagree that relies on slim tables and is incompatible with minutely updates is too strong, it is an accurate description of the current situation as far as i can see. If your code can be modified to allow minutely updates in a way that works reasonably well on a world wide database i don't know. |
|
@imagico: Is there an approach to minutely updates in which osm2pgsql does not rely on slim tables? Can you give me a link to a discussion of such a thing? I see on the OSM wiki:
There's very likely something of which I'm unaware here. Otherwise, it seems rather odd that osm2pgsql may depend on slim tables to perform incremental updates and apply planet diffs to 'roads', 'line', 'point' and 'polygon', but I may not depend on them for incremental update of the added tables that support route concurrences. That's why I was careful to clarify that I do not depend on the slim tables at render time, only to perform updates. |
|
Note i have made no connection between the requirement to work without slim tables (as stated by @pnorman) and the requirement to allow minutely updates (which implicitly stems from the use on the OSMF servers). The requirement to work without slim tables is based on two main motivations
Neither of these has anything to do with the question if the slim tables are used during rendering or not. |
|
I'm not going to comment further here. I think it's doable - but not done. I don't see any architectural issues blocking it. I'm sure that I won't have time to do it quickly - anyone who's sufficiciently familiar with the code for osm2pgsql or Mapnik, or that is familiar with tile expiry mechanisms, I'd like to pick your brains about how best to integrate changes. I've opened a project to track this. |
|
I suppose I should make a progress report. The code in osm-shields now handles ref=* tags on either a route relation or directly on a way. (The ones on ways are intentionally not processed if the way participates in a group relation.) See the corner of New Hampshire, Vermont and Massachusetts for a transition from "refs mostly on route relations rendered with pictorial shields" to "refs mostly on the way itself, rendered in classic OSM style" Concurrencies among route relations are handled with the GroupSymbolizer. Let this serve as a proof of concept that having both representations in the same map isn't a show-stopper. |
|
If it helps to do some spot checking, I did some fixes in the last couple hours to the state of Georgia, at Georgia seems to be pretty mature in terms of 'most routes at the state and US level have relations, and counties have a few too.' I wouldn't claim I found every error in the above, but I do think I got a lot of them. I found a couple dozen. |
In addition to the requirements I listed, there are the global ones for the stylesheet
To solve this in osm2pgsql, implement osm2pgsql-dev/osm2pgsql#230. This is a big job because it in turn requires changes to osm2pgsql's Lua API. It is possible to implement this meeting all of the criteria except SQL readability and performance. I did so in 2014 but the SQL was too messy for me. If I were to tackle it today I'd start by cleaning up the roads SQL. If that were cleaner, it'd be easier to work with, and new complexity would be more tolerable. I'm not sure there's a direct connection on the SQL side, but cutting back the roads MSS would certainly help. Then comes the tough question of how to implement. The requirements of working with osm2pgsql as we call it (no slim tables) and understandable SQL are hard requirements. I'd see if one of the requirements about features needed could be weakened. If none of them can, then I don't believe it's possible to solve, given the constraints. Edit: Fixed issue link |
|
Given the constraint that you express that 'the schema as implemented with a minimalistic osm2pgsql import today cannot be expanded in any way in the future', I think you are right that the problem cannot be solved. If you are indeed that adamant, I would also assume that any change to osm2pgsql to create auxiliary tables to track membership in route relations would also be summarily rejected. (I further assume that this would be the case even if enabled only on request by a command-line option.) The schema limitations are the one constraint that I find hobbles me. I can live with all the others. Well, maybe not with the lack of GroupSymbolizer, but I think I could duplicate its function in the pipeline that generates graphics, and then use prerendered groups in Mapnik. It would probably be easier to implement support for GroupSymbolizer in CartoCSS, but I'm presuming that an attempt at that would also be rejected out of hand. If I'm correct that you are unalterably opposed to the ideas, I'd rather find out sooner or later, since I don't want to find myself in the position of maintaining a fork. I'd rather spend the effort that I'd put into osm2pgsql on retooling to some other piece of software. If you're still reading this, I quite agree with you about the 'roads' table. I understand that compatibility will require retaining it, but I hardly use it. If I create indices using statements like: CREATE INDEX idx_line_primary ON planet_osm_line USING GIST(way) WHERE highway IN ('motorway', 'trunk', 'primary')I find that I hardly miss it. I deal with routes using the tables that track membership of the individual ways in route relations. By the way, those tables are tiny by OSM standards - for North America, there are about fifty thousand route=road comprising about a million and a half ways. If I were to replace the 'role' column with a simple flag indicating whether a way is or is not 'role=link', each row in the 1.5-million-row table would be 25 bytes (8 for osm_id of the relation, 8 for osm_id of the way, 4 for position of the way within the relation, 1 for flags). Joining through it is cheap, since PostgreSQL tends just to keep in in memory while rendering is going on. And it's surely not something that would add hours to an import. |
I think there's some miscommunication here. Paul was saying that anything like schema changes or extra tables needs to be upstream in osm2pgsql, not done here in this style. ( His link should have been to the relevant issue in the osm2pgsql repo, namely osm2pgsql-dev/osm2pgsql#230 ) What he then explained was that it is in fact possible to solve the relation symbolizers without changing how osm2pgsql works, but that it leads to very messy SQL. So it's again highlighting that solving that osm2pgsql issue is the way forward. Please don't assume that fixes provided to osm2psql will be rejected.
I can say as a CartoCSS maintainer that GroupSymbolizer support would be welcomed.
There are hundreds of downstream users of osm2pgsql that would appreciate better route relation handling! So please do consider discussing your ideas on that tracker, and I hope to see your efforts there. |
|
I will concede I got confused by the fact that @pnorman wears multiple hats - and misinterpreted him as speaking in his rôle as lead developer of osm2pgsql, not strictly as a developer of the OSM Carto style. |
|
Bolstered by the encouragement from @gravitystorm, I'm floating a formal proposal to develop the necessary osm2pgsql infrastructure to address the problem. Consider this to be a request, nay, rather, a plea for comments from those who understand the issues. The proposal is most emphatically not in its final form - in particular, there is an Open issues section that describes some requirements that I'm fairly sure exist, but that I don't understand. I hope that writing up this portion of the issue, as I understand it, demonstrates a good-faith attempt to "do the homework" and will not be dismissed out of hand as yet another ignorant oversimplification of the problem. I recognize that I'm still at an early stage in understanding, but I'm reluctant to move forward without at least some feedback from wiser heads. I'll make every effort to insure that your help will not be a waste of time. (Oh, quick note - it might be easier to consolidate replies in this ticket.) |
|
The issue osm2pgsql-dev/osm2pgsql#230 is now fixed by osm2pgsql PR1037, so it should be possible to implement this. Is anyone interested in submitting a PR to render the ref from the route relation in addition? |
Presumably it'd need more than a change to this map style - it'd also need the way that osm2pgsql is used on osm.org to be invoked differently? |
|
@jeisenbe I'm following the development of the 'flex' backend eagerly, but my time is severely limited, possibly for the entirety of 2021 (we'll see how things start sorting out later in the year. I'm considering the rendering of route relations on the main server to be Out Of Scope for me at the present time, owing to the fact that performance is unproven. Paul and Sarah are rightly concerned that reconstruction of concurrencies at render time might be an unacceptable cost. I am surely not tooled to assess the performance at production scale, so without a champion on the team, I'd rather be blocked. I will confess that I suspended further development of my own pictorial-shield rendering because the performance worries extended to an indication that the functionality, the way I use it, would never be workable using a stock 'osm2pgsql'. I had no desire to maintain a fork, and frankly lacked energy to assess a switch to ImpOSM or another database pipeline. Could @jeisenbe or another developer of the 'flex' backend revisit the schema I'm using and verify that it would be feasible to
The key structural differences are:
I'm at a point where I critically need a second pair of eyeballs on most of the route rendering stuff. Paul and Sarah have indicated that they consider my current design unacceptable, but have offered only sparse guidance as to what requirements it fails to address or how to move it forward. (I don't expect anything else. They're busy individuals, and I'm not paying them for the work.) I have the sense that something is badly wrong with the ideas, but I've no clue what it is! |
|
We should check if release 1.4.0 of osm2pgsql provides the proper functionality: https://github.com/openstreetmap/osm2pgsql/releases/tag/1.4.0 |
|
@jeisenbe - The I have it processing minutely update on my home server (which has data for only North America, but isn't an incredibly powerful machine) for several months now and the data import appears to be stable. I have three big issues to overcome. (1) I refuse to stand up Apache on this machine and mod_tile appears to be the only tile caching engine that isn't abandonware. (2) @pnorman has indicated that any table join in the rendering pipeline is unacceptable, and I haven't yet found a way to denormalize the data to satisfy this, (3) Carto-CSS (not OSM-Carto, the underlying system) doesn't do the GroupSymbolizer for rendering shields. These issues obviously cut across multiple big code bases and it's hard for me to get a handle on all of them at once. I'd say that |
Joins are okay. We already have them. Using osm2pgsql's internal tables isn't. |
|
@pnorman Glad to hear that. I'm not using the slim tables at all - that's all replaced with The update procedure has been running minutely on my box, in front of a PostGIS database that started from a Actually assigning route to way is moderately complex (hadling concurrencies without breaking marker placement is kind of difficult), and it's done in a set of stored procedures. They look to be hideous multiway JOIN operations, but in practice the common table expressions get optimizied and virtually all operations are in-memory. While the procedures are complicated, I think they're reasonably well documented. https://github.com/kennykb/osm-shields/blob/master/queryprocs.sql.in If you're now interested, the structure is that The pipeline for route relations also has a fallback case for OSM-style markers if there's no special graphic available for a given network. There's a whole insane mess of Tcl code (which I'm entirely willing to refactor) that selects shields for some four hundred networks. There are many more networks to be added, awaiting mostly the artwork. What's there are interstates, Trans-Canada Highway, Mexico Autopistas, state highways for all fifty states, provincial highways of all classes in all ten Canadian provinces and Nunavut, and many county, city and township markers. (Yes, it's been tested with West Virginia's peculiar double-numbered county roads, and renders them correclty. The route relations aren't available in very many places, but this map shows a reasonable sampling of state/county secondary/county tertiary hierarchy. If you scroll about from there, you'll start seeing OSM-style markers for the ones that lack relations.) I'm a bit stymied about the path forward from here. The next steps have to be standing up a tile cache on my server so that I can move away from static rendering, and working on Carto-CSS to make it support Moreover, I freely concede that I'm not a terrific cartographer. About the best that I can say for the map that I render is that it works for what I use it for. (PS: Another project that I want to do is to add markers for |
|
#4431 will unblock this |
The following issue has been moved over from trac. It needs more discussion as it is not clear how this should be implemented.
Highway shields are only rendered when the ways themselves have a ref tag. Now, the different ways that form a road route are assembled into relations. For the sake of avoiding duplicate data the ref tags should be removed from the individual ways and applied to the relation only.
However, currently we only use ref tags on roads for rendering, and we ignore the ref tag on the relation.
Any idea how we could implement this?
The text was updated successfully, but these errors were encountered: