Switch to Lua transforms and change database schema

[pnorman]

Not yet ready for merge

What this does

This PR accomplishes four main tasks

Lua transforms

It switches to Lua transforms, which allows pre-processing of tags like layer. This allows SQL like

ORDER BY COALESCE(layer,0), way_area DESC

Instead of the longer

ORDER BY CASE WHEN layer~E'^-?\\d+$' AND length(layer)<10 THEN layer::integer ELSE 0 END, way_area DESC

There are also more significant changes for z_order, but that can be done post-merge

Disjoint areas

It uses -G, so disjoint areas are imported as PostGIS MULTIPOLYGONs instead of multiple rows as POLYGONs. These are often found on administrative boundary relations where islands are disjoint.

Tag columns

It uses --hstore, so tags that don't have a column end up in a special tags column. This allows for more flexability, and the number of columns has been significantly reduced

Multipolygon handling

The multipolygon handling is in the transforms, so instead of the C transforms, we are responsible for defining the handling.

For the purposes of this PR and the transforms, a multipolygon is either new-style or old-style. A new-style multipolygon has a tag other than type=multipolygon on the relation, and tags on its members are ignored. An old-style multipolygon is one with only a type=multipolygon and all the member ways have the same tags, or a member may be untagged. For technical reasons, deleted tags (e.g. source=*) are ignored.

There is also implicitly the definition of an invalid multipolygon, which is a type=multipolygon relation which doesn't meet the above criteria.

Some properties of MPs as defined like this are

• Any new-style MP will never change to an old-style MP by only changing tags on its member ways, nor will it become invalid. Sudden changes to interpretation were possible with the C transform logic, where adding building=yes to a small object would set the building column to yes for a large object generated by a relation.

• Any old-style MP will never change to an new-style MP by only changing tags on its member ways, but may become invalid if the tags become conflicting

• type=multipolygon relations are always areas

• Processing mulitpolygons does not require any knowledge of tagging semantics like is required for area formation from polygons. For technical reasons, deleted keys are ignored, which does involve tagging semantics.

Area handling

We can now determine if a way corresponds to an area feature or a linear feature however we want, instead of just based on tag keys. We want to avoid the interpretation of a correctly mapped feature depending on if a way is closed or not.

Open issues

• Add comments to openstreetmap-carto.style pointing to INSTALL.md

• Add something on using hstore to the code style guide (tags->'foo' instead of tags -> 'foo', etc) (Add SQL style notes on hstore #2552)

• The interim need to backport PRs against both master and 3.x needs documenting in CONTRIBUTING.md

• Define more precisely how long we will maintain 3.x and 4.x compatibility

• Better document MPs somewhere

Pre-merge changes

The changelog and README.md version information need changing

A new git branch for 3.x.x work needs creating

Post-merge changes

Mark database layout milestone as finished

Fixes #101, #1362, #1504

Preview

I have a demo at https://lua.osm-carto.paulnorman.ca/. It is not consuming OSM updates. Before reporting a bug please

• See if it's covered already in the discussion
• See if it's an expected change
• Check the OSM tagging so you can report what it is, if relevant
• Check if the OSM data has recently changed

[nebulon42]

Regarding old-style MPs I would be much in favour of that. I'm sure that the cleanup will progress to a point where this is no problem anymore.

[matthijsmelissen]

That's great news. I would also be strongly in favour of dropping support for old-style multipolygons.

[pnorman]

This is now ready. Before merging we want to

• clean out the PR backlog,
• do another full planet demo,
• release 3.3.0, and
• impose a feature freeze for cartographic changes after 3.3.0 until 4.0.0 is released

I'd also like to move religion to hstore, but if that doesn't get done it won't hold up the release.

[pnorman]

I squeezed some more disk space out of one of my servers, so it looks like I can do a test over the weekend, then we can proceed.

[pnorman]

https://lua.osm-carto.paulnorman.ca is running with the latest code, I'm looking to tag 3.3.0 and freeze in a couple days.

[imagico]

Nice. Comparison is somewhat difficult since the style version is different in the standard map at the moment and the low zoom tiles are also very old there right now.

Am i right to assume that this does not yet use libosmium for geometry building (osm2pgsql-dev/osm2pgsql#684)?

[matthijsmelissen]

Looks good!

I had no time to analyse this yet, but here is a difference:
https://lua.osm-carto.paulnorman.ca/#18/49.58117/6.12971

[pnorman]

Am i right to assume that this does not yet use libosmium for geometry building

No, it's using released osm2pgsql.

[pnorman]

I had no time to analyse this yet, but here is a difference:
https://lua.osm-carto.paulnorman.ca/#18/49.58117/6.12971

Object in question is http://www.openstreetmap.org/way/77686091 with leisure=track, so this is an expected difference, and, as it happens, a tagging error.

[matthijsmelissen]

Object in question is http://www.openstreetmap.org/way/77686091 with leisure=track, so this is an expected difference, and, as it happens, a tagging error.

We might want to add a rendering rule though. We could render it equal to racetracks for now, so there is at least something.

[imagico]

Yes, we should render leisure=track on lines but no, not like racetracks. The latter is for motor sports, the former for running sports, very different things we also render very differently.

Ideally i think leisure=track on lines should be the same as on areas, i.e. line with thin casing. And preferably in a way that combines with areas, i.e. render casings below and fills above areas.

[imagico]

Ok, i inspected the map a bit more and it seems to me most of the differences that are not either due to the style version difference or due to data changes are

• the vanishing of independently labeled multipolygon components and the resulting duplicate labels which is mostly a very positive effect (see Recommend -G flag for multipolygons #59).
• the different interpretation of closed ways with leisure=track as @math1985 already spotted.
• the lack of transfer of member way tags to multipolygon relations due to the removal of support for old style MPs. Like for example here:

http://www.openstreetmap.org/way/100827142

This water area tag is now interpreted while previously it was ignored because it was redundant with the relation tag so the islands in the river vanish. I expected this effect to be much more widespread, especially for water areas since in the past it had been common practice to simply add waterway=riverbank to all outer ways of a a river polygon and if a water area tag was added to the relation it was not removed from the ways. But it is less frequent than i thought although still pretty visible in some areas, in particular India and China. Since this particular kind of faulty double tagging is not covered by OSMI and Osmose and this is kind of connected to the elimination of old style multipolygons because it only matters if you ignore old style tagging it might be useful to think about including these in the multipolygon cleanup effort - cc @joto.

[pnorman]

I guess the question is, do we want to hold up 3.3.0/4.0.0 to make any of these changes?

[imagico]

I don't think so. If anyone wants to add rendering of leisure=track as lines that would be most welcome but it does not really matter much if this happens before or after the release. It is not that anything that has been rendered correctly before is rendered incorrectly with this change and would be rendered exactly right if we add new support for this. We drop support for rendering closed ways with leisure=track and without area=yes as areas and we would in the future like to add support for rendering them as lines.

And the faulty double tagging can be fixed as easily after the change than before - in fact if the change is rolled out the motivation for fixing is probably higher since it would fix a visible error while right now it would be an invisible change.

[matthijsmelissen]

I guess the question is, do we want to hold up 3.3.0/4.0.0 to make any of these changes?

As far as I'm concerned, no.

[pnorman]

I'll look at tagging 3.3.0 tonight, which will also involve a feature freeze until 4.0.0.

[sommerluk]

A big “thanks” for making this working!