Experimental feature to merge multiple OSM layers together #380
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This includes tests that pass for various combinations of simple point layers, but should also work for any layers that have property only geometry encoders. There are failing tests for merging into OSM, which should not be supported, and for merging one OSM into another, which we hope to support in the next update. In fact, this commit is preparatory work for OSM-OSM merging capabilities, which are planned, but not implemented yet.
This required some refactoring of the OSM model with changes to OSMImporter, OSMLayer and OSMDataset. In addition we added OSMModel for collecting general contants and functions appropriate to OSM, and we added OSMMerger with the intelligence to merge two OSM datasets together. This is currently only tested in a trivial case, which is two identical models with one trivial change (added a free floating OSM node). One change to the OSM model is that we moved User nodes to a linked list like we do for Ways and Relations.
This involved a big change to the OSM model. The OSM model uses less linked lists and more label-property indexes, in particular User, Changset, Way and Node are all now accessible via hashed labels and their OSM ids, using schema indexes. And all have been removed from their layer linked lists, except the OSMNode which remains in the OSMWay->OSMWayNode->OSMNode tree/chain. The OSMImporters LabelHasher was moved to the OSMDataset to be shared with the OSMGeometryEncoder, bringing the imported model and the CRUD model closer. The OSMGeometryEncoder now can generate new fake OSM ids for all supported types (relation, way, node) saving the max value in the layer node so restarts will not re-use ids. The fake ids are all negative. This allows, in particular, an imported OSM dataset to have new data added through the normal layer interface without clashing OSM ids (real are positive, and fake are negative).
We made the code more verbose for now, but we could simplify to simply not care about missing geometries.
# Conflicts: # src/main/java/org/neo4j/gis/spatial/AbstractGeometryEncoder.java # src/main/java/org/neo4j/gis/spatial/DefaultLayer.java # src/main/java/org/neo4j/gis/spatial/GeometryEncoder.java # src/main/java/org/neo4j/gis/spatial/SpatialTopologyUtils.java # src/main/java/org/neo4j/gis/spatial/osm/OSMDataset.java # src/main/java/org/neo4j/gis/spatial/osm/OSMGeometryEncoder.java # src/main/java/org/neo4j/gis/spatial/osm/OSMImporter.java # src/main/java/org/neo4j/gis/spatial/osm/OSMLayer.java # src/main/java/org/neo4j/gis/spatial/osm/OSMRelation.java # src/main/java/org/neo4j/gis/spatial/osm/RoadDirection.java # src/main/java/org/neo4j/gis/spatial/procedures/SpatialProcedures.java # src/main/java/org/neo4j/gis/spatial/rtree/RTreeIndex.java # src/test/java/org/neo4j/gis/spatial/LayersTest.java # src/test/java/org/neo4j/gis/spatial/OsmAnalysisTest.java # src/test/java/org/neo4j/gis/spatial/TestOSMImport.java # src/test/java/org/neo4j/gis/spatial/TestSpatial.java # src/test/java/org/neo4j/gis/spatial/pipes/GeoPipesDocTest.java # src/test/java/org/neo4j/gis/spatial/procedures/SpatialProceduresTest.java
# Conflicts: # src/test/java/org/neo4j/gis/spatial/procedures/SpatialProceduresTest.java
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The idea here is that if we have imported multiple overlapping chunks of OSM data, it should be possible to stitch them together. The process does as follows:
spatial.merge.into(main,other)OSMNodenodes inother- for each check if there is a matching node inmainand update to other properties (if the node moves, record that fact for re-indexing). If there is no matching node, re-label the node to the new layer (moving it in internal indexes). All nodes with geometry nodes attached are recorded for later re-indexingOSMWaynodes inother- for each check if there is a matching node inmainand if not move the layer by re-labeling. If there is a matching way, then do a node level merge by finding common nodes and re-sorting.OSMRelationtoomainnodes from the RTree index and run re-indexing