v0.2.48..v0.2.49 changeset changeset derive replacement.asciidoc
Garret Voltz edited this page Oct 2, 2019
·
1 revision
diff --git a/docs/commands/changeset-derive-replacement.asciidoc b/docs/commands/changeset-derive-replacement.asciidoc
index 15e07ee..23296a0 100644
--- a/docs/commands/changeset-derive-replacement.asciidoc
+++ b/docs/commands/changeset-derive-replacement.asciidoc
@@ -3,120 +3,209 @@
=== Description
-The +changeset-derive-replacement+ command creates an OSM changeset file that represents the difference between two input
-OSM datasets within a given bounds where the data from the secondary input dataset completely replaces data in the
-reference dataset.
-
-The changeset derivation algorithm avoids unnecessary clipping of features (e.g. buildings crossing the AOI) and
-stitches up areas where features must be clipped (e.g. roads crossing the AOI). The output changeset file can be
-applied directly to an OSM API database or the Rails Port with the +changeset-apply+ command. Generally, the reference
-data is sourced from an authoritative data store, like an OSM API database. Element IDs for the first input (reference
-data) are retained and treated as data resident in an authoritative data store. Element IDs for the second input
-(secondary dataset) are not retained and treated as completely new data.
-
-* +input1+ - OSM data reference input (e.g. .osm file); must support bounded queries
-* +input2+ - OSM data secondary input (e.g. .osm file); must support bounded queries
-* +bounds+ - Comma delimited bounds: minx,miny,maxx,maxy; e.g. "38,-105,39,-104"
-* +featureFilter+ - A single criterion class name used to determine which type of features are replaced in the
- reference dataset. Must be a conflatable criterion (see "Type Filtering" section below). e.g. "hoot::BuildingCriterion"
-* +output+ - Output location (e.g. .osc or .osc.sql file)
-* +osmApiDatabaseUrl+ - Target OSM API database the changeset is to be applied to. Used to maintain element ID continuity with a particular
- database instance when generating SQL changesets only (.osc.sql).
-* +--lenient-bounds+ - By default, features will only replaced if they lie completely within the specified bounds. Activating this option
- makes the bounds handling more lenient (has no effect on point datasets). See the "Bounds Handling" section below
- for more detail.
-* +--stats+ - Stats flag, when turned on a table of create, modify, delete for each of the types node, way, relation (only valid
- for .osc output files)
-* +--write-bounds+ - If the `convert.bounding.box` configuration option is specified, optionally outputs a file containing the input
- bounds. The location of the file is controlled via the `bounds.output.file` configuration option.
+The +changeset-derive-replacement+ command creates an OSM changeset file that represents the difference between two OSM datasets within a
+specified bounds, where the data from the secondary input dataset replaces data in the reference dataset.
+
+The command can be configured to replace all reference data within the specified bounds or only reference data in areas where the reference
+and secondary data overlap. Additional filtering may be applied to the secondary input to control which types of features from it are added to
+the output. Also, additional filtering may be applied to the reference input to prevent certain types of features from being replaced.
+
+The changeset derivation algorithm avoids unnecessary clipping of features (e.g. buildings crossing a specified lenient bounds) and stitches up
+areas where features must be clipped (e.g. roads crossing a specified strict bounds). The output changeset file can be applied directly to an
+OSM API database or the Rails Port API with the +changeset-apply+ command. Generally, the reference data is sourced from an authoritative
+data store, like an OSM API database, and the secondary source can be any type of data. Element IDs for the reference dataset are retained,
+while element IDs for the secondary data are not retained and the elements treated as completely new data.
+
+* +input1+ - Reference input with features to replace; may be any supported input format that supports bounded reading
+ (e.g. OSM file). See the "Bounds Handling" section for more detail.
+* +input2+ - Secondary input with the features used for replacement; may be any supported input format that supports
+ bounded reading (e.g. OSM file). See the "Bounds Handling" section for more detail.
+* +bounds+ - Comma delimited bounds: minx,miny,maxx,maxy; e.g. "-71.4698,42.4866,-71.4657,42.4902"
+* +output+ - Output location; must be an .osc or .osc.sql file.
+* +osmApiDatabaseUrl+ - Target OSM API database the changeset is to be applied to. Used to maintain element ID continuity with a
+ particular database instance when generating SQL changesets only (.osc.sql).
+* +--full-replacement+ - Determines how much of the reference data is replaced. The default behavior replaces reference feature with
+ overlap with secondary features only. The option is incompatible with the +--retainment-filters+ option and
+ will result in an error when combined with it. See the "Bounds Handling" section for more detail.
+* +--strict-bounds+ - Determines how strict the interpretation of the specified bounds is during feature replacement. The default
+ behavior involves a lenient bounds interpretation. See the "Bounds Handling" section for more detail.
+* +--geometry-filters+ - Optional criteria used to determine the geometry type of the features that are replaced in the reference
+ dataset. The default behavior is to replace features of any geometry type. See the "Geometry Filtering"
+ section for more detail.
+* +--replacement-filters+ - Optional filtering criteria beyond what is specified in +--geometry-filters+ for determining which secondary
+ dataset features are used as replacement features. See the "Replacement Filtering" section for more detail.
+* +--chain-replacement-filters+ - Determines how the criteria specified in --replacement-filters are applied. See the "Replacement Filtering"
+ section for more detail.
+* +--replacement-filter-options+ - Optional configuration options for the filters specified in +--replacement-filters+. See the
+ "Replacement Filtering" section for more detail.
+* +--retainment-filters+ - Optional filtering criteria beyond what is specified in +--geometry-filters+ for determining which reference
+ dataset features are to be prevented from being replaced. The option is incompatible with the
+ +--full-replacement+ option and will result in an error when combined with it. See the
+ "Retainment Filtering" section for more detail.
+* +--chain-retainment-filters+ - Determines how the criteria specified in --retainment-filters are applied. See the "Retainment Filtering"
+ section for more detail.
+* +--retainment-filter-options+ - Optional configuration options for the filters specified in +--retainment-filters+. See the
+ "Retainment Filtering" section for more detail.
+* +--stats+ - Stats flag, when turned on a table of create, modify, delete for each of the types node, way, relation (only
+ valid for .osc output files)
+* +--write-bounds+ - If the `convert.bounding.box` configuration option is specified, optionally outputs a file containing the
+ input bounds. The location of the file is controlled via the `bounds.output.file` configuration option.
=== Usage
--------------------------------------
-changeset-derive-replacement (input1) (input2) (bounds) (featureFilter) (output) [osmApiDatabaseUrl] [--stats] [--write-bounds]
+changeset-derive-replacement (input1) (input2) (bounds) (output) [osmApiDatabaseUrl] [--full-replacement] [--strict-bounds] [--geometry-filters] [--replacement-filters] [--chain-replacement-filters] [--replacement-filter-options] [--stats] [--write-bounds]
--------------------------------------
=== Examples
---------------------------------------
-hoot changeset-derive-replacement input1.osm input2.osm "38,-105,39,-104" hoot::BuildingCriterion outputChangeset.osc
-
-hoot changeset-derive-replacement -D convert.ops="op1;op2" input1.osm input2.osm "38,-105,39,-104" hoot::BuildingCriterion outputChangeset.osc --lenient-bounds
-
-hoot changeset-derive-replacement input1.osm input2.osm "38,-105,39,-104" hoot::BuildingCriterion outputChangeset.osc --stats
-
-hoot changeset-derive-replacement input1.osm input2.osm "38,-105,39,-104" hoot::BuildingCriterion outputChangeset.osc.sql osmapidb://username:password@localhost:5432/osmApiDatabaseName
---------------------------------------
-
-=== Type Filtering
-
-Currently, only one criterion may be specified per feature filter and at least one must always be specified. Future work will allow for
-specifying no filter or multiple criteria per filter (https://github.com/ngageoint/hootenanny/issues/3360). To see a list of valid
-conflatable element criteria for use as a feature filter:
------
-hoot info --conflatable-criteria
------
+* https://github.com/ngageoint/hootenanny/blob/master/docs/user/CommandLineExamples.asciidoc#applying-changes[Basic]
+* https://github.com/ngageoint/hootenanny/blob/master/docs/user/CommandLineExamples.asciidoc#applying-changes-1[Advanced]
-=== Bounds Handling
+=== Input Data
All inputs must support bounded reading. To list the Hootenanny input formats that support bounded reading:
-----
hoot info --formats --input-bounded
-----
-Unless the reference data is being read from a direct connection to an OSM API database (osmapidb://), reference input
-datasets containing linear data should be slightly larger than the replacement AOI, so as not to drop linear features in the changeset output
+Unless the reference data is being read from a direct connection to an OSM API database (osmapidb://), reference input datasets containing
+linear data should be slightly larger than the replacement AOI, so as not to drop linear features in the changeset output
(osmapidb:// reference inputs handle this issue automatically).
-==== Strict Bounds Handling
+GeoJSON output from the Overpass API is not supported by this command, since it does not contain way nodes.
+
+=== Full vs Overlapping Only Replacement
+
+If the +--full-replacement+ command option is absent, reference features inside the specified bounds will only be replaced if the area they
+cover intersects with the area covered by secondary features. The covered area is determined by an Alpha Shape. If the +--full-replacement+
+option is present, all reference features within the specified bounds will be removed.
+
++--full-replacement+ is incompatiable with the +--retainment-filters+ option, as +--full-replacement+ being specified will always cause all
+features in the reference layer to be replaced in the changeset output, thus overriding the filters in +--retainment-filters+.
+
+Full replacement works well with points and polygons (e.g. POIs and buildings) when you absolutely do not want any of the original reference
+data left in an AOI. If there are large gaps across the AOI between the reference and secondary data, you may want to do an overlapping only
+replacement to enable continuity of the data across the AOI.
-The following describes how the AOI is interpreted by the command when the +--lenient+ bounds option is absent.
+Full replacement may not work well with linear data in certain circumstances with the strict bounds interpretation (e.g. roads), as it may end
+up leaving a large enough gap between the reference and secondary data that the data cannot easily be joined back together in the output.
-===== Points
+=== Strict Bounds Handling
-- Only reference features completely inside the bounds will be replaced.
-- No secondary feature on or outside the boundary will be included in the output.
+The following describes how the specified bounds is interpreted by the command when the +--strict-bounds+ option is present.
-===== Lines
+==== Points
-* No reference feature crossing the bounds will be modified outside of the bounds. Reference features may be split.
-* No parts of secondary features crossing the bounds will be included in the output.
+* Only reference features completely inside the specified bounds and overlapping with the combined shape of the secondary features will be
+ replaced.
+* No secondary feature lying on or outside the specified bounds will be included in the output.
-===== Polygons
+==== Lines
-* No reference features crossing the bounds will be modified.
-* No secondary features crossing the bounds will be included in the output.
+* Only reference features overlapping with the combined shape of the secondary features will be replaced.
+* No reference feature crossing the specified bounds will be modified outside of the bounds. Reference features may be split at the specified
+ bounds.
+* No parts of secondary features crossing the specified bounds will be included in the output.
-==== Lenient Bounds Handling
+==== Polygons
-The following describes how the AOI is interpreted by the command when the +--lenient+ bounds option is present.
+* Only reference features overlapping with the combined shape of the secondary features will be replaced.
+* No reference features crossing the specified bounds will be modified.
+* No secondary features crossing the specified bounds will be included in the output.
-===== Points
+=== Lenient Bounds Handling
+
+The following describes how the specified bounds is interpreted by the command when the +--strict-bounds+ bounds option is absent.
+
+==== Points
N/A - Point bounds relationships are only handled in a strict fashion.
-===== Lines
+==== Lines
+
+* Only reference features overlapping with the combined shape of the secondary features will be replaced.
+* Reference features crossing the specified bounds will be completely replaced by secondary features.
+
+==== Polygons
-* Reference features crossing the bounds will be completely replaced by secondary features.
+* Only reference features overlapping with the combined shape of the secondary features will be replaced.
+* Reference features crossing the specified bounds may be modified. They will not be split, and will only be conflated with secondary features.
+* Secondary features crossing the specified bounds may be included unmodified in the output or conflated with reference features.
-===== Polygons
+=== Filtering
-* Reference features crossing the bounds may be modified. They will not be split, and will only be conflated with secondary features.
-* Secondary features crossing the bounds may be included unmodified in the output or conflated with reference features.
+==== Geometry Filtering
+
+The command option, +--geometry-filters+, controls feature filtering by geometry type. One or more element criterion class names associated with
+a geometry type can be used to determine the geometry type of the features that are replaced in the reference dataset . The criteria specified
+must be geometry type criteria (e.g. "hoot::BuildingCriterion" or "hoot::PointCriteron").
+
+To see a list of valid geometry type criteria for use as a feature filter:
+-----
+hoot info --geometry-type-criteria
+-----
+
+A feature may pass the geometry filter by satisfying any one of the individual specified filters. From the command line, combine multiple
+criteria with a semicolon and surround the entire value string with quotes. If no filter is specified, features of all geometry types within
+the bounds will be replaced. Geometry filters are handled separately from the filters specified in +--replacement-filters+ since Hootenanny
+executes a different replacement changeset generation workflow dependent upon the geometry type of the feature being replaced.
+
+==== Replacement Filtering
+
+The command option, +--replacement-filters+, allows for further restricting the features from the secondary dataset added to the output beyond
+geometry type. One or more criterion class names can be added to +--replacement-filters+ to further filter features that are used for
+replacement from the secondary dataset. The criteria specified in +--replacement-filters+ may not be geometry type element criteria. When
+populating the option value from the command line, combine multiple criteria with a semicolon and surround the entire value string with quotes.
+To see a list of available filtering criteria:
+-----
+hoot info --filters
+-----
+
+The behavior of +--replacement-filters+ is further configurable by the +--chain-replacement-filters+ option. If the
++--chain-retainment-filters+ is used, then a feature must pass all criteria specified in +--replacement-filters+ in order to be included in the changeset output.
+
+Configuration options may be passed separately to the criteria in +--replacement-filters+ via the +--replacement-filter-options+ parameter. The
+option takes the form "<option name 1>=<option value 1>;<option name 2>=<option value 2>...". Do not prepend these options with "-D" as is
+normally done with configuration options passed in from the command line. Any identically named configuration options passed into the command
+prepended by "-D" may override these filtering configuration options.
+
+==== Retainment Filtering
+
+The command option, +--retainment-filters+, allows for further restricting the features from the reference dataset that are replaced in the
+output beyond geometry type. One or more criterion class names can be added to +--retainment-filters+ to further restrict the features that are
+replaced in the reference dataset. The criteria specified in +--retainment-filters+ may not be geometry type element criteria. When populating
+the option value from the command line, combine multiple criteria with a semicolon and surround the entire value string with quotes. To see a
+list of available filtering criteria:
+-----
+hoot info --filters
+-----
+
+The behavior of +--retainment-filters+ is further configurable by the +--chain-retainment-filters+ option. If the +--chain-retainment-filters+
+option is used, then a feature must pass all criteria specified in +--retainment-filters+ in order to prevent it from potentially being
+replaced in the changeset output.
+
+Configuration options may be passed separately to the criteria in +--retainment-filters+ via the +--retainment-filter-options+ parameter. The
+option takes the same form as the +--replacement-filter-options+ option described in the previous section.
+
++--retainment-filters+ is incompatiable with the +--full-replacement+ option, as +--full-replacement+ being specified will always cause all
+features in the reference layer to be replaced in the changeset output, thus overriding the filters in +--retainment-filters+.
=== Versioning
-If the target of the resulting changeset is an OSM API database, all input features from the reference dataset must
-be populated with the correct changeset versions or application of the resulting changeset will fail.
+If the target of the resulting changeset is an OSM API database, all input features from the reference dataset must be populated with the
+correct changeset versions or application of the resulting changeset will fail.
For Overpass API queries, add "out meta" to the query retrieving the reference data.
-=== Unsupported Formats
+=== Notes
-GeoJSON output from the Overpass API is not supported by this command, since it does not contain way nodes.
+As part of "this issue":https://github.com/ngageoint/hootenanny/issues/3429 , we'll look into providing the option to replace all reference
+data within the specified bounds rather than just the reference data that overlaps secondary data.
=== See Also
* `changeset-derive` command
* `changeset.*` configuration options
* `snap.unconnected.ways.*` configuration options
+* "Supported Input Formats":https://github.com/ngageoint/hootenanny/blob/master/docs/user/SupportedDataFormats.asciidoc