v0.2.47..v0.2.48 changeset OsmApiDb.asciidoc
Garret Voltz edited this page Sep 27, 2019
·
1 revision
diff --git a/docs/user/OsmApiDb.asciidoc b/docs/user/OsmApiDb.asciidoc
index c132414..965980a 100644
--- a/docs/user/OsmApiDb.asciidoc
+++ b/docs/user/OsmApiDb.asciidoc
@@ -1,9 +1,9 @@
-== Working with OpenStreetMap Editing API Database Data in Hootenanny
+== Working with OpenStreetMap Editing API Database Data in Hootenanny
-Hootenanny offers the capability to convert data between an OpenStreetMap Editing API Database and any supported Hootenanny data output format. The OpenStreetMap Editing API database allows fetching and saving of raw geo-data for OpenStreetMap. The current supported API version is 0.6. More information about the database and API is found at: https://wiki.openstreetmap.org/wiki/API.
+Hootenanny offers the capability to convert data between an OpenStreetMap Editing API Database and any supported Hootenanny data output format. The OpenStreetMap Editing API database allows fetching and saving of raw geo-data for OpenStreetMap. The current supported API version is 0.6. More information about the database and API is found at: https://wiki.openstreetmap.org/wiki/API.
-Hootenanny supports reading and writing from the OpenStreetMap API database using its `convert` command. A description of how to use Hootenanny's capability for 'writing to' and 'reading from' an OpenStreetMap Editing API Database is described in this section.
+Hootenanny supports reading and writing from the OpenStreetMap API database using its `convert` command. A description of how to use Hootenanny's capability for 'writing to' and 'reading from' an OpenStreetMap Editing API Database is described in this section.
Postgres database tuning is very important to support large data ingests. The Hootenanny default Postgres database instance comes
with moderate tuning, but you may need to perform additional tuning for your database. Postgres tuning is outside of the scope of this document, so visit the Postgres website for more information.
@@ -22,7 +22,7 @@ If you do not already have an OSM API database available, you can run the follow
$HOOT_HOME/scripts/database/CleanAndInitializeOsmApiDb.sh
----------------------------
-This will create an empty OSM API database with a single test user. The test database has this URL:
+This will create an empty OSM API database with a single test user. The test database has this URL:
--------
osmapidb://hoot:hoottest@localhost:5432/osmapi_test
@@ -36,7 +36,7 @@ databases (no other writers present), while the online mode sacrifices some perf
Its easiest to demonstrate the two workflows with some examples:
-==== Use Case #1 (offline) ====
+==== Use Case #1 (offline) ====
Write to a SQL file or CSV files and handle all record ID updates outside of Hootenanny:
@@ -56,7 +56,7 @@ Write directly to a database, handle all record ID updates outside of Hootenanny
hoot convert -D changeset.user.id=1 -D osmapidb.bulk.inserter.disable.database.constraints.during.write=true -D osmapidb.bulk.inserter.disable.database.indexes.during.write=true -D osmapidb.bulk.writer.output.files.copy.location=myOsmDataOutput.sql myOsmDataInput.osm.pbf osmapidb://hoot:hoottest@localhost:5432/osmapi_test
-----------------
-==== Use Case #2 (offline) ====
+==== Use Case #2 (offline) ====
Write to a SQL file or CSV files and manually specify record starting IDs with Hootenanny:
@@ -76,7 +76,7 @@ Write directly to a database, manually specify record starting IDs with Hootenan
hoot convert -D changeset.user.id=1 -D osmapidb.bulk.inserter.disable.database.constraints.during.write=true -D osmapidb.bulk.inserter.disable.database.indexes.during.write=true -D apidb.bulk.inserter.starting.node.id=10 -D apidb.bulk.inserter.starting.way.id=10 -D apidb.bulk.inserter.starting.relation.id=10 -D apidb.bulk.writer.output.files.copy.location=myOsmDataOutput.sql myOsmDataInput.osm.pbf osmapidb://hoot:hoottest@localhost:5432/osmapi_test
-----------------
-==== Use Case #3 (online) ====
+==== Use Case #3 (online) ====
Write to a SQL file and reserve a range of record IDs so that you can execute the SQL at a later time against a live database safely:
@@ -96,25 +96,25 @@ Write directly to a database, reserve a range of record IDs so that you can exec
hoot convert -D changeset.user.id=1 -D osmapidb.bulk.inserter.reserve.record.ids.before.writing.data=true -D apidb.bulk.inserter.output.files.copy.location=myOsmDataOutput.sql myOsmDataInput.osm.pbf osmapidb://hoot:hoottest@localhost:5432/osmapi_test
-----------------
-When `osmapidb.bulk.inserter.reserve.record.ids.before.writing.data` is set to true, the data write operation is completely
-record ID safe, meaning that it can be used while other concurrent users are writing to the target database without fear of
+When `osmapidb.bulk.inserter.reserve.record.ids.before.writing.data` is set to true, the data write operation is completely
+record ID safe, meaning that it can be used while other concurrent users are writing to the target database without fear of
record ID conflicts.
If you are writing extreme amounts of data and have adequate memory available, you may be able to improve the performance of the write operation with additional tweaks. The configuration option, `max.elements.per.partial.map`, determines how many file lines are held in memory before they are flushed out to disk when generating the output files. The default setting is fairly large, but you may be able to improve write performance by increasing it (conversely, if your memory is limited, you may need to decrease this setting). This optimization deals *only* with the output file generation, however, and does nothing to increase the performance of the SQL execution against the target database (database tuning may help there).
-The configuration option, `osmapidb.bulk.inserter.disable.database.constraints.during.write`, allows for disabling database constraints during writing.
-This may speed up data loads. Similarly, the configuration option, `apidb.bulk.inserter.disable.database.indexes.during.write`, allows for dropping database
-indexes during writing and re-enabling them after writing is finished, which may also speed up data loads. Both options should only be used with databases
+The configuration option, `osmapidb.bulk.inserter.disable.database.constraints.during.write`, allows for disabling database constraints during writing.
+This may speed up data loads. Similarly, the configuration option, `apidb.bulk.inserter.disable.database.indexes.during.write`, allows for dropping database
+indexes during writing and re-enabling them after writing is finished, which may also speed up data loads. Both options should only be used with databases
that have been taken completely offline.
-The configuration option, `task.status.update.interval`, simply determines how often you see progress updates when generating the output files. The default
+The configuration option, `task.status.update.interval`, simply determines how often you see progress updates when generating the output files. The default
value is fairly large, but for very large amounts of data you may wish to increase the value of this option to avoid too frequent of progress update messages.
IMPORTANT: The configuration option, `changeset.user.id`, is required and must match a valid user ID in your OSM API database. All writes you do with `hoot convert` will be associated with this user ID.
IMPORTANT: OSM PBF (.osm.pbf) is currently the most efficient OSM source format for writing large amounts of data to an OSM API database, due to the fact the format was designed for optimal performance.
-See the User Guide descriptions of all settings prefixed with `apidb.bulk.inserter` or `osmapidb.bulk.inserter` for a complete overview of the OSM API
+See the User Guide descriptions of all settings prefixed with `apidb.bulk.inserter` or `osmapidb.bulk.inserter` for a complete overview of the OSM API
database writing options available.
=== Reading from an OpenStreetMap Editing API Database
@@ -125,7 +125,7 @@ Reading from an OpenStreetMap Editing API database requires the use of Hootenann
% Hootenanny 'convert' command: hoot convert [args] input output
% Input source:
-a sample OpenStreetMap Edit API Database: osmapidb://hoot:hoottest@localhost:5432/osmapi_test This database is the test database and should be active for testing. However, any database input source may be supplied here.
+a sample OpenStreetMap Edit API Database: osmapidb://hoot:hoottest@localhost:5432/osmapi_test This database is the test database and should be active for testing. However, any database input source may be supplied here.
% Output destination:
define an output osm file: /tmp/ToyTestA-out.osm
@@ -146,7 +146,7 @@ After writing to the OpenStreetMap Editing API Database, the data can be validat
pg_dump -U hoot -h localhost -p 5432 osmapi_test -a > /tmp/ToyTestA-dump.sql
% password=hoottest
% Note: Assume ToyTestA.osm was written to the db previously
-% Note: Assume the arg -D services.db.writer.output.id.mapping=/tmp/idmaps.xml was used
+% Note: Assume the arg -D services.db.writer.output.id.mapping=/tmp/idmaps.xml was used
when writing.
% Install dependencies for the compare script
@@ -189,7 +189,7 @@ Deep inspection of data integrity: PASS
real 0m0.086s
user 0m0.082s
sys 0m0.004s
-------
+--------
=== Validating Reads from the OpenStreetMap Editing API Database
@@ -203,11 +203,11 @@ pg_dump -U hoot -h localhost -p 5432 osmapi_test -a > /tmp/ToyTestA-dump.sql
% create new idmaps xml (idmaps2.xml) from the output data file
./generateIdMapXmlFromOsm /tmp/ToyTestA-out.osm /tmp/idmaps2.xml
-% Note: Assume ToyTestA-out.osm was read from the db and written to file
+% Note: Assume ToyTestA-out.osm was read from the db and written to file
-% compare the output with the database
+% compare the output with the database
time scripts/compareOsmXmlToOsmApiDbDumpWriter /tmp/ToyTestA-out.osm /tmp/idmaps2.xml /tmp/ToyTestA-dump.sql
------
The reading validation output should be similar to the output for the writing validation. If there is a mismatch, then contact the developer with the result so that the software can be checked for bugs.
-
+