in2csv: GeoJSON metadata discarded

[jayvdb]

The following metadata fields from the "FeatureCollection" object are discarded:

-  "generator": "overpass-ide",
-  "copyright": "....",
-  "timestamp": "2017-07-17T03:42:02Z",

Probably others also.

Perhaps they could be stored in a column header, or a special row, which would assist with Round-tripping data ( #868 ).

[jpmckinney]

It's not clear how to store these in a way that still produces a generic CSV (rather than a CSV that only csvkit knows how to read), but I'll leave the issue open for creative suggestions.

[jayvdb]

I suggested embedding in column header.

[jpmckinney]

I suppose it could be an opt-in flag - as otherwise most users will find it surprising to have extra columns in their CSV output.

[jayvdb]

I was not meaning an extra column; instead add it into a existing column header.

anyway ... a better idea would be to read the metadata from the original. Assuming csvkit has a sensible streaming json reader, it could read only as much of the original to capture the metadata at the top, and emit that in the output.

[jpmckinney]

Yeah, my challenge is determining where to output it. in2csv with GeoJSON just outputs a row for each feature (as you know). If we put metadata in a special row or in a special header, then that pollutes the data with non-data that other tools won't know how to parse, because storing metadata like that in CSVs is not standardized. And then we'd have to add support to all csvkit tools for parsing that metadata to avoid it interfering with other operations.

[jayvdb]

Another, probably better idea, is adding an "append" / "merge" option, so that I can provide a very small .geojson file with the metadata to be used and then the extra records can be merged into it (in-place rather than stdout would be my preference).

Then in2csv could also emit the metadata as a small geojson file, containing any data it couldnt put into the csv.

[jpmckinney]

That sounds reasonable. Like a --write-metadata option for in2csv, and a --read-metadata for csvjson.