Batch import from CSV via client

[ysimonson]

Once #157, it'd be great to add support for batch importing from a CSV file. CSV would act as a decent lowest-common-denominator format. In the future, other formats could be added as well (e.g. RDF XML) based off demand.

[davidkuhta]

Happy to take this on, unless @binoychitale feels its an organic progression of their work.

[binoychitale]

No I think you can pick this :)

[ysimonson]

Thanks @davidkuhta! Here's what I'm thinking for CSV formats, but let me know if you disagree or otherwise think there's a better way to structure things.

We'd support two CSV formats. One for batch inserting vertices and vertex properties, and one for batch inserting edges and edge properties.

Vertices

Format: ID,Type,Property 1,...,Property N. The first two header columns would be ignored, but all subsequent values would be used to determine the names of properties to be inserted.

e.g.:

ID,Type,movie-name,year
e08c8968-5b38-4c55-8dd1-56d7880708f2,movie,Plan 9 from Outer Space,1957
dc8c93c4-dc92-4d00-923d-cc9191f9a946,movie,The Room,2003

Would create vertices with properties movie-name and year.

Edges

Format: Outbound ID,Type,Inbound ID,Property 1,...,Property N. Similarly, the first three header columns would be ignored, but all subsequent values would be used to determine the names of properties to be inserted.

e.g.:

Outbound ID,Type,Inbound ID,year
6bdc1304-e62f-4c38-8a3d-78e923dcb176,acted-in,dc8c93c4-dc92-4d00-923d-cc9191f9a946,2003

Would create an edge with a year property.

[davidkuhta]

@ysimonson agree with the overall concept, but a couple of thoughts came to mind:

1. Validation - What are thoughts on failure for edges (say if inbound or outbound ids don't exist). Any considerations for property types?
2. Would we treat each row as a transaction or the CSV as a whole?
3. Is a uuid as key the anticipated use case or is realistically an integer or string key, which would need to be referenced between a given import of vertices and edges.

[ysimonson]

For this we'd use the bulk insert API. Datastores provide different guarantees on validation and transactionalization.

Good point on (3), since most datasets don't use UUIDs. I think right now the keys in the CSV can realistically only be UUIDs. For users that don't have UUID keys, they'll need to generate a new CSV with them, and maintain a mapping of key -> UUID. I've opened an issue to better support this though. It's not clear to me what the best solution is yet, but does seem like something that should be addressed.