Transaction appears to time out / overflow

[glesica]

A user has observed the following:

tx = Neo4j.transaction(con);
for (counter, row) in zip(1:nrow(dataframe), eachrow(dataframe))
   tx("MERGE (g:Type1 {name: '$(row[:text1])'})
                MERGE (s:Type2 {name: '$(row[:text2])'})
                ON CREATE SET s :MyLabel
                MERGE (s)-[syn:HAS_CONNECTION_TO]->(g);", submit=false);
end
commit(tx);

After approximately 147 rows he receives a 404, connection failed error. If he runs just the first 100 it works fine.

We're not sure if the transaction is just filling up, or timing out, or if there is another problem, perhaps a bug in the library and the way it uses the Cypher API.

@lyonwj

[glesica]

Since the code sets submit=false we're not making any requests during this loop, or we shouldn't be... https://github.com/glesica/Neo4j.jl/blob/master/src/transaction.jl#L36

[glesica]

We do make a request when the transaction is created, so this could be a timeout issue... https://github.com/glesica/Neo4j.jl/blob/master/src/transaction.jl#L24

[StefanHaunsberger]

@glesica
The way I currently handle it is to submit every n rows and do a Neo4j.commit every y rows.
For example, to add data contained in a DataFrame x to the database I do the following:

tx = Neo4j.transaction(con);
for row in DataFrames.eachrow(x)
   try
      tx(query,
         "field1" => row[:column1],
         "field2" => row[:column2];
         submit = row.row % 3000 == 0
      );
      if row.row % 20000 == 0
         println("\t->commit 20000 records...[$(row.row)]");
         res = Neo4j.commit(tx);
         tx = Neo4j.transaction(con);
      end
   catch y
      println(y.msg);
      error("Error at row: + $(row.row)");
   end
end
res = Neo4j.commit(tx)

I know it is just a workaround and a bit of overhead but once wrapped into a function, it's straightforward.