running multiple indexers simulatneously blocks the database

[karlicoss]

Basically the following scenario

• you have an indexer that runs, say, for an hour
• you have an indexer that runs for a minute

If you start the former first, it will get exclusive write lock (here, so the one that runs for minute will spend an hour waiting for it to be released before it can do anything.
As far as I understand, sqlite doesn't support multiple simultaneous writers, and even if it did need to think what should happen if both indexers update the same visits

Some options

• only take the write lock to actually write the collected visits to the db. Currently it happens iteratively so we don't need to keep all the visits in the RAM
• write to a temporary database first, and then merge with the main one?