Better benchmark

[adamgundry]

I've been looking at the existing benchmark, and it seems there are a couple of issues:

• The state data type is [[[Int]]], which leaves lots of opportunities for laziness (e.g. when deserialising a checkpoint).
• The benchmark merely measures the cost of opening and closing the state, without running any updates/queries which would force evaluation.

At least in some applications, it is better to use a strict state type and fully evaluate the state at load time, which might take longer but should use less memory and may avoid the first query/update running slowly due to the need to force lots of thunks in the state. The benchmark should better reflect this mode of use.

[stepcut]

Something else to think about is the 'memory' backend. acid-state has a backend that stores the state in memory instead of on the disk. That, of course, means the state is lost when the application exits. The purpose is to make it easier to write tests because you don't have to worry about cleaning up the files on the disk.

But perhaps it is also useful for the purpose of benchmarking? The 'memory' backend should be the fastest backend since it doesn't have to do disk I/O. So, that would give you some idea how much extra time is being spent on the disk I/O?

Perhaps its not useful, but I thought I would at least mention that it exists.

[adamgundry]

I agree it would be good to have benchmarks comparing the memory backend to local backends (with various serialisation layers). I'll look at generalising the benchmarks over different backends.