How to maximize concurrency in persisting?

[rickyyx]

Tracking issue for the below question:

Question Statement :

Given that dependency between transactions can be known, are there any optimizations possible to maximize concurrency in persisting the transactions (both logs, data, and the commit flag)

Assumptions:

1. Write-ahead logging (Undo logs)
2. Expanded persistence domain in hardware ([Asynchronous DRAM Refresh](https://software.intel.com/en-us/blogs/2016/09/12/deprecate-pcommit-instruction) enabled).

Context:

In general, a transaction has to impose the below ordering when using write-ahead logging to ensure persistence.

LOG | DATA | COMMIT-FLAG

Such ordering has been fulfilled with a combination of PCOMMIT and SFENCE, aka PFENCE currently. However, with the new hardware (Assumption 2), there is potential for a simpler programming order. We only need to have one SFENCE as the barrier, rather than two SFENCE around the PCOMMIT as before. As long as writes have arrived the Write Pending Queue in the memory subsystem(WPQ), they will eventually show up at crash recovery. The draining of the WPQ is no longer needed. We can run pmem_flush instruction or its equivalent in order, and use SFENCE to order them. Besides the reduced number of SFENCE, reducing the number of pmem_flush issued is another small thing to improve the concurrency. For example, only do flush at the end of the transaction rather than doing flush at every single write so that writes to the same location will only incur one flush.

However, on a multi-core systems, there can be conflicting transactions. There is thus additional ordering when persisting them concurrently. The minimal ordering required follows Peter Chen's paper.
For transactions that have conflicts (Write and Write):

• Their logs have to be ordered, therefore, an additional barrier between persisting their logs are needed.
• Their data persists need to ordered, however, this is already implicitly enforced by memory consistency.
• Their commit flags will need to be ordered: this should work in the same way as their logs, an additional barrier is needed.

Design

In order not to couple the transaction volatile concurrency management with persistence, we could achieve greater persists concurrency by having dedicated persisting threads that receive persists requests (pmem_flush and pmem_drain) from working threads, and executing them in parallel among the persisting threads.

Notes

1. CLFLUSHOPT operation
   The instruction is ordered with other CLFLUSHOPT to the same cache line. (So data from conflicting transactions will be ordered)
2. CLWB operation
   Do not order with writes. So prefetching of data can be re-ordered with the flush.