SPIKE: Design doc for auditing tool for Waltz

[whynick1]

We will need an auditing tool for Waltz to validate that no transactions are ever lost. Ideally, this tool will run continuously, and immediately alert us to any lost transactions.

@ymatsuda, have you thought about any strategies for this? It seems loosely similar to some of the auditing work for Kafka.

I think that there are two things that need to be audited: 1) that a transaction wasn't lost before it was even persisted to Waltz (i.e. client thinks it was persisted, but it wasn't successfully), and 2) that a transaction was persisted to Waltz, but later disappeared for some reason.