Separate persistence sensitive and insensitive messages

[BusyJay]

raft library uses a Vec to store all the generated messages. They are sent out only after the readiness is handled unless it's leader. In such case, it's recommended to send out all those messages before writing to the disk to improve latency. Let's call those messages as persistence insensitive messages.

Actually followers have persistence insensitive messages too. For example, it's totally OK to respond leader's heartbeats without handling any readiness. Because they don't provide any additional information to leader except raft's term. And term only changes when there is an election or receiving a message from new leader/candidate. In both cases, term should be persistent in quorum as campaign related messages are sent after persistence. So even minorities' term goes back, election safety is still guaranteed. This also applies to MsgReadIndex.

To summarize, all following messages are persistence insensitive:

    MsgPropose
    MsgAppend
    MsgSnapshot
    MsgHeartbeat
    MsgHeartbeatResponse
    MsgTimeoutNow
    MsgReadIndex
    MsgReadIndexResp

Send MsgHeartbeatResponse and MsgReadIndex eagerly can reduce read latency in some cases.

If we use two Vecs to separate the sensitive and insensitive messages, it's easier to manipulate the order on the application side.