force change one node to become a single-node cluster

[jujing]

It is better to give a way to force change one running node to become a single-node cluster which was one of the several-nodes cluster before.
It can be used in following scenarios:

1. Majority of the nodes crashed and cannot be restored. We should use one of the left nodes to provide service, though some committed log maybe lost.
2. In two nodes scenario, when one node crashed, we can use the left one to provide service.

[ongardie]

Link back to original discussion on raft-dev: https://groups.google.com/forum/#!topic/raft-dev/2gjCi6OdhJs

[ongardie]

[Diego] You're going to need to make one change to make this work: you'll need a way to tell a LogCabin server to force-append a configuration entry to its log. I think I'd suggest doing this as a logcabinctl RPC handler, and you can probably steal some code from the LogCabin --bootstrap path. If you're prepared to work on this, please start an issue for this on GitHub and let's move the discussion about how to implement it there.

[Song] I tried to modify it. It seems work well. I modified base on Example/Reconfigure. I fill the request.old_id() a zero. Then, in RaftConsensus::setConfiguration, use a branch to handle it when (request.old_id() == 0), force-append a configuration entry to its log. I will start an issue for it, and make the implement more formally. The problem is that, I cannot submit code to github in my working environment because of network limitation.

The problem with using the Example/Reconfigure path is that it will try to submit the request to the cluster leader, and such requests normally get rejected by non-leaders. But in this situation, there is no leader, yet we still need to process the request. That's why I suggested a logcabinctl/ServerControl RPC in my email. ClientService requests are currently all routed to the Raft leader, whereas ServerControl requests are sent to a particular server.

[ongardie]

(Changed the title slightly because we have a tool here at Salesforce called Supportforce, and I got confused every time I read the old title.)