split the Raft implementation into its own library

[emaxerrno]

I was looking into building a replicated state machine and I really like this C++ implementation.

I was wondering if the team has any interest in separating the log cabin impl of raft into it's own library and integrating back into this project.

1. I'm willing to put in the work and port the tests
2. I also was reading cloudera's recent Kudu implementation and feel there is a need for a solid c++ impl that is easy to integrate into other projects.
3. Maybe the guys at rethinkdb can also integrate it into their project if it makes sense for them at some point.

Thanks.

[ongardie]

Hi Alex,

Splitting out a library is a goal I've always had for LogCabin, but it's never made it to the top of my priority list. It's also come up in discussions with RAMCloud (cc @cstlee), where the RAMCloud coordinator may be able to leverage a library to get better performance compared to accessing a separate service. And we considered doing this for Scale Computing (cc @nhardt), but then decided to go with a network service instead, at least for now.

A library does face a few challenges, however. The API may be tricky to define, and the language or threading model or other assumptions may not meet everyone's needs. There was a nice discussion on Hacker News about this a few months ago: https://news.ycombinator.com/item?id=9491130 . There's also probably a few leaky abstractions in LogCabin that would need to be worked out.

So it'd be a good thing, but it may be difficult. I don't know the best approach, honestly. I bet if you just dove in to try it, you'd quickly find something that needs to be refactored. That could make an independent and useful patch set, and then try with the librarification again. After enough iteration, hopefully you'd succeed.

One thing to keep in mind is that LogCabin needs to keep supporting rolling upgrades and such. So changes to internals with no external effects are fine, but be careful with file and network formats, public client APIs, and the config settings. I think that should be doable.

Check out the terribly organized slides at https://logcabin.github.io/talk/ if you haven't seen them already. They might help you navigate the code base or understand some aspects that aren't immediately obvious otherwise.

Happy to chat more if you choose to dive in.

-Diego

[emaxerrno]

Hi Diego,

Good feedback - exactly what I was looking for. I'll do some deep dive this weekend to have more realistic estimates and probably ask for guidance as far as navigating the the internals, etc.

Thanks again!

[sangechen]

https://github.com/willemt/raft
This small project is a good one. ^_^

[ongardie]

@senior7515 any news on this? If not, let's rename the issue title at least :)

[emaxerrno]

@ongardie

Hi Diego,

Yes, thanks for checking in.

I did do a dive into the internals, and found it pretty easy to navigate the code - it's pretty clean.

I basically got distracted with other projects - no excuse :( - sry for all the pomp and circumstance. Feel free to close, I'll reopen when I have more time, still on my todo list for this year.

[tnachen]

@ongardie I was going to propose this and found this issue. I'm also searching for a good Raft C++ implementation that has been tested, and think LogCabin's impl suites best.
@senior7515 if you don't have time I'd like to take a stab as well, including some other interested folks that I know of looking for a good C++ impl to use in production.

[emaxerrno]

Hey tim sounds good! Are you Thinking of removing the multipaxos out of mesos? Pretty cool. 
Post the branch name :) 

Sent from mobile, please forgive my handwriting.

On Thu, Mar 3, 2016 at 9:33 AM -0800, "Timothy Chen" notifications@github.com wrote:

@ongardie I was going to propose this and found this issue. I'm also searching for a good Raft C++ implementation that has been tested, and think LogCabin's impl suites best.

@senior7515 if you don't have time I'd like to take a stab as well, including some other interested folks that I know of looking for a good C++ impl to use in production.

—
Reply to this email directly or view it on GitHub.

[cstlee]

I've actually been thinking about this too and will likely build or use a raft library for my next project in the coming months. The question that has been lingering in my mind is what the right module/abstraction boundary for this would be. If it makes sense, I'd love to collaborate.

@tnachen and @senior7515 What kind of abstractions were you both looking for? (@ongardie stop me if this is the wrong channel to have this discussion)

[tnachen]

@cstlee I haven't dived deep yet, but will love to collaborate depending on your availability. I think I need to get to know the current interface and I'd like as a first step to just start seperating out the Raft classes into a seperate place and see what's the dependency looking like.

[ongardie]

Thanks for the update, @senior7515, and it's awesome to see this much interest. Let's definitely keep this issue open, and this seems like the right place to have the discussion. I'll take the liberty of editing the title a little.

Here's some questions to think about. It'd be interesting if everyone agreed :).

• Would you want the library to do networking? If not, would you want it to even do serialization?
• Would you want it to write to storage (disk)? For the log? For snapshots?

[tnachen]

Hi @ongardie, my 2cents around this is that ideally networking is not handled or pluggable by the library, and therefore I don't think serializing should either.
Storage however I'm not sure of yet, what is your suggestion?

[ongardie]

Hmm. I suspect most people that want to embed a Raft library will already have a networking stack they want to use, but that doesn't necessarily imply they want to do their own serialization. Having the user serialize also makes the user code more tightly coupled with the library internals in case of new fields in the messages and such. Maybe a good compromise is for the library to build up the protobuf structure and pass that up to the user code; the user code can either:

• Have the protobuf library serialize it (loose coupling, user code has no knowledge of message contents), or
• Dig into that protobuf message to do its own custom serialization (tight coupling, inspecting field by field).

I'd also guess at least some users embedding a Raft library would be happy with LogCabin's built-in storage module. Many of the systems embedding a Raft library may not otherwise write to disk at all.

[tnachen]

@ongardie I agree on the networking and storage parts, and yes I think serialization as you mentioned makes sense. I think for v1 of the library I don't think it requires too much customization and plugin for serialization, but can definitely see benefits longer term to wire that in.

@ongardie perhaps we can set something up to chat about what next steps should be like as you know much more about the code base (w/ @cstlee @senior7515) and map out the steps necessary. How that sounds?

[ongardie]

@tnachen, I'm open to that. How's next Monday (3/21) or Tuesday (3/22) afternoon (PDT) for people? Let's use this Hangout room [edited to remove link, will post a new one later]. You're also welcome to join me at the Salesforce office at 1 Market St in San Francisco if that's convenient--just let me know.

@cstlee @senior7515 @nhardt

[tnachen]

I'm open next Monday, Mesosphere office is pretty close by so I think I'll pay a visit and we can join the hangout from there.

[astigsen]

I would love to help out with this. It would be awesome to have a pluggable C++ implementation of Raft with a real community around it. Our office is pretty close by as well, so if you have room I would be happy to join you next Monday afternoon.

[nhardt]

Have some hard to schedule personal matters coming up soon, but I will put it on my calendar and join the hangout if I am available.

[emaxerrno]

@ongardie awesome. I have some thoughts of how I was planning on using it, and definitely think we can always do this in steps. i.e.: no reason not to go with the serialization library being part of the framework in the beginning and stub it out later. Also seems like a much more reasonably scoped project.

I would also in fact encourage the storage part to be a module as well. For example, I was thinking of using NVM (non-volatile memory) for the storage (pmem.io). Again, happy to use LogCabin's default to start out.

can you post the time or invite me via email? (I'm in NYC, so can't make the salesforce meeting)

Exciting!

[ongardie]

Ok, the meeting will take place on Monday, March 21 at 2-4pm PDT.

Local folks that I know are interested: I just sent you a calendar invite with the details. If I missed anyone, let me know.

Remote folks, including @senior7515: The hangout URL is: https://plus.google.com/hangouts/_/salesforce.com/logcabin
(When the time comes, I'll need to click Approve as people join the hangout, but other than that technicality, we can consider it public. The reason for using a Google Apps Hangouts URL is that there's a physical Chrome for Meetings device in the meeting room that I don't think would connect to a non-Google Apps Hangouts URL.)

[ongardie]

cc @Ghatage @apavlo

[cstlee]

@senior7515 The meeting is starting in case you are still interested in joining.

[emaxerrno]

yes.!

[ongardie]

As we talked about, the new mailing list is at https://groups.google.com/forum/#!forum/logcabin-dev . Let's still err on the side of GitHub issues for anything actionable, but use logcabin-dev for questions and open-ended discussions.