Make database/storage pluggable

[metakeule]

It would be nice to be able to plug in any database that implements an interface instead of having to use redis. For example boltdb could be an interesting backend that could also be embedable.

[albrow]

Thanks for bringing this up! Someone asked the same question on Reddit. Here was my response:

In short, that falls outside of the scope of what I want to do with this project. It's not quite as simple as an interface for "job persistence". The library relies very heavily on Redis for certain guarantees (for example, the fact that Redis is single-threaded and no other command can run in the middle of a transaction or lua script), and I'm not comfortable enough with every aspect of SQL or other NoSQL databases to be able to provide the same guarantees. As a specific example, at the core of the library is a sorted set which represents all the queued jobs, called the "queued set". Each worker pool periodically gets the next N jobs from the queued set and moves them to an "executing set". Because Redis is single-threaded, it's guaranteed that no two pools will attempt to start working on the same job. In addition, the jobs library relies on Redis pub/sub for communication between worker pools, and without that I think you would need to implement some sort of service discovery, which really complicates the implementation. If another developer really wanted to take this on and submit a PR, I would consider it. But the code would need to be heavily audited and I'm not sure I know enough to do that for other database implementations.

I'm not familiar with boltdb. At first glance it looks like the read/write transactions could cover the same functionality as redis transactions and lua scripts. However, unless there's a really compelling reason to use it, it's not something I'll be adding to the project personally. What are your thoughts?

[albrow]

It's been a few days. I'm thinking of closing this issue, but I wanted to offer a chance for people to speak their mind first. Does anyone have any additional feedback here? This is not something I want to add in the near term, but if there is a really compelling use case and there is a way to offer the same functionality and guarantees it's something I'll consider. Especially if I have help. Otherwise I'll keep moving forward as planned and only support Redis.

[kdar]

You can take a look at https://github.com/siddontang/ledisdb, which itself supports different backends. I haven't really looked at what is needed by jobs, but if you want, take a look and see if it fulfills your needs.

[m0sth8]

👍 for rabbitmq and mongodb support

[albrow]

Okay thank you all for your feedback. @kdar ledisdb looks pretty amazing, but AFAICT it doesn't support transactions or pub/sub, both of which are vital for offering the guarantees I want to offer. @m0sth8 I'm roughly familiar with both rabbitmq and mongodb, and I'll look at both of them more closely later on.

I'm going to make this issue a low priority for now and take a look at it again after version 1.0 is released. Since it's apparent that people are really interested in this feature I promise I will take a look at it again. There are a few important things I want to get right before adding additional features and complexity.

[pyrossh]

@albrow Nice explanation on how you implemented it using redis sorted sets. Maybe that should go in the Readme as well. I was thinking of a storage plug for postgres as the database and etcd for synchronizing the workers (which would be complicated) but gives us complete ACID transaction capabilities of postgres. But anyway its just a thought Redis is just fine. But it would be good if you made it pluggable like a interface which anyone can implement so if they need to add a datastore it would be easier for them to integrate.
It might end up becoming like this https://github.com/xyproto/permissions2