Merge Request - Noizu Fork

[noizu]

Hello sheharyarn, as mentioned previously I have a forked version of your library which adds support for priority levels, sharded workers (to avoid supervisor bottlenecks), node tenancy (jobs only run on the server they were created on or specified to run on), and dirty mnesia transactions using raw :mnesia module due to performance bottlenecks under high load.

I'd like to get things tidy enough to be able to merge upstread and avoid the need to maintain a separate fork. With that in mind I've made these features compile time configurable with the defaults behaving essentially like vanilla Que with the exception of the priority level changes and an error handler that treats :no_proc responses as success (which is negotiable) as they are generally caused by que jobs that exit too quickly.

However there is a breaking change to the table definition to include the priority level and node tenancy fields, and I am not sure if there is an elegant way to work around that with Memento or if I should further hide that change behind config options and modify the table structure only if tenancy and or priority levels are enabled.

When you have a chance give things a peak and let me know of any feedback or criticism you have that would need to be dealt with before making a merge request.

master...noizu:master

[noizu]

Forgive the typos in the readme section, haven't had enough coffee yet this morning.

[sheharyarn]

Wow! I went over your fork and I love all the interesting ideas and features you've been working on. I would love to collaborate with you on them and merge them into the package, but I believe we should focus on one at a time with individual Issue & PR for each feature:

• Priority Queues (Priority Queues #20)
• Distributed Workers
• Throughput Optimization
• Sharded Workers

This would ensure that we don't miss anything or accidentally introduce any regressions, and have up-to-date docs and full test coverage for each feature. We'll also be able to let users reliably update to newer versions by automatically migrating tables so definition changes don't break the code.

We can start with priority queues. I'll create an issue where we can discuss specific implementation details.

[noizu]

Sounds Good.