Seeking feedback: bee-queue 2.0 plan

[LewisJEllis]

While investigating #103 I re-read the process() implementation and how we conjoin promises + callbacks for job handlers and it got me thinking a little bit - I want to get away from the callback API, and it's also just sort of that time where we get fully onboard the promises + async/await train.

Plan:

• Use, treat, and support async/await as a first-class thing
• Stop supporting/using callbacks for anything
• Migrate Redis client to ioredis
• No longer require promise-callbacks dependency
• Support only Node 8.x+ for native async/await

Pros:

• Remove promise-callbacks dependency (and save a little bit of corresponding overhead)
• Simplify some dual-pronged code paths and error handling scenarios
• Improve overall codebase readability
• Better utilize native Promise and async/await performance improvements (which have been significant in the last year)

Cons:

• Will be backwards-incompatible, with no more callback support
  • Should not require code changes for users already using promise-based handler functions
  • Users requiring the callback API can use 1.x
• Will not support old Node versions
  • Node 4 is EOL, Node 6 is old, everyone who will upgrade bee-queue to 2.0 will be on Node 8+

In general, asynchronous control flow stuff is easy to get wrong and having one API instead of two reduces the surface area for potential problems (like the confusion in #103). I also think the codebase could get a lot more readable with some async await mixed in and I refuse to introduce a build step.

I don't think 2.0 will change how bee-queue works at a Redis level - just improving the JavaScript interface.

So, seeking feedback:

1. How does this sound?
2. Are there any other things that should be thought about for 2.0, since it'll be a major version?

We can add other potentially-minor-version things to the milestone too, but let's especially make sure to think about changes we might want that could only go into major versions.

/cc @skeggse @bradvogel

[bradvogel]

Yes - is great!

[v4l3r10]

Waiting for new release! 😄

[LewisJEllis]

Started poking into this today and realized that migrating to ioredis is also going to be part of this update; when #21 was originally opened, it wasn't as clear whether ioredis was a better choice, but now it's clear that ioredis has pulled ahead, and as a bonus their API supports promises out of the box.

[hyeomans]

Hey ioredis is great, another thing that supports is redis cluster which npm redis does not support out of the box.

[bradvogel]

+1 on ioredis. Excited for this update Lewis!

[denigmus]

Hope with ioredis you will learn bee-queue, i.e. redis.createClient support automatic re-connections on redis connection lost.

[hughsw]

What is the status of bee-queue 2.0 ? I'm interested in contributing, particularly as I seem to have a knack for finding issues that are hard to really fix without making breaking changes #186 #184 #189

It'd like to add to the list of features:

• add support for job priority (perhaps at the Queue configuration level)
• simplify Job interface to reduce statefulness of Job instances
• sort out and resolve the Redis reconnection issue
• optimize support for large job data (my use case is emails with PDF and image attachments)
• Queue pause/resume
• improve documentation to help avoid duplicate RTFM Issues that keep coming up
• improve and add to examples with realistic best-practices, including Docker usage
• groom and close Issues

Behind the scenes:

• performance improvements and fewer lines of code
• testing with clients and workers in separate Node processes to flush out race conditions
• Convert to Typescript and modern Node

[jasrusable]

Hey @hughsw, I've noticed that you have been quite active on the bee-queue repo and I thought you might find my project, ConveyorMQ interesting.

I am the author of ConveyorMQ which I have built with the intention of it being a fast, robust and reliable distributed task queue for Node using Redis. My intention is to make ConveyorMQ the most performant, feature-full and defacto-standard job/task queue system for Node.

ConveyorMQ already implements all of bee-queue's features, as well as the following:

• Simple, efficient design
• Uses ioredis
• Queue pause/resume
• Written in Typescript with async/await & promise API
• Supports large job/task data

Additionally, my road map currently includes:

• Task prioritization
• Queue rate limiting
• A Dashboard/WebUI

I would be grateful for any feedback you might have to offer :)

[stavalfi]

node-redis is not maintained, and it uses weird dependencies which bundlers don't like. what is the status of moving to ioredis?

[compwright]

@stavalfi not planning to move to ioredis.