Add support for multiple queue databases in a single app

[rglover]

I made shortsighted move in respect to queues. I currently limit an app to one queues database via the config, but there could be a scenario where you want to target different databases for different queues.

Proposal I came up with is to add a database option to queue definitions like this:

{
  database: {
    provider: 'mongodb',
    name: 'app',
  },
  runOnStartup: true,
  cleanup: {
    completedAfterSeconds: THREE_DAYS_IN_SECONDS,
    failedAfterSeconds: ONE_WEEK_IN_SECONDS,
  },
  retryJobsRunningBeforeRestart: true,
  concurrentJobs: 100,
  jobs: {
    provision_instance: {
      requeueOnFailure: true,
      run: async (payload = {}, job = {}) => {
        await provision_instance({ ...payload, job });
      },
    },
    check_instance_status_on_provider: {
      requeueOnFailure: true,
      run: async (payload = {}, job = {}) => {
        await check_instance_status_on_provider({ ...payload, job });
      },
    },
  },
}

Internally, Joystick would check to see if this value is defined. If it is, it would set the target database to that provider/database name combo. If it's not set, it would default back to what it is now (just use the database with queues: true set).

Only thing I don't like is that it could get confusing to have both options. May be worth forcing database specification instead of the queues: true thing. Problem there is if I only have one database, I wouldn't have a name (that only applies when I have multiple db connections for the same provider). That could be mitigated by checking internally to see if there are multiples of the specified provider and if not, just get the first one (and if there are, check for the specificied name).

[rglover]

If you decide the force option, you could warn on server startup in the console if a queue definition is missing its database option.

[rglover]

Had a lightbulb on this. You could just tell Joystick that a queue is external and on one of your other databases:

export default {
  external: true,
  database: {
    provider: 'mongodb',
    name: 'provision',
  },
};

This would be the only thing in the queue definition. That way, you could attach an external queue to your app at process.queues.<queue_name> but the data source would be in another database.

As long as a connection to the specified database exists, the mapping would take place and you could queue jobs across databases seamlessly.