Jobs are picked up from queues that the server is not supposed to listen to

[webseb]

I found several issues that sound similar but never a real solution.

I have two Hangfire servers. Let's call them A and B. Server A is configured to listen to QueueA (and only QueueA, no "default" queue!) and Server B listens to QueueB (also no "default" queue). I have a job that is scheduled on QueueA (doesn't matter if I schedule it as a background or a recurring job). But the job is (sometimes) picked up by Server B.

services.AddHangfireServer((sp, options) =>
{
    options.Queues = new[] { "queue_a" };
    options.WorkerCount = 1;
    options.ServerName = "server-a";
});


services.AddHangfireServer((sp, options) =>
{
    options.Queues = new[] { "queue_b" };
    options.WorkerCount = 1;
    options.ServerName = "server-b";
});

recurringJobManager.AddOrUpdate<JobA>(
                    nameof(JobA),
                    "queue_a",
                    job => job.Execute(CancellationToken.None),
                    "<some cron expression>");

The problem: Server B does not reference the same set of assemblies and is thus not even able to load the job from the database. Instead of letting the other server pick it up, Server B keeps trying until the retry attempts are exhausted and the job fails.

Hangfire.Common.JobLoadException: Could not load the job. See inner exception for the details.
 ---> System.IO.FileNotFoundException: Could not resolve assembly 'JobA'.
   at System.Reflection.TypeNameParser.ResolveAssembly(String assemblyName)
   at System.Reflection.TypeNameParser.GetType(String typeName, ReadOnlySpan`1 nestedTypeNames, String assemblyNameIfAny)
   at System.Reflection.TypeNameParser.NamespaceTypeName.ResolveType(TypeNameParser& parser, String containingAssemblyIfAny)
   at System.Reflection.TypeNameParser.Parse()
   at System.Reflection.TypeNameParser.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Assembly requestingAssembly, Boolean throwOnError, Boolean ignoreCase, Boolean extensibleParser)
   at System.Type.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError)
   at Hangfire.Common.TypeHelper.DefaultTypeResolver(String typeName) in C:\projects\hangfire-525\src\Hangfire.Core\Common\TypeHelper.cs:line 78
   at Hangfire.Storage.InvocationData.DeserializeJob() in C:\projects\hangfire-525\src\Hangfire.Core\Storage\InvocationData.cs:line 96
   --- End of inner exception stack trace ---
   at Hangfire.Storage.InvocationData.DeserializeJob() in C:\projects\hangfire-525\src\Hangfire.Core\Storage\InvocationData.cs:line 120
   at Hangfire.RecurringJobExtensions.TriggerRecurringJob(IBackgroundJobFactory factory, JobStorage storage, IStorageConnection connection, IProfiler profiler, RecurringJobEntity recurringJob, DateTime now) in C:\projects\hangfire-525\src\Hangfire.Core\RecurringJobExtensions.cs:line 115
   at Hangfire.Server.RecurringJobScheduler.ScheduleRecurringJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, RecurringJobEntity recurringJob, DateTime now) in C:\projects\hangfire-525\src\Hangfire.Core\Server\RecurringJobScheduler.cs:line 333

I have a job filter that makes sure the original queue is used, even on retries. So the issue is not related to the job being re-queued with the wrong queue.

Why does a server pick up jobs from queues, that it is not meant to listen to and how can i fix that?

BTW: This issue is not related to the storage used. I see it with SqlServer, Redis and PostgreSql.