scheduledJobsRequestSize not working as comment describes: is it a bug or just misleading comment?

[choesw]

Hi, I am testing the features of your projects for application to my company project. As I am modifying the configurable properties, I found this comments on the properties in JobRunrProperties.java:

      /**
       * Sets the maximum number of jobs to update from scheduled to enqueued state per polling interval.
       */
      private Integer scheduledJobsRequestSize = 1000;

      /**
       * Sets the query size for misfired jobs per polling interval (to retry them).
       */
      private Integer orphanedJobsRequestSize = 1000;

      /**
       * Sets the maximum number of jobs to update from succeeded to deleted state per polling interval.
       */
      private Integer succeededJobsRequestSize = 1000;

I want to know if the comments are written right.
I have set the properties as below and scheduled 20 jobs at the same time. (I am using .yaml in Spring Boot project.)

...
org:
  jobrunr:
    background-job-server:
      enabled: true
      worker-count: 2
      poll-interval-in-seconds: 60
      delete-succeeded-jobs-after: 10m
      permanently-delete-deleted-jobs-after: 10m
      succeeded-jobs-request-size: 5
      scheduled-jobs-request-size: 5
      orphaned-jobs-request-size: 5
...

According to comment above, the expected behaviour is:

• 5 jobs are enqueued in the first poll interval.
• next 5 jobs are enqueued in second poll interval.
• all 20 jobs are enqueued after 4 minutes.

But it actually does not work like that. No matter how many jobs I schedule, backgroundJobrunrServer always ends up enqueuing all the existing scheduled jobs in single poll interval.

These are mysql DB logs. As you can see, all 20 jobs are enqueued within a minute.

2024-04-04T00:31:48.000572Z	 1773 Query	select jobAsJson from jobrunr.jobrunr_jobs where state = 'SCHEDULED' and scheduledAt <= '2024-04-04 00:32:47.998976' ORDER BY updatedAt ASC LIMIT 5 OFFSET 0
2024-04-04T00:31:48.019486Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.023546Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.025699Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.027270Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.030430Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.037564Z	 1773 Query	select jobAsJson from jobrunr.jobrunr_jobs where state = 'SCHEDULED' and scheduledAt <= '2024-04-04 00:32:48.036589' ORDER BY updatedAt ASC LIMIT 5 OFFSET 0
2024-04-04T00:31:48.038507Z	 1772 Query	select * from jobrunr.jobrunr_jobs_stats ORDER BY total ASC LIMIT 1 OFFSET 0
2024-04-04T00:31:48.048330Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.053817Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.055782Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.057057Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.058564Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.060496Z	 1773 Query	select jobAsJson from jobrunr.jobrunr_jobs where state = 'SCHEDULED' and scheduledAt <= '2024-04-04 00:32:48.059723' ORDER BY updatedAt ASC LIMIT 5 OFFSET 0
2024-04-04T00:31:48.065874Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.067804Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.068859Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.069764Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.070626Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.072725Z	 1773 Query	select jobAsJson from jobrunr.jobrunr_jobs where state = 'SCHEDULED' and scheduledAt <= '2024-04-04 00:32:48.072146' ORDER BY updatedAt ASC LIMIT 5 OFFSET 0
2024-04-04T00:31:48.079738Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.081026Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.081947Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.083082Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.084936Z	 1773 Query	update jobrunr.jobrunr_jobs SET version = 2, jobAsJson = '{"version":2,"jobSignature":"com.sample.pidove.domain.job.JobrunrTestController.doJobAfterSeconds(java.lang.Stri
2024-04-04T00:31:48.086006Z	 1773 Query	select jobAsJson from jobrunr.jobrunr_jobs where state = 'SCHEDULED' and scheduledAt <= '2024-04-04 00:32:48.085764' ORDER BY updatedAt ASC LIMIT 5 OFFSET 0
2024-04-04T00:31:48.087429Z	 1773 Query	select jobAsJson from jobrunr.jobrunr_jobs where state = 'PROCESSING' AND updatedAt <= '2024-04-04 00:27:47.968387' ORDER BY updatedAt ASC LIMIT 5 OFFSET 0
2024-04-04T00:31:48.088210Z	 1773 Query	select jobAsJson from jobrunr.jobrunr_jobs where state = 'SUCCEEDED' AND updatedAt <= '2024-04-04 00:21:48.087934' ORDER BY updatedAt ASC LIMIT 5 OFFSET 0
2024-04-04T00:31:48.088772Z	 1773 Query	delete from jobrunr.jobrunr_jobs where state = 'DELETED' AND updatedAt <= '2024-04-04 00:21:48.088506'

and this is debug log from application server.

2024-04-04T09:31:48.086+09:00 DEBUG 87145 --- [pool-6-thread-2] org.jobrunr.server.JobZooKeeper          : Found 20 scheduled jobs to enqueue.
2024-04-04T09:31:48.087+09:00 DEBUG 87145 --- [pool-6-thread-2] org.jobrunr.server.JobZooKeeper          : Found 0 orphan jobs.
2024-04-04T09:31:48.088+09:00 DEBUG 87145 --- [pool-6-thread-2] org.jobrunr.server.JobZooKeeper          : Found 0 succeeded jobs that moved to DELETED state as part of JobRunr maintenance
2024-04-04T09:31:48.088+09:00 DEBUG 87145 --- [pool-6-thread-2] org.jobrunr.server.JobZooKeeper          : Found 0 deleted jobs that were permanently deleted as part of JobRunr maintenance
2024-04-04T09:31:48.089+09:00 DEBUG 87145 --- [pool-6-thread-2] o.j.s.s.BasicWorkDistributionStrategy    : Can onboard 2 new work (occupiedWorkerCount = 0; workerCount = 2).

I don't think the comment is describing the properties right, so I wonder if it is a bug or just not written right. Or, maybe my configurations are wrong?
I also wonder if there is any way that can actually limit the size of jobs that are enqueued per poll interval.
Thank you in advance!

[rdehuyss]

I think the docs are indeed not 💯 % clear. It's the amount of jobs that are fetched per database round-trip. Some companies / developers use it to optimize throughput. We will update our docs to make it clearer.

JobRunr Pro v7 will support rate limiting and allows you to limit the amount of enqueued jobs (using concurrent rate limiter or sliding time window rate limiter).

[choesw]

Thank you for your answer. I'll be looking forward to new features in v7 :)