Only one worker process being created despite of passing workers as 3 and having simultaneous tasks to perform

[rudraksh1997]

Below is the code to check whether we can use multiple processes in luigi and workers get used to handle same level (concurrent tasks).

In the output, one can see only one worker process being created and used.

If multiprocessing works
Computation of 30, 31 should happen concurrently
Computation of 32, 33 should happen concurrently

Python version 3.7.3
OS: Windows 11 / Debian
Luigi version: 2.8.12

import datetime
import time
import luigi

class FactorialNumberTask(luigi.Task):
    number = luigi.IntParameter()
    def output(self):
        return luigi.LocalTarget(f"{self.number}.txt")
    def requires(self):
        if self.number in [32, 33]:
            return [FactorialNumberTask(number=31), FactorialNumberTask(number=30)]

        elif self.number in [30, 31]:
            return [FactorialNumberTask(number=29)]

    def run(self):
        start_time = datetime.datetime.now()
        squared = slow_computation_factorial(self.number, self.number)
        with self.output().open("w") as f:
            f.write(f"Start Time: {start_time}\n")
            f.write(str(squared) + "\n")
            end_time = datetime.datetime.now()
            f.write(f"End Time: {end_time}\n")

class FactorialNumbersPipeline(luigi.Task):
    numbers = luigi.ListParameter()
    def requires(self):
        return [FactorialNumberTask(number=n) for n in self.numbers]
    def is_complete(self):
        return self.complete()

def slow_computation_factorial(n, num):
    if n == 0:
        return 1
    else:
        output = slow_computation_factorial(n - 1, num)
        print(f"{output}_{num}_{time.time()}")
        return n * output

if __name__ == "__main__":
    a = luigi.run(
        [
            "FactorialNumbersPipeline",
            "--local-scheduler",
            "--numbers",
            "[32, 33]",
            "--workers",
            "3",
        ]
    )
    print(a)
    
    
    

Its output

DEBUG: Checking if FactorialNumbersPipeline(numbers=[32, 33]) is complete
C:\python\lib\site-packages\luigi\worker.py:409: UserWarning: Task FactorialNumbersPipeline(numbers=[32, 33]) without outputs has no custom complete() method
  is_complete = task.complete()
DEBUG: Checking if FactorialNumberTask(number=32) is complete
DEBUG: Checking if FactorialNumberTask(number=33) is complete
INFO: Informed scheduler that task   FactorialNumbersPipeline__32__33__92733131c9   has status   PENDING
DEBUG: Checking if FactorialNumberTask(number=31) is complete
DEBUG: Checking if FactorialNumberTask(number=30) is complete
INFO: Informed scheduler that task   FactorialNumberTask_33_95a3608720   has status   PENDING
DEBUG: Checking if FactorialNumberTask(number=29) is complete
INFO: Informed scheduler that task   FactorialNumberTask_30_3868c5adad   has status   PENDING
INFO: Informed scheduler that task   FactorialNumberTask_29_4a7a12c433   has status   PENDING
INFO: Informed scheduler that task   FactorialNumberTask_31_56b45475f2   has status   PENDING
INFO: Informed scheduler that task   FactorialNumberTask_32_a00ba62bb3   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 3 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 6
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: FactorialNumberTask_29_4a7a12c433 is currently run by worker Worker(salt=626167328, workers=3, host=NB090053, username=rudrakshsa, pid=30276)
1_29_1701177023.3400033
1_29_1701177023.3400033
2_29_1701177023.3400033
6_29_1701177023.3400033
24_29_1701177023.3400033
120_29_1701177023.3400033
720_29_1701177023.3400033
5040_29_1701177023.3400033
40320_29_1701177023.3400033
362880_29_1701177023.3400033
3628800_29_1701177023.3400033
39916800_29_1701177023.3400033
479001600_29_1701177023.3400033
6227020800_29_1701177023.3501592
87178291200_29_1701177023.3501592
1307674368000_29_1701177023.3501592
20922789888000_29_1701177023.3501592
355687428096000_29_1701177023.3501592
6402373705728000_29_1701177023.3501592
121645100408832000_29_1701177023.3501592
2432902008176640000_29_1701177023.3501592
51090942171709440000_29_1701177023.3501592
1124000727777607680000_29_1701177023.3501592
25852016738884976640000_29_1701177023.3501592
620448401733239439360000_29_1701177023.3536966
15511210043330985984000000_29_1701177023.3536966
403291461126605635584000000_29_1701177023.3536966
10888869450418352160768000000_29_1701177023.3536966
304888344611713860501504000000_29_1701177023.3536966
INFO: Informed scheduler that task   FactorialNumberTask_29_4a7a12c433   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 5
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 4
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: FactorialNumberTask_30_3868c5adad is currently run by worker Worker(salt=626167328, workers=3, host=NB090053, username=rudrakshsa, pid=30276)
DEBUG: FactorialNumberTask_31_56b45475f2 is currently run by worker Worker(salt=626167328, workers=3, host=NB090053, username=rudrakshsa, pid=30276)
1_30_1701177023.6997929
1_30_1701177023.6997929
2_30_1701177023.6997929
6_30_1701177023.6997929
24_30_1701177023.6997929
120_30_1701177023.6997929
720_30_1701177023.6997929
5040_30_1701177023.6997929
40320_30_1701177023.6997929
362880_30_1701177023.6997929
3628800_30_1701177023.7042098
39916800_30_1701177023.7042098
479001600_30_1701177023.7042098
6227020800_30_1701177023.7042098
87178291200_30_1701177023.7042098
1307674368000_30_1701177023.7042098
20922789888000_30_1701177023.7042098
355687428096000_30_1701177023.7042098
6402373705728000_30_1701177023.7052207
121645100408832000_30_1701177023.7052207
2432902008176640000_30_1701177023.7052207
51090942171709440000_30_1701177023.7052207
1124000727777607680000_30_1701177023.7052207
25852016738884976640000_30_1701177023.7052207
620448401733239439360000_30_1701177023.7052207
15511210043330985984000000_30_1701177023.706222
403291461126605635584000000_30_1701177023.706222
10888869450418352160768000000_30_1701177023.706222
304888344611713860501504000000_30_1701177023.706222
8841761993739701954543616000000_30_1701177023.706222
INFO: Informed scheduler that task   FactorialNumberTask_30_3868c5adad   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: FactorialNumberTask_31_56b45475f2 is currently run by worker Worker(salt=626167328, workers=3, host=NB090053, username=rudrakshsa, pid=30276)
1_31_1701177023.8399456
1_31_1701177023.8399456
2_31_1701177023.8399456
6_31_1701177023.8399456
24_31_1701177023.8399456
120_31_1701177023.8399456
720_31_1701177023.8399456
5040_31_1701177023.8399456
40320_31_1701177023.8399456
362880_31_1701177023.8399456
3628800_31_1701177023.8399456
39916800_31_1701177023.8399456
479001600_31_1701177023.8399456
6227020800_31_1701177023.8399456
87178291200_31_1701177023.8399456
1307674368000_31_1701177023.8399456
20922789888000_31_1701177023.8399456
355687428096000_31_1701177023.8399456
6402373705728000_31_1701177023.850162
121645100408832000_31_1701177023.850162
2432902008176640000_31_1701177023.850162
51090942171709440000_31_1701177023.850162
1124000727777607680000_31_1701177023.850162
25852016738884976640000_31_1701177023.850162
620448401733239439360000_31_1701177023.850162
15511210043330985984000000_31_1701177023.850162
403291461126605635584000000_31_1701177023.850162
10888869450418352160768000000_31_1701177023.850162
304888344611713860501504000000_31_1701177023.850162
8841761993739701954543616000000_31_1701177023.850162
265252859812191058636308480000000_31_1701177023.850162
INFO: Informed scheduler that task   FactorialNumberTask_31_56b45475f2   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 3
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 2
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: FactorialNumberTask_33_95a3608720 is currently run by worker Worker(salt=626167328, workers=3, host=NB090053, username=rudrakshsa, pid=30276)
DEBUG: FactorialNumberTask_32_a00ba62bb3 is currently run by worker Worker(salt=626167328, workers=3, host=NB090053, username=rudrakshsa, pid=30276)
1_33_1701177024.229543
1_33_1701177024.229543
2_33_1701177024.2300498
6_33_1701177024.2300498
24_33_1701177024.2300498
120_33_1701177024.2300498
720_33_1701177024.2300498
5040_33_1701177024.2300498
40320_33_1701177024.2300498
362880_33_1701177024.2300498
3628800_33_1701177024.2300498
39916800_33_1701177024.2300498
479001600_33_1701177024.2300498
6227020800_33_1701177024.2300498
87178291200_33_1701177024.2300498
1307674368000_33_1701177024.2300498
20922789888000_33_1701177024.2300498
355687428096000_33_1701177024.2300498
6402373705728000_33_1701177024.2300498
121645100408832000_33_1701177024.2300498
2432902008176640000_33_1701177024.2300498
51090942171709440000_33_1701177024.2300498
1124000727777607680000_33_1701177024.2300498
25852016738884976640000_33_1701177024.2300498
620448401733239439360000_33_1701177024.2300498
15511210043330985984000000_33_1701177024.2300498
403291461126605635584000000_33_1701177024.2300498
10888869450418352160768000000_33_1701177024.2300498
304888344611713860501504000000_33_1701177024.2300498
8841761993739701954543616000000_33_1701177024.2300498
265252859812191058636308480000000_33_1701177024.2300498
8222838654177922817725562880000000_33_1701177024.2300498
263130836933693530167218012160000000_33_1701177024.2300498
INFO: Informed scheduler that task   FactorialNumberTask_33_95a3608720   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: FactorialNumberTask_32_a00ba62bb3 is currently run by worker Worker(salt=626167328, workers=3, host=NB090053, username=rudrakshsa, pid=30276)
1_32_1701177024.384904
1_32_1701177024.384904
2_32_1701177024.3870583
6_32_1701177024.3870583
24_32_1701177024.3870583
120_32_1701177024.3870583
720_32_1701177024.3870583
5040_32_1701177024.3870583
40320_32_1701177024.3870583
362880_32_1701177024.3870583
3628800_32_1701177024.3880663
39916800_32_1701177024.3880663
479001600_32_1701177024.3880663
6227020800_32_1701177024.3880663
87178291200_32_1701177024.3880663
1307674368000_32_1701177024.3880663
20922789888000_32_1701177024.3880663
355687428096000_32_1701177024.3880663
6402373705728000_32_1701177024.3889978
121645100408832000_32_1701177024.3889978
2432902008176640000_32_1701177024.3889978
51090942171709440000_32_1701177024.3889978
1124000727777607680000_32_1701177024.3889978
25852016738884976640000_32_1701177024.3889978
620448401733239439360000_32_1701177024.3889978
15511210043330985984000000_32_1701177024.3889978
403291461126605635584000000_32_1701177024.3889978
10888869450418352160768000000_32_1701177024.3900082
304888344611713860501504000000_32_1701177024.3900082
8841761993739701954543616000000_32_1701177024.3900082
265252859812191058636308480000000_32_1701177024.3900082
8222838654177922817725562880000000_32_1701177024.3900082
INFO: Informed scheduler that task   FactorialNumberTask_32_a00ba62bb3   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: FactorialNumbersPipeline__32__33__92733131c9 is currently run by worker Worker(salt=626167328, workers=3, host=NB090053, username=rudrakshsa, pid=30276)
INFO: Informed scheduler that task   FactorialNumbersPipeline__32__33__92733131c9   has status   DONE
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
INFO: Worker Worker(salt=626167328, workers=3, host=NB090053, username=rudrakshsa, pid=30276) was stopped. Shutting down Keep-Alive thread
INFO:
===== Luigi Execution Summary =====

Scheduled 6 tasks of which:
* 6 ran successfully:
    - 5 FactorialNumberTask(number=29...33)
    - 1 FactorialNumbersPipeline(numbers=[32, 33])

This progress looks :) because there were no failed tasks or missing dependencies

===== Luigi Execution Summary =====

True