Kill all workers when main process exits in prefork model

[matusvalo]

Note: Before submitting this pull request, please review our contributing
guidelines.

Description

This PR ensures that OS will kill all remaining workers in prefork model when main process terminates. This is applicable only for linux platform. Other OS won't handle/kill orphaned workers. This PR should help also with memory leaks caused by orphained workers - see [1]. This PR needs billiard with version >= 3.6.0.0. Fixes #102.

Example:

master branch:

# Start Celery in different terminal: matus@matus-debian:~/dev/celery$ celery -A tasks worker --loglevel=INFO
matus@matus-debian:~$ ps aux | grep celery
matus       6611  9.4  1.1  57484 43796 pts/0    S+   22:58   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6613  0.0  0.8  56232 35172 pts/0    S+   22:58   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6614  0.0  0.8  56236 35052 pts/0    S+   22:58   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6615  0.0  0.8  56240 35052 pts/0    S+   22:58   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6616  0.0  0.8  56244 35056 pts/0    S+   22:58   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6618  0.0  0.0   6148   720 pts/1    S+   22:58   0:00 grep --color=auto celery
matus@matus-debian:~$ kill -KILL 6611
matus@matus-debian:~$ ps aux | grep celery
matus       6613  0.0  0.8  56232 35172 pts/0    S    22:58   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6614  0.0  0.8  56236 35052 pts/0    S    22:58   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6615  0.0  0.8  56240 35052 pts/0    S    22:58   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6616  0.0  0.8  56244 35056 pts/0    S    22:58   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6620  0.0  0.0   6148   656 pts/1    S+   22:58   0:00 grep --color=auto celery

PR branch:

# Start Celery in different terminal: matus@matus-debian:~/dev/celery$ celery -A tasks worker --loglevel=INFO
matus@matus-debian:~$ ps aux | grep celery
matus       6561  2.0  1.1  57760 44324 pts/0    S+   22:56   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6563  0.0  0.9  62772 39248 pts/0    S+   22:56   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6564  0.1  0.9  62780 39252 pts/0    S+   22:56   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6565  0.1  0.9  62784 39252 pts/0    S+   22:56   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6566  0.0  0.9  62788 39256 pts/0    S+   22:56   0:00 /home/matus/dev/celery39/bin/python /home/matus/dev/celery39/bin/celery -A tasks worker --loglevel=INFO
matus       6572  0.0  0.0   6148   720 pts/1    S+   22:56   0:00 grep --color=auto celery
matus@matus-debian:~$ kill -KILL 6561
matus@matus-debian:~$ ps aux | grep celery
matus       6575  0.0  0.0   6148   660 pts/1    S+   22:57   0:00 grep --color=auto celery

[1] https://medium.com/squad-engineering/two-years-with-celery-in-production-bug-fix-edition-22238669601d

[codecov]

Codecov Report

Merging #6942 (34746d7) into master (917088f) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #6942   +/-   ##
=======================================
  Coverage   89.24%   89.24%           
=======================================
  Files         138      138           
  Lines       16708    16716    +8     
  Branches     2116     2117    +1     
=======================================
+ Hits        14911    14919    +8     
  Misses       1574     1574           
  Partials      223      223           

Flag	Coverage Δ
unittests	89.24% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
celery/concurrency/prefork.py	97.77% <100.00%> (+0.02%)	⬆️
celery/platforms.py	97.00% <100.00%> (+0.05%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 917088f...34746d7. Read the comment docs.

[matusvalo]

Hmm unittests are failing on Windows since SIGKILL are defined only on linux. I will move the logic to platform.py and exclude windows...

[lgtm-com]

This pull request introduces 2 alerts when merging ce299b6 into 917088f - view on LGTM.com

new alerts:

• 2 for Wrong name for an argument in a call

[lgtm-com]

This pull request introduces 2 alerts when merging 4ced35b into 917088f - view on LGTM.com

new alerts:

• 2 for Wrong name for an argument in a call