Dynamic numproc change

[ivan1986]

This would be a great feature - command to dynamically set the numprocs variable and span or kill procrocs when it change

[wodow]

+1!

[alphatok]

+1

[krowinski]

+1

[LeoCavaille]

+1

[borfast]

Do the authors have any intention of implementing this? I'm evaluating similar tools and would like to know what to count on.

[mnaberez]

No one is currently working on it but this feature could also be implemented as a plugin.

[hasbrettholt]

+1

[blaskojan]

+1

[footplus]

👍

[jmspring]

+1

[mfrlin]

How would plugin for this look like?
From what I can tell there is no way you can dynamically add processes into configuration. The only thing I can do is set up the [program:x] with numproces set to a large number. Then I can start and stop workers via XMLRPC, but I can not start more workers than what was configured.
Is this the idea of a plugin? It would be nice to have an option to increase numproces dynamically so you would not be limited by what you set in config file.

[mnaberez]

How would plugin for this look like?
From what I can tell there is no way you can dynamically add processes into configuration.

The supervisor_twiddler plugin does not adjust numprocs but it does dynamically add and remove programs. The same techniques it uses could be applied to make a plugin that adjusts numprocs.

[ralberto]

+1

[tfe2012]

+1

[tautrimas]

+1

[bradcis]

+1

[axsapronov]

+1

[GameCharmer]

+1

[roadrunner]

+1

[wgjak47]

I write a plugin supervisor_grace to solve this program forked from supervisor_twiddler. It's need to change the config file right now. I found it is complex to change the numproc online.

[FractalizeR]

Is this now done? Or can this be done via changing config file only now?

[fpetkovski]

+1

[vahid-amintabar]

+1

[TheAliAbbasi]

+1

[nektru]

+1

[ChqThomas]

+1

[SlytherinCz]

+1

[ColaMint]

+1

[pretty]

+1

[mightydok]

+1

[rogervila]

+1

[nexhub]

+1

[HattabbI4]

+1

[XMoon]

+1

[makukha]

+1

[mmarynich]

+1

[yakiza]

+1

[remusnegrota]

+1

[dilip-grexit]

+1

[vladshut]

+1

[daymien]

+1

[trajche]

+1

[planetchili]

+1

[Qanah]

+1

[sobwoofer]

+1

[planetchili]

+1

[michaelarnauts]

Please stop this +1'ing. You're spamming 51 people every time.

Sorry for one more spam mail...

[julien6387]

I've just done this feature in my plugin:
https://supvisors.readthedocs.io/en/0.10/xml_rpc.html#supvisors.rpcinterface.RPCInterface.update_numprocs

supervisorctl update_numprocs <program_name> <value>

It works dynamically on programs and event listeners. I did not test with real FastCGI programs but it has been implemented.
if numprocs is greater than the current value, (numprocs - current) programs are added to Supervisor. If the autostart program option is set to true, the corresponding processes are then spawned.
if numprocs is lower than the current value, the processes in excess are eventually stopped and then removed from Supervisor.

Basic configuration and tests can be found here:
https://github.com/julien6387/supvisors/tree/master/supvisors/test/test_supervisor_177

It's quite more tricky than what Mike did in supervisor_twiddler and I don't believe that I have overcomplicated things.
The relevant information is not stored in the Supervisor internal options structure - unless I missed it - so I had to re-realize the ServerOptions to get it without reinventing the wheel.

The only drawback in doing this from a plugin is that an event listener cannot subscribe to the new events (PROCESS, PROCESS_ADDED, PROCESS_REMOVED) in its configuration file as these events won't be recognized by Supervisor. The listener has to subscribe to EVENT (yeah, sob).
It looks like adding new event types would require a - very simple - update of Supervisor itself. Unfortunately, I did not find a way to inject them from the plugin before the program sections are evaluated in the ServerOptions.

Mike, for future versions of Supervisors, what about adding a CUSTOM_EVENT_TYPE / PLUGIN_EVENT_TYPE (empty class CustomEventType / PluginEventType inheriting from Event) to support such feature ? Not perfect but that's a simple way to reduce the events. Ideally a plugin callback to be called before the program sections are evaluated would be nice.

[Dmytro-Shvetsov]

Hello, is anyone still working on this feature?
Or can you suggest the most appropriate workaround to provide numproc dynamically?

[GameCharmer]

@Dmytro-Shvetsov if you're running separate config files for each process, instead of using numprocs, just duplicate your configs with _1, _2, _3, etc at the end. Stop all processes that you don't need, give yourself some room for expansion. It's super poor, but if you're managing your processes outside of supervisor, it works.

[Dmytro-Shvetsov]

@GameCharmer yeah, it does sound like a hell of a workaround. However, I truly cannot estimate on what hardware I'll run my project so the number of processes need to be scaled dynamically. Can you maybe suggest alternatives to supervisor to do so, or is there like a plug-and-play plugin for supervisor?

[GameCharmer]

@Dmytro-Shvetsov You can give Twiddler a shot, but I've never tried it successfully: https://github.com/mnaberez/supervisor_twiddler

[Dmytro-Shvetsov]

@GameCharmer

Wow, the Twiddler works like a charm. I managed to get the processes spawned dynamically with the following code:

from xmlrpc import client
from supervisor.xmlrpc import SupervisorTransport

server = client.ServerProxy('http://localhost', SupervisorTransport('', '', 'unix:///home/dmytro/supervisor.sock'))

cmd = '...my-command.sh'

workers_group_name = 'worker'

for gpu_id in range(num_devices):
    proc_group_name = f'procs_{gpu_id}'
    server.twiddler.addProgramToGroup(
        workers_group_name, 
        proc_group_name, 
        {'command': cmd,
         'autostart':'false', 
         'autorestart': 'True', 
         'logfile': '%(ENV_HOME)s/logs/mlworker_%(process_num)s.log',
         'startsecs': '10', 
         'process_name': '%(host_node_name)s_%(process_num)s',
         'numprocs': '5',
        }
    )
    server.supervisor.startProcessGroup(workers_group_name)
    server.twiddler.log(f"Added process group: {proc_group_name}", "INFO")

Thank you!