support to change maxWorkers dynamically

[YueHonghui]

We really want the feature to change maxWorkers dynamically. Any plan to support this feature? I try to implement a SetMaxWorkers for WorkerPool but find that the implementation of Pause rely on a static value of maxWorkers which make the situation complicated.

[gammazero]

I had not had any plan to support changing may workers dynamically, as that is not a typical use case. I am assuming that your use case is to:

• Increase the available workers when too many tasks are queued, or when other parts of you application are less busy?
• Decrease max workers when other parts of the app need more resources, and all workers in the pool are in use?

Even if Pause were not a problem, changing max workers will require additional synchronization, and more logic in processWaitingQueue. These are things I would like to avoid unless absolutely necessary. Here are some other possible solutions:

• Consider creating a pool of workerpools, and sending a task to any member of the pool. When you want fewer max workers, remove a workerpool and stop it, letting it complete any queued tasks. When needing more workers, add a workerpool.
• Consider "freezing" some number of workers by having them run a function that waits on a channel. When more workers are needed, send a message to the channel, one for each worker to unfreeze.

If non of those work, let me know and provide more detail about your use case.

[hut8]

I think this is a useful feature. Let's say you're trying to download thousands of files via HTTP. If you just download one at a time, it's unlikely you will be able to utilize all your bandwidth, plus one slow server will slow down everything else in the process. Therefore an unbounded, concurrent worker queue like your (amazing) library is what we need.

At the other end (running them all concurrently), things are less simple. If you download all 1000 of them concurrently, then there's a lot of context switching overhead, and it may take practically forever for each one to complete (if you have 100 MB/s of bandwidth and each file is 1 GB, then downloading one at a time would take 10 seconds each, but if you download 1000 concurrently, assuming each download shares an equal slice of bandwidth, it would take 27 hours each, and it's highly likely a lot of those downloads will fail and will need to be restarted. Under real conditions, the sum of the download speeds would be much lower than a few at a time.

So that leaves an important question: how many do I set? It's important because I have to set it before creating the pool 😄 , and therefore before I start the process.

Maximizing the sum of the download speeds is a real challenge, and in practice I found that under different network conditions, with different servers, I constantly have to tweak the number of downloads. Sometimes 5 is best, sometimes 20 or more. Being able to look at historical bandwidth usage and have the service respond accordingly would be super useful so I can avoid restarting the whole process 😄 Torrent clients, for example, allow you to set a maximum number of active torrents, as well as active connections. So, sort of the same thing. Originally I was just using a semaphore (having the work done in order is a plus, but if it's easier not to, no big deal) -- unfortunately the sync.WeightedSemaphore doesn't work well with dynamically changing the upper bound either.

I Googled how to do this recently ("variable size worker pool in go"). The first hit is a question I asked back in 2014, when I first learned Go, which I totally forgot about: https://stackoverflow.com/questions/23837368/idiomatic-variable-size-worker-pool-in-go

I can take a look at implementing it unless you've already started.