Feature: worker pooling

[dbkaplun]

Hello,

It would be great if I could create a pool of workers. That way, parallelizing hot paths could be completely abstracted using workerize.

Considerations:

• To implement worker pooling, a worker may have to know when another worker is busy, so it can handle take over handling new requests until becomes busy itself
• Since workers independently manage memory, they would all have to run the same code, and there would have to be a way to run a function on all workers simultaneously in order to share state

It will be interesting to see what we will need to do for this. Would you accept a PR?

Thanks for considering!

[TimvanScherpenzeel]

@developit recently added this gist that explores the idea of a worker pool: https://gist.github.com/developit/65a2212731f6b00a8aaa55d70c594f5c

I was wondering @developit, is this something you are planning to add to the repo at some stage or should it be seen more like a rough sketch or like a standalone extension of the library?

It looks like an attempt to solve the TODO's mentioned in workerize:

workerize/src/index.js

Lines 2 to 4 in 683631f

	/** TODO:
	* - pooling (+ load balancing by tracking # of open calls)
	* - queueing (worth it? sortof free via postMessage already)

[jzyzxx]

Hello,

It would be great if I could create a pool of workers. That way, parallelizing hot paths could be completely abstracted using workerize.

Considerations:

• To implement worker pooling, a worker may have to know when another worker is busy, so it can handle take over handling new requests until becomes busy itself
• Since workers independently manage memory, they would all have to run the same code, and there would have to be a way to run a function on all workers simultaneously in order to share state

It will be interesting to see what we will need to do for this. Would you accept a PR?

Thanks for considering!

Good idea!