enhance scheduling to act more predictably

[erikdw]

We should have an option to force scheduling onto the number of workers requested (Config: "topology.workers"). That will prevent various pathological scenarios where we cannot prevent topologies from becoming unbalanced with respect to the storm executors per storm worker process.

For example, say you have requested 10 workers and you have 50 executors, thus desiring 5 executors per worker. When you first submit let's assume 10 workers are able to be launched. Then later 2 workers die. During the subsequent attempt to relaunch those 2 workers, let's say the Nimbus only gets enough resources to schedule 1 replacement worker. That would cause 10 executors to be put onto the 9th worker, making it unbalanced as compared to the other 8 workers that have 5 executors. Under that scenario, the only ways to fix the balancing require manual intervention:

1. running a "rebalance" through the Storm nimbus
2. killing that one worker with 10 executors, and hoping we have enough resources now to schedule onto 2 workers
3. killing & restarting the topology