vine: Possibility of the manager sending a library to all connected workers during one FunctionCall scheduling

[tphung3]

When a manager schedules a FunctionCall to a worker, it first checks the fitness of every worker to the FunctionCall (unless a short circuit condition is true which depends on the scheduling algorithm), then commits the FunctionCall to the best worker. However, the process of checking fitness may require sending a library to a worker that the FunctionCall is not necessarily scheduled to later. In the worst case, the manager can send a library to all connected workers, and the FunctionCall is scheduled to the last checked worker which happens to be the best one. So one FunctionCall now results in an overhead of N library deployments where N is the number of workers, while preferably should result in an overhead of at most 1 library deployment.

The cause is due to the process of sending a library in the checking step, which is before committing the FunctionCall to a worker.

The fix is likely to move the library sending step to the FunctionCall committing step, and the check should not send anything (read-only).

CC'ing relevant people @dthain @BarrySlyDelgado from #3780.