component tests for load balancer

[prasadtalasila]

#242 The following tests are advisable.

1. Multiplexing of submit evaluation request from main server and send scores request from execution node.
2. Repeat above test for high load scenario.

We would like the load balancer to evenly balance the input and output requests. The load balancer should take minimal time for processing both the input and output requests.

Q) Would a messaging solution such as zeroMQ deal with the problem better?

[prasadtalasila]

List of Tests:

1. Correctly maintains the list of execution nodes during status check.
2. Verifies the authenticity of each execution node before adding it to the worker pool.
3. Accept requests / results only from authorized execution nodes.
4. Verifies the origins of each evaluation result to make sure that the request comes from main server only.
5. Remains online and active when database is offline.
6. Connects to database when database comes back online.
7. Gives sensible response when GitLab is not available for backup of user repositories. Does not fail in this case.
8. Correctly maintains the list of execution nodes during errors in neighbouring components.
   1. MySQL DB down
   2. A few execution nodes down
   3. All the execution nodes are down
9. Sets aside execution nodes that are offline.
10. Submit an evaluation request and receive an evaluation result.
    1. valid case
    2. Correctly handles results json with large logs of 50000 lines.
    3. Ignores invalid messages from execution nodes.
    4. Invalid cases - userid invalid / repository invalid / commit hash invalid / language invalid
11. Correctly maintains the list of execution nodes during concurrent return of results.
12. Incoming requests are distributed uniformly among all execution nodes.
13. A user cancelled evaluation request is removed from job queue.
14. Informs execution node about a cancelled job.
15. Uses a valid SSL certificate.