Manage log streams when runnign multiple tests in parallel

[adamnovak]

I'm trying to debug Toil running the CWL conformance tests; a few random tests seem to hit their timeouts.

I'm running cwltest with -j8 or so to run multiple tests in parallel, and I've set Toil to log debug logs even with the --quiet that cwltest passes. But my terminal ends up being logs from all the simultaneously running and passing tests, and I'm never going to be able to tease out the logs specifically from the tests that time out (different tests every time) to see what is going wrong in those cases.

I need cwltest to be able to capture the output streams from the runner processes for each test, and to print out the streams for the tests that fail or time out after the failure or timeout happens, without interleaving with the streams from other simultaneous successful tests.

[adamnovak]

I think this is already possible somewhat with --junit-xml, but it would be nice to have this work at the console too, like pytest does it.

[mr-c]

Agreed @adamnovak ; speaking of pytest, have you seen #74 ?

[adamnovak]

#74 looks pretty interesting.

I tried using --junit-xml, and while it does keep the tests output straight from each other, one of my tests that timed out got no output stored in the XML; there's no system-out or system-err elements like for other tests.

This is because we're using communicate() with a timeout, but not doing the special dance from the docs to actually get the output from the process if the timeout expires.