Add --fail-if-warning

[fornellas]

Let's add an option --fail-if-warning, which'll look over issued warnings during test execution and fail them if warning issued. This can probably be done easily by patching Python's warning method and checking if it was called.

I see 3 cases to check for warnings:

1 - When importing test modules.
2 - When running individual tests.
3 - On shutdown.

For 1 & 2, this should be easy, not sure about 3.

An important feature of this, is to allow include / exclude paths to accept / ignore warnings. Eg: if a warning came from a third party library, there's sometimes little one can do, and we should ignore it. If warning came from the code being tested, it should surface.

[roman-rezinkin]

I would be interested in attempting to add this option. Could I be assigned?

[fornellas]

I bet @deathowl will be happy to merge a PR.

@roman-rezinkin there are some open question in this issue

• how to deal with (3).
• define an interface to define a list of modules to ignore warnings.

Do you have ideas for those?