Unhandled exception while running action_probe_http.Kill

[dgzlopes]

Describe the bug
Unhandled exception while running action_probe_http.Kill.

To Reproduce

I'm not sure. I get this a couple of times when I run Powerfulseal inside k8s. Then... It disappears.

But I think we should handle it gracefully :)

➜  ~ kubectl --namespace powerfulseal logs powerfulseal-5dc57d7f76-5w77g
2020-07-08 20:33:35 INFO __main__ verbosity: None; log level: INFO; handler level: INFO
2020-07-08 20:33:35 INFO __main__ Creating kubernetes client with in-cluster config
2020-07-08 20:33:35 INFO k8s_client Initializing with config: None
2020-07-08 20:33:35 INFO __main__ No cloud driver - some functionality disabled
2020-07-08 20:33:35 INFO __main__ Using stdout metrics collector
2020-07-08 20:33:35 INFO __main__ Starting the UI server (0.0.0.0:8000)
2020-07-08 20:33:35 INFO __main__ STARTING AUTONOMOUS MODE
2020-07-08 20:33:36 INFO scenario.Kill one nginx  Starting scenario 'Kill one nginx pod in powerfulseal-sandbox, make sure the service responds' (2 steps)
2020-07-08 20:33:36 INFO action_nodes_pods.Kill one nginx  Matching 'labels' {'labels': {'namespace': 'powerfulseal-sandbox', 'selector': 'app=nginx'}}
2020-07-08 20:33:36 INFO action_nodes_pods.Kill one nginx  Matched 2 pods for selector app=nginx in namespace powerfulseal-sandbox
2020-07-08 20:33:36 INFO action_nodes_pods.Kill one nginx  Initial set length: 2
2020-07-08 20:33:36 INFO action_nodes_pods.Kill one nginx  Filtered set length: 1
2020-07-08 20:33:36 INFO action_nodes_pods.Kill one nginx  Pod killed: [pod #1 name=nginx-deployment-56db997f77-d8spk namespace=powerfulseal-sandbox containers=1 ip=172.18.0.4 host_ip=172.17.0.3 state=Running labels:app=nginx,pod-template-hash=56db997f77 annotations:]
2020-07-08 20:33:36 INFO action_probe_http.Kill one nginx  Making a call: http://10.106.207.25:80/, get, {}, 1000, 200, , , True
2020-07-08 20:33:36 ERROR action_probe_http.Kill one nginx  Exception while calling http://10.106.207.25:80/
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1354, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1354, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/powerfulseal/policy/action_probe_http.py", line 92, in make_call
    verify=verify,
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2020-07-08 20:33:36 ERROR action_probe_http.Kill one nginx  No more retries allowed. Failing step
2020-07-08 20:33:36 WARNING scenario.Kill one nginx  Step returned failure {'probeHTTP': {'endpoint': '/', 'target': {'service': {'name': 'my-service', 'namespace': 'powerfulseal-sandbox', 'port': 80}}}}. Finishing scenario early
2020-07-08 20:33:36 ERROR policy_runner Scenario failed, reporting and carrying on
2020-07-08 20:33:36 INFO policy_runner Sleeping for 215 seconds

Environment (please complete the following information):

➜  powerfulseal git:(master) ✗ powerfulseal -V
powerfulseal 3.0.0rc9

kubernetes-version v1.14.10