Seal crashes if deployment not found

[jcstanaway]

Running 2.9.1 (yes, I know, it's old). We have several different clusters, but don't have the same k8s deployments in each cluster. Tried to use a generic policy definition. When it is applied in a cluster which doesn't have one of the deployments, seal doesn't handle the 404 from k8s and dies. Yes, we can change the policies, but also seal shouldn't die in this scenario.

While this was with 2.9.1, I'd suspect it probably still exists. Might want to also check for similar issues with k8s queries.

Changed the actual deployment name below to protect the innocent.

2020-08-26 23:34:15 powerfulseal-6c9b9797c-4n55m powerfulseal.k8s.k8s_client[1] ERROR (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f3383e1d-2cc9-498e-90fe-8c710f799b7a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 26 Aug 2020 23:34:15 GMT', 'Content-Length': '296'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"deployments.extensions \"some-deployment-name\" not found","reason":"NotFound","details":{"name":"some-deployment-name","group":"extensions","kind":"deployments"},"code":404}

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/powerfulseal/k8s/k8s_client.py", line 128, in get_deployment
    name=name,
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api/extensions_v1beta1_api.py", line 5958, in read_namespaced_deployment
    (data) = self.read_namespaced_deployment_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api/extensions_v1beta1_api.py", line 6049, in read_namespaced_deployment_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 345, in call_api
    _preload_content, _request_timeout)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 366, in request
    headers=headers)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 241, in GET
    query_params=query_params)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f3383e1d-2cc9-498e-90fe-8c710f799b7a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 26 Aug 2020 23:34:15 GMT', 'Content-Length': '296'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"deployments.extensions \"some-deployment-name\" not found","reason":"NotFound","details":{"name":"some-deployment-name","group":"extensions","kind":"deployments"},"code":404}


Traceback (most recent call last):
  File "/usr/local/bin/powerfulseal", line 8, in <module>
    sys.exit(start())
  File "/usr/local/lib/python3.6/site-packages/powerfulseal/cli/__main__.py", line 684, in start
    main(sys.argv[1:])
  File "/usr/local/lib/python3.6/site-packages/powerfulseal/cli/__main__.py", line 637, in main
    metric_collector=metric_collector
  File "/usr/local/lib/python3.6/site-packages/powerfulseal/policy/policy_runner.py", line 91, in run
    scenario.execute()
  File "/usr/local/lib/python3.6/site-packages/powerfulseal/policy/scenario.py", line 59, in execute
    initial_set = self.match()
  File "/usr/local/lib/python3.6/site-packages/powerfulseal/policy/pod_scenario.py", line 50, in match
    for pod in method(params):
  File "/usr/local/lib/python3.6/site-packages/powerfulseal/policy/pod_scenario.py", line 74, in match_deployment
    deployment_name=deployment_name,
  File "/usr/local/lib/python3.6/site-packages/powerfulseal/k8s/k8s_inventory.py", line 93, in find_pods
    deployment_name=deployment_name,
  File "/usr/local/lib/python3.6/site-packages/powerfulseal/k8s/k8s_client.py", line 142, in list_pods
    deployment = self.get_deployment(namespace, deployment_name)
  File "/usr/local/lib/python3.6/site-packages/powerfulseal/k8s/k8s_client.py", line 128, in get_deployment
    name=name,
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api/extensions_v1beta1_api.py", line 5958, in read_namespaced_deployment
    (data) = self.read_namespaced_deployment_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api/extensions_v1beta1_api.py", line 6049, in read_namespaced_deployment_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 345, in call_api
    _preload_content, _request_timeout)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 366, in request
    headers=headers)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 241, in GET
    query_params=query_params)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f3383e1d-2cc9-498e-90fe-8c710f799b7a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 26 Aug 2020 23:34:15 GMT', 'Content-Length': '296'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"deployments.extensions \"some-deployment-name\" not found","reason":"NotFound","details":{"name":"some-deployment-name","group":"extensions","kind":"deployments"},"code":404}

At this point the powerfulseal container exits and is restarted by k8s.

[QAInsights]

I got this same issue. Any updates on the fix please?