You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Running 2.9.1 (yes, I know, it's old). We have several different clusters, but don't have the same k8s deployments in each cluster. Tried to use a generic policy definition. When it is applied in a cluster which doesn't have one of the deployments, seal doesn't handle the 404 from k8s and dies. Yes, we can change the policies, but also seal shouldn't die in this scenario.
While this was with 2.9.1, I'd suspect it probably still exists. Might want to also check for similar issues with k8s queries.
Changed the actual deployment name below to protect the innocent.
2020-08-26 23:34:15 powerfulseal-6c9b9797c-4n55m powerfulseal.k8s.k8s_client[1] ERROR (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f3383e1d-2cc9-498e-90fe-8c710f799b7a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 26 Aug 2020 23:34:15 GMT', 'Content-Length': '296'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"deployments.extensions \"some-deployment-name\" not found","reason":"NotFound","details":{"name":"some-deployment-name","group":"extensions","kind":"deployments"},"code":404}
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/powerfulseal/k8s/k8s_client.py", line 128, in get_deployment
name=name,
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api/extensions_v1beta1_api.py", line 5958, in read_namespaced_deployment
(data) = self.read_namespaced_deployment_with_http_info(name, namespace, **kwargs) # noqa: E501
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api/extensions_v1beta1_api.py", line 6049, in read_namespaced_deployment_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 345, in call_api
_preload_content, _request_timeout)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
_request_timeout=_request_timeout)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 366, in request
headers=headers)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 241, in GET
query_params=query_params)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f3383e1d-2cc9-498e-90fe-8c710f799b7a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 26 Aug 2020 23:34:15 GMT', 'Content-Length': '296'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"deployments.extensions \"some-deployment-name\" not found","reason":"NotFound","details":{"name":"some-deployment-name","group":"extensions","kind":"deployments"},"code":404}
Traceback (most recent call last):
File "/usr/local/bin/powerfulseal", line 8, in <module>
sys.exit(start())
File "/usr/local/lib/python3.6/site-packages/powerfulseal/cli/__main__.py", line 684, in start
main(sys.argv[1:])
File "/usr/local/lib/python3.6/site-packages/powerfulseal/cli/__main__.py", line 637, in main
metric_collector=metric_collector
File "/usr/local/lib/python3.6/site-packages/powerfulseal/policy/policy_runner.py", line 91, in run
scenario.execute()
File "/usr/local/lib/python3.6/site-packages/powerfulseal/policy/scenario.py", line 59, in execute
initial_set = self.match()
File "/usr/local/lib/python3.6/site-packages/powerfulseal/policy/pod_scenario.py", line 50, in match
for pod in method(params):
File "/usr/local/lib/python3.6/site-packages/powerfulseal/policy/pod_scenario.py", line 74, in match_deployment
deployment_name=deployment_name,
File "/usr/local/lib/python3.6/site-packages/powerfulseal/k8s/k8s_inventory.py", line 93, in find_pods
deployment_name=deployment_name,
File "/usr/local/lib/python3.6/site-packages/powerfulseal/k8s/k8s_client.py", line 142, in list_pods
deployment = self.get_deployment(namespace, deployment_name)
File "/usr/local/lib/python3.6/site-packages/powerfulseal/k8s/k8s_client.py", line 128, in get_deployment
name=name,
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api/extensions_v1beta1_api.py", line 5958, in read_namespaced_deployment
(data) = self.read_namespaced_deployment_with_http_info(name, namespace, **kwargs) # noqa: E501
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api/extensions_v1beta1_api.py", line 6049, in read_namespaced_deployment_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 345, in call_api
_preload_content, _request_timeout)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
_request_timeout=_request_timeout)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 366, in request
headers=headers)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 241, in GET
query_params=query_params)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f3383e1d-2cc9-498e-90fe-8c710f799b7a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 26 Aug 2020 23:34:15 GMT', 'Content-Length': '296'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"deployments.extensions \"some-deployment-name\" not found","reason":"NotFound","details":{"name":"some-deployment-name","group":"extensions","kind":"deployments"},"code":404}
At this point the powerfulseal container exits and is restarted by k8s.
The text was updated successfully, but these errors were encountered:
Running 2.9.1 (yes, I know, it's old). We have several different clusters, but don't have the same k8s deployments in each cluster. Tried to use a generic policy definition. When it is applied in a cluster which doesn't have one of the deployments, seal doesn't handle the 404 from k8s and dies. Yes, we can change the policies, but also seal shouldn't die in this scenario.
While this was with 2.9.1, I'd suspect it probably still exists. Might want to also check for similar issues with k8s queries.
Changed the actual deployment name below to protect the innocent.
At this point the powerfulseal container exits and is restarted by k8s.
The text was updated successfully, but these errors were encountered: