You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When starting promtimer against many cbcollects, the machine can get overwhelmed causing annotation creation to fail (see the backtrace below.) Promtimer should catch this exception, back-off and retry (and log to indicate that this is happening.)
In addition it would be good to have an option to bypass annotation creation exposed as a command-line argument.
/dev/workspace/promtimer/bin/promtimer -g /usr/local/Cellar/grafana/9.1.7/bin/ -p /usr/local/Cellar/prometheus/2.39.1/bin/prometheus
using grafana home path:/usr/local/Cellar/grafana/9.1.7/bin/
starting prometheus server on 0.0.0.0:13301 against cbcollect_info_ns_1@10.57.0.109_20221101-142437/stats_snapshot; logging to .promtimer/logs/prom-10.57.0.109.log
starting prometheus server on 0.0.0.0:13302 against cbcollect_info_ns_1@10.57.0.119_20221101-142437/stats_snapshot; logging to .promtimer/logs/prom-10.57.0.119.log
....snip... (to save space -- the rest complete like these ones)
starting prometheus server on 0.0.0.0:13374 against cbcollect_info_ns_1@10.78.101.0_20221101-142437/stats_snapshot; logging to .promtimer/logs/prom-10.78.101.0.log
starting grafana server (on localhost:13300; logging to .promtimer/logs/grafana.log)
starting browser using http://localhost:13300/dashboards
Potential annotations: 34
Traceback (most recent call last):
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 941, in connect
self.sock = self._create_connection(
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/socket.py", line 845, in create_connection
raise err
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/socket.py", line 833, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/joe/dev/workspace/promtimer/bin/../promtimer/promtimer.py", line 308, in <module>
main()
File "/Users/joe/dev/workspace/promtimer/bin/../promtimer/promtimer.py", line 304, in main
annotations.get_and_create_annotations(grafana_port, stats_sources, not args.cluster)
File "/Users/joe/dev/workspace/promtimer/promtimer/annotations.py", line 314, in get_and_create_annotations
create_annotations(grafana_port, events)
File "/Users/joe/dev/workspace/promtimer/promtimer/annotations.py", line 297, in create_annotations
existing = get_existing_annotations(top_level_url)
File "/Users/joe/dev/workspace/promtimer/promtimer/annotations.py", line 95, in get_existing_annotations
response = util.execute_request(host_url, ANNOTATIONS_API_PATH, retries=5)
File "/Users/joe/dev/workspace/promtimer/promtimer/util.py", line 100, in execute_request
response = opener.open(request)
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 519, in open
response = self._open(req, data)
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1377, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 61] Connection refused>
The text was updated successfully, but these errors were encountered:
When starting
promtimer
against many cbcollects, the machine can get overwhelmed causing annotation creation to fail (see the backtrace below.) Promtimer should catch this exception, back-off and retry (and log to indicate that this is happening.)In addition it would be good to have an option to bypass annotation creation exposed as a command-line argument.
The text was updated successfully, but these errors were encountered: