feat: Add metrics, gcp logging to tokenserver scripts #1555
Conversation
This adds GCP logging JSON output formatting for logs as well as DataDog style metric reporting. New Environs: * `METRIC_API_KEY` - Optional Datadog API key * `METRIC_APP_KEY` - Optional Datadog APP key * `METRIC_HOST` - Collector Host address * `METRIC_PORT` - Collector Host port This also adds the `--human` arg if you want to use the older, human readable logs instead.
|
Why's datadog used (we switched off it a while ago)? |
|
I thought that the metric collector was datadog compliant at least? |
| # We need to reformat a few things to get the record to display correctly | ||
| # This includes "escaping" the message as well as converting the timestamp | ||
| # into a parsable format. | ||
| class GCP_JSON_Formatter(logging.Formatter): |
There was a problem hiding this comment.
I'll note google provides a google-cloud-logging package. For some reason instead of a Formatter it provides a couple different handlers such as StructuredLogHandler for outputting to stdout. I gave it a quick try and it doesn't even include the timestamp from the log record (which is optional anyway, the log capturer probably just fills one in for the time it received it in that case). Just as an FYI: I was going to suggest switching to it but it's a bit odd? I'm fine with your version (though I suppose some of the extra metadata it adds could be handy later)
{"message": "Adding node foo to service sync-1.5","severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "tokenserver.scripts.add_node"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 23, "file": "/home/pjenvey/src/moz/syncstorage-rs/tools/tokenserver/add_node.py", "function": "add_node"}, "httpRequest": {} }
{"message": "Error while adding node\nTraceback (most recent call last):\n File \"/home/pjenvey/src/moz/syncstorage-rs/tools/tokenserver/add_node.py\", line 25, in add_node\n database = Database()\n File \"/home/pjenvey/src/moz/syncstorage-rs/tools/tokenserver/database.py\", line 260, in __init__\n engine = create_engine(os.environ['SYNC_TOKENSERVER__DATABASE_URL'])\n File \"/usr/lib/python3.10/os.py\", line 680, in __getitem__\n raise KeyError(key) from None\nKeyError: 'SYNC_TOKENSERVER__DATABASE_URL'","severity": "ERROR", "logging.googleapis.com/labels": {"python_logger": "tokenserver.scripts.add_node"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 28, "file": "/home/pjenvey/src/moz/syncstorage-rs/tools/tokenserver/add_node.py", "function": "add_node"}, "httpRequest": {} }
There was a problem hiding this comment.
Yeah, I figure we can subclass or monkey patch that one at some point.
…syncstorage-rs into feat/SYNC-4259_metrics
…nto feat/SYNC-4259_metrics
| class Retry(requests.HTTPError): | ||
|
|
||
| def is_retryable(self): | ||
| self.response.status_code in [502, 503, 504] | ||
|
|
||
|
|
There was a problem hiding this comment.
Not used?
| class Retry(requests.HTTPError): | |
| def is_retryable(self): | |
| self.response.status_code in [502, 503, 504] |
There was a problem hiding this comment.
Yeah, thanks. Forgot to pull that.
…syncstorage-rs into feat/SYNC-4259_metrics
Description
This adds GCP logging JSON output formatting for logs as well as DataDog style metric reporting.
New Environs:
METRIC_HOST- Collector Host addressMETRIC_PORT- Collector Host portMETRIC_PATH- Socket path to collector (if available)This also adds the
--humanarg if you want to use the older, human readable logs instead.Issue(s)
Closes SYNC-4259