📈 grafaman

Metrics coverage reporter for Graphite and Grafana.

💡 Idea

$ grafaman coverage \
    --grafana https://grafana.api/ --dashboard DTknF4rik \
    --graphite https://graphite.api/ \
    --metrics apps.services.awesome-service
# +-----------------------------------------+--------+
# | Metric of apps.services.awesome-service | Hits   |
# +-----------------------------------------+--------+
# | jaeger.finished_spans_sampled_n         |      0 |
# | rpc.client.success.ok.percentile.75     |      1 |
# | rpc.client.success.ok.percentile.95     |      1 |
# | rpc.client.success.ok.percentile.99     |      2 |
# | rpc.client.success.ok.percentile.999    |      1 |
# | ...                                     |    ... |
# | go.pod-5dbdcd5dbb-6z58f.threads         |      0 |
# +-----------------------------------------+--------+
# |                                   Total | 65.77% |
# +-----------------------------------------+--------+

A full description of the idea is available here.

🏆 Motivation

At Avito, we develop many services built on top of our excellent PaaS and internal modules. These services send a lot of metrics about their internal state which are then output to Grafana dashboards.

I need a tool that helps me to understand what metrics are published by services and how many of them are presented at Grafana dashboards.

🤼‍♂️ How to

Metrics coverage report

$ grafaman coverage \
    --grafana https://grafana.api/ -d DTknF4rik \
    --graphite https://graphite.api/ \
    -m apps.services.awesome-service \
    --last 24h \
    --exclude='*.max' --exclude='*.mean' --exclude='*.median' --exclude='*.min' --exclude='*.sum'

Supported environment variables:

• APP_NAME
• GRAFANA_URL
• GRAFANA_DASHBOARD
• GRAPHITE_URL
• GRAPHITE_METRICS

Supported config files by default:

• .env.paas
• app.toml

located at current working directory.

Supported output formats:

• table view
  • default
  • compact
  • compact-lite
  • markdown
  • rounded
  • unicode
• json

$ grafaman coverage ... -f json | jq
# [
#   {
#     "name": "apps.services.awesome-service.jaeger.finished_spans_sampled_n",
#     "hits": 0
#   },
#   ...
#   {
#     "name": "apps.services.awesome-service.go.pod-5dbdcd5dbb-6z58f.threads",
#     "hits": 0
#   }
# ]

• tsv

$ grafaman coverage ... -f tsv | column -t
# apps.services.awesome-service.jaeger.finished_spans_sampled_n         0
# apps.services.awesome-service.rpc.client.success.ok.percentile.75     1
# apps.services.awesome-service.rpc.client.success.ok.percentile.95     1
# apps.services.awesome-service.rpc.client.success.ok.percentile.99     2
# apps.services.awesome-service.rpc.client.success.ok.percentile.999    1
# ...                                                                 ...
# apps.services.awesome-service.go.pod-5dbdcd5dbb-6z58f.threads         0

Fetch metrics from Graphite

$ grafaman metrics --graphite https://graphite.api/ -m apps.services.awesome-service --last 24h

Fetch queries from Grafana

$ grafaman queries --grafana https://grafana.api/ -d DTknF4rik \
    -m apps.services.awesome-service \
    --sort

🧩 Installation

Homebrew

$ brew install kamilsk/tap/grafaman

Binary

$ curl -sSfL https://raw.githubusercontent.com/kamilsk/grafaman/master/bin/install | sh
# or
$ wget -qO-  https://raw.githubusercontent.com/kamilsk/grafaman/master/bin/install | sh

Don't forget about security.

Source

# use standard go tools
$ go get github.com/kamilsk/grafaman@latest
# or use egg tool
$ egg tools add github.com/kamilsk/grafaman@latest

egg is an extended go get.

Bash and Zsh completions

$ grafaman completion bash > /path/to/bash_completion.d/grafaman.sh
$ grafaman completion zsh  > /path/to/zsh-completions/_grafaman.zsh
# or autodetect
$ source <(grafaman completion)

See kubectl documentation.

🤲 Outcomes

👨‍🔬 Research

Metric index to autocomplete

• github.com/armon/go-radix
• github.com/fanyang01/radix
• github.com/gobwas/glob
• github.com/tchap/go-patricia

---

made with ❤️ for everyone