Extend Promscale dashboard with other useful panels

[onprem]

Describe the solution you'd like.
During benchmarking we found out that some key panels are missing from the default Promscale dashboard that can be helpful for debugging problems with ingestion rate, etc. It'd be great to have these panels to be added to the Promscale dashboard.

We want to have the following panels in the dashboard:

Promscale Ingest

• Active write requests

  sum by (app, namespace, instance) (promscale_ingest_active_write_requests{namespace="$namespace"})
• Ingest throughput (bytes)

  rate(promscale_ingest_requests_bytes_total{namespace="$namespace",type="metric"}[$__rate_interval])
• Metadata ingest rate

  rate(promscale_ingest_items_total{namespace="$namespace", kind="metadata"}[$__rate_interval])
• Ingest time lag

  timestamp(promscale_ingest_max_sent_timestamp_milliseconds{namespace="$namespace"})- (promscale_ingest_max_sent_timestamp_milliseconds{namespace="$namespace"}/1000)
• Ingest duration in seconds

  histogram_quantile(0.95, sum(rate(promscale_ingest_insert_duration_seconds_bucket{namespace="$namespace"}[$rate_interval])) by (le, kind))

Promscale Ingest Batching

• Total rows inserted per transaction
  • P90

    histogram_quantile(0.90, sum(rate(promscale_ingest_rows_per_batch_bucket{namespace="$namespace",subsystem="copier"}[$rate_interval])) by (le))
  • AVG

    sum(rate(promscale_ingest_rows_per_batch_sum{namespace="$namespace", subsystem="copier"}[$rate_interval])) / sum(rate(promscale_ingest_rows_per_batch_count{namespace="$namespace", subsystem="copier"}[$rate_interval]))
• Heatmap of total rows inserted per transaction

  sum by (app, namespace, le) ( increase(promscale_ingest_rows_per_batch_bucket{namespace="$namespace"}[$rate_interval]))
• Heatmap of number of series in a batch

  sum by (app, namespace, le) ( rate(promscale_ingest_flush_series_bucket{namespace="$namespace"}[$rate_interval]))
• Rows inserted per COPY
  • P90

    histogram_quantile(0.90, sum(rate(promscale_ingest_rows_per_insert_bucket{namespace="$namespace", subsystem="copier"}[$rate_interval])) by (le))
  • AVG

    sum(rate(promscale_ingest_rows_per_insert_sum{namespace="$namespace", subsystem="copier"}[$rate_interval])) / sum(rate(promscale_ingest_rows_per_insert_count{namespace="$namespace", subsystem="copier"}[$rate_interval]))
• Number of batches per transactions

  sum(rate(promscale_ingest_inserts_per_batch_count{namespace="$namespace"}[$rate_interval]))

[paulfantom]

My only suggestion is that we should use {namespace=~"$namespace"} (note regex).