Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add process and go runtime metrics for controller #6966

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add process and go runtime metrics for controller #6966

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
13 changes: 12 additions & 1 deletion pkg/metrics/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,12 @@ package metrics
import (
"net"
"net/http"
"regexp"
"time"

"github.com/go-logr/logr"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/collectors"
"github.com/prometheus/client_golang/prometheus/promhttp"
"k8s.io/utils/clock"

Expand Down Expand Up @@ -186,10 +188,19 @@ func New(log logr.Logger, c clock.Clock) *Metrics {
)
)

// Create Registry and register the recommended collectors
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest rewording this comment to explain why rather than what the following code does.
Perhaps:

We want to avoid using and mutating global state so we avoid
prometheus.DefaultRegisterer. Instead we create a local Registry and manually register
the same two collectors that are used by DefaultRegisterer.

Process collector

Exports the current state of process metrics including CPU, memory and file
descriptor usage as well as the process start time. The detailed behavior is
defined by the provided ProcessCollectorOpts. The zero value of
ProcessCollectorOpts creates a collector for the current process with an empty
namespace string and no error reporting.

The collector only works on operating systems with a Linux-style proc
filesystem and on Microsoft Windows. On other operating systems, it will not
collect any metrics.

Go collector

Exports metrics about the current Go process using debug.GCStats (base
metrics) and runtime/metrics.

registry := prometheus.NewRegistry()
registry.MustRegister(collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}))
registry.MustRegister(
collectors.NewGoCollector(
collectors.WithGoCollectorRuntimeMetrics(collectors.MetricsAll),
collectors.WithoutGoCollectorRuntimeMetrics(regexp.MustCompile("^/godebug/.*")),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment to the code explaining why we omit the godebug metrics.
And give an example of the metrics that will not be available.

OR

Remove that line and include all the Go metrics.

The DefaultRegisterer doesn't omit them.
Nor could I find any examples of other Go processes that omit these.

I tried removing that line and observed ~75 extra lines in the metrics.

$ kubectl get --raw /api/v1/namespaces/cert-manager/services/cert-manager:9402/proxy/metrics | wc -l
395

# Vs
$ kubectl get --raw /api/v1/namespaces/cert-manager/services/cert-manager:9402/proxy/metrics  | wc -l
470

And 25 extra metrics:

$ kubectl get --raw /api/v1/namespaces/cert-manager/services/cert-manager:9402/proxy/metrics  | grep '^go_godebug_'
go_godebug_non_default_behavior_execerrdot_events_total 0
go_godebug_non_default_behavior_gocachehash_events_total 0
go_godebug_non_default_behavior_gocachetest_events_total 0
go_godebug_non_default_behavior_gocacheverify_events_total 0
go_godebug_non_default_behavior_gotypesalias_events_total 0
go_godebug_non_default_behavior_http2client_events_total 0
go_godebug_non_default_behavior_http2server_events_total 0
go_godebug_non_default_behavior_httplaxcontentlength_events_total 0
go_godebug_non_default_behavior_httpmuxgo121_events_total 1
go_godebug_non_default_behavior_installgoroot_events_total 0
go_godebug_non_default_behavior_jstmpllitinterp_events_total 0
go_godebug_non_default_behavior_multipartmaxheaders_events_total 0
go_godebug_non_default_behavior_multipartmaxparts_events_total 0
go_godebug_non_default_behavior_multipathtcp_events_total 0
go_godebug_non_default_behavior_panicnil_events_total 0
go_godebug_non_default_behavior_randautoseed_events_total 0
go_godebug_non_default_behavior_tarinsecurepath_events_total 0
go_godebug_non_default_behavior_tls10server_events_total 0
go_godebug_non_default_behavior_tlsmaxrsasize_events_total 0
go_godebug_non_default_behavior_tlsrsakex_events_total 0
go_godebug_non_default_behavior_tlsunsafeekm_events_total 0
go_godebug_non_default_behavior_x509sha1_events_total 0
go_godebug_non_default_behavior_x509usefallbackroots_events_total 0
go_godebug_non_default_behavior_x509usepolicies_events_total 0
go_godebug_non_default_behavior_zipinsecurepath_events_total 0

$ kubectl get --raw /api/v1/namespaces/cert-manager/services/cert-manager:9402/proxy/metrics  | grep '^go_godebug_' | wc -l
25

),
)
// Create server and register Prometheus metrics handler
m := &Metrics{
log: log.WithName("metrics"),
registry: prometheus.NewRegistry(),
registry: registry,

clockTimeSeconds: clockTimeSeconds,
clockTimeSecondsGauge: clockTimeSecondsGauge,
Expand Down