Feat: Auto-label profiles based on request baggage

[bryanhuhta]

In order to support integrations with k6 load testing, we need to be able to pull key-value pairs out of the request baggage and create labels to add to the profile for that request. Instead of targeting specifically k6 baggage, I create a semi-extensible feature to attach arbitrary labels from the request baggage.

To use this feature in conjunction with k6, the LabelsFromBaggageHandler should wrap the routes that should have this auto-labeling behavior (ideally it wraps the http server mux to enable it for all handlers), like so:

http.Handle("/", otelhttp.NewHandler(http.HandlerFunc(index), "IndexHandler"))
http.Handle("/bike", otelhttp.NewHandler(http.HandlerFunc(bikeRoute), "BikeHandler"))
http.Handle("/scooter", otelhttp.NewHandler(http.HandlerFunc(scooterRoute), "ScooterHandler"))
http.Handle("/car", otelhttp.NewHandler(http.HandlerFunc(carRoute), "CarHandler"))

autoLabelHandler := pyroscope.LabelsFromBaggageHandler(http.DefaultServeMux, K6Options()...)
log.Fatal(http.ListenAndServe(":5000", autoLabelHandler))

This example adds auto-labeling to all the routes (/, /bike, /scooter, /car) for any request baggage with k6-specific members.

Of course, it's not limited to just k6 baggage, arbitrary baggage members can be filtered and transformed before applying labels using the WithFilters and WithTransforms options. For example:

http.Handle("/", otelhttp.NewHandler(http.HandlerFunc(index), "IndexHandler"))
http.Handle("/bike", otelhttp.NewHandler(http.HandlerFunc(bikeRoute), "BikeHandler"))
http.Handle("/scooter", otelhttp.NewHandler(http.HandlerFunc(scooterRoute), "ScooterHandler"))
http.Handle("/car", otelhttp.NewHandler(http.HandlerFunc(carRoute), "CarHandler"))

autoLabelHandler := pyroscope.LabelsFromBaggageHandler(http.DefaultServeMux,
	WithFilters(
		// Only accept keys of length 9 or shorter.
		func(key string) bool {
			return len(key) < 10
		}
	),
	WithTransforms(
		// Prefix all keys with "auto_".
		func(key string) string {
			return "auto_" + key
		}
	),
)
log.Fatal(http.ListenAndServe(":5000", autoLabelHandler))

[CLAassistant]

All committers have signed the CLA.

[kolesnikovae]

Nice job! I've left a number of comments, but the main suggestion is to move the feature to a submodule (e.g., contrib/k6)

[kolesnikovae]

nit: Lables() does allocate labels. Consider returning nil

[kolesnikovae]

Consider using pprof.SetGorouteLabels instead: TagWrapper and pprof.Do introduce some overhead and also cause appearance of quite noisy frames in the stack traces

[kolesnikovae]

I think that something like func(baggage.Baggage) Labels would be:

1. More flexible and extensible. The function would have more context: the caller already knows that it has to deal with the W3C baggage, there is no reason to hide the details.
2. More performant in a common case, as it requires less iterations and function calls
3. Simpler

[kolesnikovae]

I'm a little concerned about the cardinality of the labels we receive through this process. Do we have an estimate of how many unique combinations this could produce over time (e.g., tens/hundreds/thousands per hour)?

[kolesnikovae]

Should we perhaps place the newly added functionality into a submodule? For example, contrib/k6? There are a number of reasons why this could be beneficial:

1. This is an experimental feature, and the API may change. It would be better if we had clear boundaries between stable and experimental features.
2. The API surface of the main module would be cleaner: I expect that only a very small fraction of users are interested in K6Options and baggage handling.
3. We intentionally keep the number of dependencies of the main module low. otel/baggage dependency is quite heavy.

[kolesnikovae]

Another issue is that in Go opentracing we will override the goroutine labels: https://github.com/grafana/dskit/blob/01ae0599d043390ab206c46d4ab165e31c67d5ef/spanprofiler/tracer.go#L48-L52.

The opentracing API does not allow for passing context in a general case, therefore there is no a trivial solution to this (if we want to reuse the tracing instrumentation for our purposes). As a workaround, we could "re-inject" the goroutine labels, but that would require binding to runtime_getProfLabel, which is unsafe and may break at any time

[kolesnikovae]

What I would also like to learn more about and discuss in detail is the integration scenario as a whole because I definitely lack the context. I'm wondering if the use of pprof labels is required at all: my understanding is that load tests never run concurrently on the test subject system (I just can't imagine when it may make sense), therefore it might be sufficient to refer to the time range for a given test run (identified with the series labels, such as service_name + version). And if the user needs request-level profiling, it's probably better to use profiling and tracing integration instead