Context

So far, there is 3 different ways to configure the Runtime random sampling in Istio:

In order of precedence:

• telemetry.RandomSamplingPercentage https://istio.io/latest/docs/reference/config/telemetry/#Tracing
• defaultConfig.tracing.sampling https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#Tracing
• PILOT_TRACE_SAMPLING env var / values.pilot.traceSampling https://istio.io/latest/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/#customizing-trace-sampling (type in the env var name PILOT_TRACE_SAMPLE)

All these options, configure a percentage based sampling mechanism. Although this is something, more robust sampling approaches exist. For example, the OpenTelemetry project defines several of them.

In a recent PR, I introduced a feature that allows users to configure such more "advanced" samplers to the OpenTelemetryTracingProvider. This now means we have yet another option to configure sampling.

One of the discussion in the PR was that we should clarify and demonstrate to users all this options and how one affect each other. This issue is to then track this documentation work.

How to improve it

Option 1:

We have this page Customizing Trace sampling which we show some of the options. We could simply expand that there and show all options and how one affect the other (which ones has precedence)

Pros:

• No need for a new page

Cons:

• Will talk about the OpenTelemetryTracingProvider although that's not related to that page. So it's a bit out of context

Option 2:

Introduce a new page under Distributed Tracing tasks specific for sampling. There we can document all the ways:

1. New option for the OpenTelemetryTracingProvider with custom samplers: Takes precedence of all others and override Runtime random sampling to 100%. This is so all spans arrive at the sampler for its decision.
2. defaultConfig.tracing.sampling default for everything
3. telemetry.RandomSamplingPercentage can be changed per namespace
4. PILOT_TRACE_SAMPLING is this still even worth documenting? The page here says it's discouraged https://istio.io/latest/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/#customizing-trace-sampling

I personally prefer option 2, as we can have examples and really drill down on the explanations, which I think will be very beneficial. When I started working with Istio I was very confused with all these different ways of setting this, and I didn't find any place that explained it.

CC @zirain @howardjohn