feat: add support for otlp endpoints

[kruskall]

The soaktest runner sends events to the intake v2 endpoints and is able to generate load from the major agents however there's no option to send events to the otlp endpoint.

We should add support for that

[endorama]

Do we want a boolean switch (sending either Elastic APM or OTLP) or an enable switch (sending Elastic APM only or both Elastic APM and OTLP)?

In case is the first occurrence, would we prefer:

• apmsoak run --protocol=apm|otlp
• apmsoak elastic-apm and apmsoak otlp run

Does internal/soaktest support generating OTLP already? Also it seems apmsoak only sends payload through HTTP, which would leave out OTLP/gRPC.

[kruskall]

Sorry the description was unclear.

apmsoak uses a scenarios file to send load to the intake api. Each scenario can be customized by tuning specific settings:

apm-perf/internal/soaktest/scenario.go

Lines 11 to 27 in 83d2b40

	type ScenarioConfig struct {
	ProjectID string `yaml:"project_id"`
	ServerURL string `yaml:"server"`
	APIKey string `yaml:"api_key"`
	AgentName string `yaml:"agent_name"`
	AgentsReplicas int `yaml:"agent_replicas"`
	EventRate string `yaml:"event_rate"`
	RewriteIDs bool `yaml:"rewrite_ids"`
	RewriteTimestamps bool `yaml:"rewrite_timestamps"`
	RewriteServiceNames bool `yaml:"rewrite_service_names"`
	RewriteServiceNodeNames bool `yaml:"rewrite_service_node_names"`
	RewriteServiceTargetNames bool `yaml:"rewrite_service_target_names"`
	RewriteSpanNames bool `yaml:"rewrite_span_names"`
	RewriteTransactionNames bool `yaml:"rewrite_transaction_names"`
	RewriteTransactionTypes bool `yaml:"rewrite_transaction_types"`
	Headers map[string]string `yaml:"headers"`
	}

Among those there is agent_name which allows us to choose specific agents. What I had in mind was to add support for something like agent_name: otlp.

AFAIK the events are hardcoded in the repository in ndjson files so we would need to store the otlp payload but I wouldn't focus too much on that. A basic/simple example is enough. We can then iterate and use a more realistic payload. WDYT ?

[endorama]

It make sense, but were you thinking of supporting both OTLP via HTTP and gRPC? I briefly looked at the code but it seems it is HTTP oriented so supporting gRPC would require more adjustments.

The main reason I'm thinking we need gRPC too is that if the purpose of the test is to create constant load we can observe differences between HTTP and gRPC that we are probably interested in.

[kruskall]

That's a good point!

I was initially thinking of otlp via HTTP but we could add gRPC too. I think HTTP should be enough/minimum for our purpose and then we can iterate and add gRPC in a followup issue to avoid growing the scope of this issue too much. WDYT ?

[simitt]

Agree, let's start with HTTP and scope this task to OTLP over HTTP.

[endorama]

I started working on this again, following @kruskall suggestion.

[carsonip]

The initial effort in #45 using otlp/http in JSON protobuf encoding is blocked by elastic/apm-server#12636 . It may be better to evaluate alternative tools like https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen

[simitt]

#58 addressed this partially. Support for apmsoak is still missing.