Agentbeat breaks configuration only Metricbeat modules

[cmacknz]

There are several test failures that appear to be introduced by agentbeat. On initial investigation it appears that agentbeat has broken lightweight Metricbeat modules that only use configuration.

Sample failures:

1 error: metricset ‘mysql/performance’ not found
1 error: metricset ‘activemq/topic’ not found
1 error: metricset ‘activemq/queue’ not found

See https://github.com/elastic/beats/tree/main/metricbeat/module/mysql/performance for example there are no .go files.

The performance module is defined in

beats/metricbeat/module/mysql/module.yml

Lines 1 to 6 in 5d1b442

	dashboards:
	- id: 66881e90-0006-11e7-bf7f-c9acc3d3e306
	file: Metricbeat-mysql-overview.json
	name: mysql
	metricsets:
	- performance

With the actual implementation defined in https://github.com/elastic/beats/blob/5d1b442d4bb3698a0ab6f3283b5514ed353966b0/metricbeat/module/mysql/performance/manifest.yml#L1-L23

It seems like the module.yml file is compiled into metricbeat and the manifest.yml is no longer present at runtime.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

https://github.com/elastic/beats/tree/main/x-pack/metricbeat/module/activemq seems to be the same, I'm sure there are more of these.

[nimarezainia]

@cmacknz should we ask the other teams (integration owners) to also run full tests? i imagine they are as part of their 8.14 efforts.

[strawgate]

Are modules used with agentbeat?

[cmacknz]

@cmacknz should we ask the other teams (integration owners) to also run full tests? i imagine they are as part of their 8.14 efforts.

Automated tests found this and we should be doing full test for every release regardless of this change. I will send out a wider notice about this change to minimize the chances of things slipping through the cracks.

Are modules used with agentbeat?

It appears that parts of them are. The assets from modules (dashboards, ingest pipelines, mappings, etc) are not as those are obsoleted by integrations in agent.

However this issue has discovered that modules can define metricsets purely with configuration, and metricsets are used in agent as they are part of the metricbeat inputs in the agent policy. We need to figure out the best way to include these in agentbeat.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)