Update xpack/filebeat & xpack/metricbeat pipelines to match Jenkins steps

[oakrizan]

Proposed commit message

In scope of https://github.com/elastic/ingest-dev/issues/3254 - updated x-pack/filebeat and x-pack/metricbeat pipelines to match same steps & commands as in Jenkins.

Jenkins vs. Buildkite comparison

• x-pack/filebeat: https://gist.github.com/oakrizan/0ef4a4df595547489538cbbc0ad1a4e5
• x-pack/metricbeat: https://gist.github.com/oakrizan/97eabb0635ad7727ae2b37e3f8469ff8

BK builds:

• x-pack/filebeat: https://buildkite.com/elastic/beats-xpack-filebeat/builds?branch=oakrizan%3Axpack-filebeat-fixes
• x-pack/metricbeat: https://buildkite.com/elastic/beats-xpack-metricbeat/builds?branch=oakrizan%3Axpack-filebeat-fixes

For both pipelines updated cloud test environment setup script and test commands to match Jenkins.

x-pack/filebeat: Cloud (MODULE) Tests failure is not related to current PR changes.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @oakrizan? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2024-05-09T11:57:24.624+0000

• Duration: 12 min 13 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2
Skipped	0
Total	2

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[oakrizan]

/test

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b xpack-filebeat-fixes upstream/xpack-filebeat-fixes
git merge upstream/main
git push upstream xpack-filebeat-fixes

[dliappis]

Thanks! I'd really like to avoid adding a new script (aws_cloud_tests.sh) and given that it does very little different to the existing one, requested a change/left a suggestion

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b xpack-filebeat-fixes upstream/xpack-filebeat-fixes
git merge upstream/main
git push upstream xpack-filebeat-fixes