Update auditbeat pipeline to match Jenkins steps

[oakrizan]

Proposed commit message

In scope of https://github.com/elastic/ingest-dev/issues/3254 - updated auditbeat pipeline to match same steps & commands as in Jenkins.
Jenkins vs. Buildkite comparison: https://gist.github.com/oakrizan/3af834fc51eabf9b9b8b910133886320
BK builds: https://buildkite.com/elastic/auditbeat/builds?branch=oakrizan%3Aauditbeat-postmigration-fixes

MacOS test failure is not related to current PR (more details in gist).
MacOS TMP folder error issue - https://github.com/elastic/ingest-dev/issues/3268
Intgration test timeout error issue - https://github.com/elastic/ingest-dev/issues/3270

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 14 min 42 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[dliappis]

@oakrizan as you're on PTO, I updated this PR with some minimal comments/changes and re:

1. MacOS TMP folder error issue - https://github.com/elastic/ingest-dev/issues/3268
2. Integration test timeout error issue - https://github.com/elastic/ingest-dev/issues/3270

I've reverted muting the tests for both as it should be done after consulting the corresponding teams.

For 2. @pkoutsovasilis raised a fix, which we tested with this PR by cherry-picking his fix in 907f94a . With that, the tests succeeded: https://buildkite.com/elastic/auditbeat/builds/4581#018f3896-f533-4678-af93-acac60fc452b/147-150 . I will revert the commit before merging, as it should be fixed via a separate PR that @pkoutsovasilis will raise.

For 1. I raised the attention on https://github.com/elastic/ingest-dev/issues/3270 which will be looked at by the corresponding team.

[dliappis]

For 1. I raised the attention on elastic/ingest-dev#3270 which will be looked at by the corresponding team.

@pkoutsovasilis also raised a fix for this pkoutsovasilis@eabfda2 which appears to have resolved the macOS failures: https://buildkite.com/elastic/auditbeat/builds/4593#018f3900-f651-42e4-bc2f-d62a14092e22

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b auditbeat-postmigration-fixes upstream/auditbeat-postmigration-fixes
git merge upstream/main
git push upstream auditbeat-postmigration-fixes

[dliappis]

With the fixes for macOS and auditbeat integration tests by @pkoutsovasilis cherry picked, all tests (using the labels macOS, :integration, :Windows, arm) are successful[^1]

https://buildkite.com/elastic/beats/builds/6929

[^1] except Linux packaging for x-pack/auditbeat which failed due to a timeout

[dliappis]

Now that the test fixes have been raised in separate PRs: #39362 / #39361, I've reverted the cherry picks #39245 (commits) so that this is a clean PR related to Buildkite auditbeat fixes only.

Once (basic) CI is green, I'll merge it and begin the backport process.

[dliappis]

LGTM based on #39245 (comment)