[8.14](backport #39126) Beats DRA pipeline #39199

mergify · 2024-04-24T15:50:43Z

Proposed commit message

Beats Packaging pipeline

Migrates packaging.groovy to buildkite.
Buildkite packaging pipeline is used to build and publish the packages for the Beats. The pipeline is triggered by a commit to the main or release branches.
The pipeline definition is located in the .buildkite/packaging.pipeline.yml

Triggers

Staging packaging DRA is triggered for the main and release branches.
Snapshot can be triggered for any branch by the /package comment in the PR. The release-manager dry-run will be used for PR builds.

Pipeline steps

Beats dashboards

Generates build/distributions/dependencies.csv and adds it to the beats-dashboards artifact. The dependencies.csv is required by the release-manager configuration

Packaging snapshot/staging

Builds the Beats packages for all supported platforms and architectures (mage package, mage ironbank)
Copies artifacts build/distributions/<beat>/ directory and adds it as an artifact. Where <beat> is the name of the beat
x-pack artifacts a also copied to build/distributions/<beat>/ directory, where <beat> is the name of the beat. For example, auditbeat, not x-pack/auditbeat

DRA publish

Downloads the artifacts from the packaging snapshot/staging step and publishes them to the Elastic DRA registry.

Snapshot publishing example (Dry run)

Beats-tester and e2e tests are not triggered

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

All Snapshot and staging artifacts are built, present, and properly transferred to the corresponding DRA steps
DRA dry run passes
All artifacts have the same size and permissions as on Jenkins. (Need to check one more time)

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

This is an automatic backport of pull request #39126 done by [Mergify](https://mergify.com).

* Packaging pipieline resource * Revetred agentbeat pipeline deletion * Cleanup * Test auditbeat packaging pipeline * Fix steps keys * Fix steps keys * Fix env vars * Fix env vars * Fix env vars * Unified artifacts dir * Implemented DRA steps * Test filebeat * Test matrix * Aligned artifacts directories * Aligned artifacts directories * Aligned artifacts directories * Aligned artifacts directories * Debug * Debug * Package all beats * Test DRA snapshot * Test DRA snapshot * Rename artifacts * fix dashboards artifacts * Cleanup * cleanup * No need to install msi tools * Apply suggestions from code review Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com> * Extraced platforms variables * Cleanup * Added RUN_SNAPSHOT condition * Cleanup * Boolean dry-run * Boolean dry-run --------- Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com> (cherry picked from commit c6444db) # Conflicts: # .buildkite/packaging.pipeline.yml

mergify · 2024-04-24T15:50:46Z

Cherry-pick of c6444db has failed:

On branch mergify/bp/8.14/pr-39126
Your branch is up to date with 'origin/8.14'.

You are currently cherry-picking commit c6444db544.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   .buildkite/packaging.pipeline.md
	new file:   .buildkite/scripts/dra.sh
	new file:   .buildkite/scripts/packaging/package-dra.sh
	new file:   .buildkite/scripts/packaging/prepare-release-manager.sh

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   .buildkite/packaging.pipeline.yml

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

elasticmachine · 2024-04-24T16:08:30Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-04-26T15:38:19.451+0000
Duration: 14 min 36 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3
Skipped	0
Total	3

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…pr-39126

This PR fixes a number of bugs from #39126

This commit fixes the error "interpolating remaining fields: Expected identifier to start with a letter, got /"

This commit fixes the release-manager error > # > java.io.FileNotFoundException: /artifacts/build/distributions/... (Permission denied) by adjusting the permissions of parent directories. Additionally there are a few logging enhancements to improve debugging, as well as the introduction of a new optional `DRA_BRANCH` environment variable, to help running and debugging via PRs. Relates elastic/ingest-dev#3095 --------- Signed-off-by: Alexandros Sapranidis <alexandros@elastic.co> Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>

…39217) Added DEV flag to snapshot builds. Specified the workflow for prepare-release-manager.sh Relates: elastic/ingest-dev#3095

pazone · 2024-04-26T15:39:21Z

@dliappis @alexsapran this one is ready

mergify · 2024-04-29T04:02:40Z

This pull request has not been merged yet. Could you please review and merge it @pazone? 🙏

dliappis

The snapshot part looks good to me based on https://github.com/elastic/ingest-dev/issues/3095#issuecomment-2082315276

I am working on a plan on how to test staging artifacts.

dliappis · 2024-04-29T09:57:01Z

.buildkite/scripts/dra.sh

+# extract the summary URL from a release manager output line like:
+# Report summary-18.22.0.html can be found at https://artifacts-staging.elastic.co/beats/18.22.0-ABCDEFGH/summary-18.22.0.html
+
+SUMMARY_URL=$(grep -E '^Report summary-.* can be found at ' rm-output.txt | grep -oP 'https://\S+' | awk '{print $1}')


we need to skip the lines below if DRY_RUN == "true"; @pazone if you are raising a fix for main, please include it here so that the build doesn't fail when running it with DRY_RUN enabled.

Fix PR is : #39240

dliappis

LGTM

mergify bot requested a review from a team as a code owner April 24, 2024 15:50

mergify bot added backport conflicts There is a conflict in the backported pull request labels Apr 24, 2024

mergify bot assigned pazone Apr 24, 2024

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 24, 2024

Merge branch '8.14' of github.com:elastic/beats into mergify/bp/8.14/…

3efb10f

…pr-39126

pazone force-pushed the mergify/bp/8.14/pr-39126 branch from 0c6b201 to 3efb10f Compare April 25, 2024 13:23

dliappis and others added 4 commits April 26, 2024 17:28

Fixes for Buildkite packaging pipeline (#39207)

634a2a6

This PR fixes a number of bugs from #39126

Fix BK DRA pipeline (#39212)

262c9a9

This commit fixes the error "interpolating remaining fields: Expected identifier to start with a letter, got /"

Added DEV flag to snapshot. prepare-release-manager.sh staging param (#…

3e10895

…39217) Added DEV flag to snapshot builds. Specified the workflow for prepare-release-manager.sh Relates: elastic/ingest-dev#3095

dliappis reviewed Apr 29, 2024

View reviewed changes

dliappis self-requested a review April 30, 2024 08:15

dliappis added the Team:Ingest-EngProd label Apr 30, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 30, 2024

dliappis approved these changes Apr 30, 2024

View reviewed changes

pazone merged commit 1ad2d38 into 8.14 Apr 30, 2024
107 of 108 checks passed

pazone deleted the mergify/bp/8.14/pr-39126 branch April 30, 2024 08:27

dliappis removed the conflicts There is a conflict in the backported pull request label May 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8.14](backport #39126) Beats DRA pipeline #39199

[8.14](backport #39126) Beats DRA pipeline #39199

mergify bot commented Apr 24, 2024

mergify bot commented Apr 24, 2024

elasticmachine commented Apr 24, 2024 •

edited

Build stats

Test stats 🧪

pazone commented Apr 26, 2024

mergify bot commented Apr 29, 2024

dliappis left a comment

dliappis Apr 29, 2024

dliappis Apr 30, 2024

dliappis left a comment

[8.14](backport #39126) Beats DRA pipeline #39199

[8.14](backport #39126) Beats DRA pipeline #39199

Conversation

mergify bot commented Apr 24, 2024

Proposed commit message

Beats Packaging pipeline

Triggers

Pipeline steps

Beats dashboards

Packaging snapshot/staging

DRA publish

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Apr 24, 2024

elasticmachine commented Apr 24, 2024 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

pazone commented Apr 26, 2024

mergify bot commented Apr 29, 2024

dliappis left a comment

Choose a reason for hiding this comment

dliappis Apr 29, 2024

Choose a reason for hiding this comment

dliappis Apr 30, 2024

Choose a reason for hiding this comment

dliappis left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 24, 2024 •

edited