Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enabling retries for Beats flaky tests #39174

Merged
merged 7 commits into from May 1, 2024
Merged

Enabling retries for Beats flaky tests #39174

merged 7 commits into from May 1, 2024

Conversation

oakrizan
Copy link
Contributor

@oakrizan oakrizan commented Apr 24, 2024 •
edited

Proposed commit message

In scope of beats migration from Jenkins to Buildkite - added retries to tests (Cloud tests not retried at the moment, see comment below).
Packaging, K8Tests are not retried according to global Jenkinsfile.
BK Builds: https://buildkite.com/elastic/beats/builds?branch=oakrizan%3Aenable-retries-for-flaky-tests

Auditbeat MacOS test failure not related/caused by current PR. Related issue: https://github.com/elastic/ingest-dev/issues/3268

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@oakrizan oakrizan added backport-v7.17.0 Automated backport with mergify backport-v8.13.0 Automated backport with mergify Team:Ingest-EngProd backport-v8.14.0 Automated backport with mergify labels Apr 24, 2024
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 24, 2024
@oakrizan oakrizan added :Windows macOS Enable builds in the CI for darwin testing arm Enable builds in the CI for ARM testing labels Apr 24, 2024
@elasticmachine
Copy link
Collaborator

elasticmachine commented Apr 24, 2024 •
edited

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2024-05-01T16:52:13.394+0000

  • Duration: 14 min 45 sec

Test stats 🧪

Test Results
Failed 0
Passed 2
Skipped 0
Total 2

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@oakrizan oakrizan force-pushed the enable-retries-for-flaky-tests branch 2 times, most recently from 1710f61 to 45202b2 Compare April 24, 2024 10:59
@oakrizan oakrizan force-pushed the enable-retries-for-flaky-tests branch from 26e67cc to cba2948 Compare April 24, 2024 14:06
@oakrizan
Copy link
Contributor Author

As per global Jenkinsfile Cloud tests are not retried. But there is no context provided why it's failure is legit, as well as not clear if that statement is still actual. As of now Cloud tests won't be retried for x-pack/filebeat & x-pack/metricbeat to align with Jenkins, but it probably needs to be revised later. cc: @dliappis @alexsapran

@oakrizan oakrizan force-pushed the enable-retries-for-flaky-tests branch 3 times, most recently from 2d3194c to 2e760a5 Compare April 26, 2024 08:09
@oakrizan oakrizan marked this pull request as ready for review April 26, 2024 08:09
@oakrizan oakrizan requested a review from a team as a code owner April 26, 2024 08:09
@oakrizan oakrizan force-pushed the enable-retries-for-flaky-tests branch from 2e760a5 to 75843bb Compare April 26, 2024 09:14
@alexsapran
Copy link
Contributor

Do I understand the PR correctly and you are adding retries to every step? (besides the ones you mentioned explicitly you are not)

@oakrizan oakrizan force-pushed the enable-retries-for-flaky-tests branch from 75843bb to 49102e9 Compare April 26, 2024 13:15
@oakrizan
Copy link
Contributor Author

Do I understand the PR correctly and you are adding retries to every step? (besides the ones you mentioned explicitly you are not)

I have added retries to same steps as it is in Jenkins.

@oakrizan oakrizan force-pushed the enable-retries-for-flaky-tests branch 3 times, most recently from 4f31e21 to 6e71cc6 Compare April 27, 2024 06:29
dliappis
dliappis reviewed May 1, 2024
View reviewed changes
Copy link
Contributor

@dliappis dliappis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that we aren't touching/adding retries to the deploy-k8s-pipeline, is there a reason for that?
Also left a few minor questions. Otherwise this looks ready to be shipped.

.buildkite/x-pack/pipeline.xpack.filebeat.yml Show resolved Hide resolved
.buildkite/x-pack/pipeline.xpack.metricbeat.yml Show resolved Hide resolved
@oakrizan oakrizan force-pushed the enable-retries-for-flaky-tests branch from c2e8f04 to 0f9fbb2 Compare May 1, 2024 09:01
@dliappis dliappis self-requested a review May 1, 2024 09:22
dliappis
dliappis approved these changes May 1, 2024
View reviewed changes
Copy link
Contributor

@dliappis dliappis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@oakrizan
Copy link
Contributor Author

oakrizan commented May 1, 2024

/test

@oakrizan oakrizan force-pushed the enable-retries-for-flaky-tests branch from 0f9fbb2 to 4aae98c Compare May 1, 2024 13:08
@oakrizan oakrizan requested a review from a team as a code owner May 1, 2024 14:37
dliappis
dliappis reviewed May 1, 2024
View reviewed changes
dev-tools/mage/pytest.go Outdated Show resolved Hide resolved
@oakrizan oakrizan force-pushed the enable-retries-for-flaky-tests branch from d630cb4 to 10fda65 Compare May 1, 2024 16:51
@oakrizan oakrizan merged commit 02ea29d into elastic:main May 1, 2024
168 of 171 checks passed
mergify bot pushed a commit that referenced this pull request May 1, 2024
@oakrizan @mergify
Enabling retries for Beats flaky tests (#39174)
9de03e0 
* added retries for auditbeat flaky tests

* added retries for filebeat flaky tests

* added retries

* test exitcode

* checged exit status code for retries

* set larger timeout for pytestOpts

* restored timeout for pytests

(cherry picked from commit 02ea29d)
mergify bot pushed a commit that referenced this pull request May 1, 2024
@oakrizan @mergify
Enabling retries for Beats flaky tests (#39174)
b4bc58e 
* added retries for auditbeat flaky tests

* added retries for filebeat flaky tests

* added retries

* test exitcode

* checged exit status code for retries

* set larger timeout for pytestOpts

* restored timeout for pytests

(cherry picked from commit 02ea29d)

# Conflicts:
#	.buildkite/packetbeat/pipeline.packetbeat.yml
#	.buildkite/winlogbeat/pipeline.winlogbeat.yml
#	.buildkite/x-pack/pipeline.xpack.heartbeat.yml
#	.buildkite/x-pack/pipeline.xpack.libbeat.yml
#	.buildkite/x-pack/pipeline.xpack.osquerybeat.yml
#	.buildkite/x-pack/pipeline.xpack.packetbeat.yml
#	.buildkite/x-pack/pipeline.xpack.winlogbeat.yml
This was referenced May 1, 2024
Merged
Merged
mergify bot pushed a commit that referenced this pull request May 1, 2024
@oakrizan @mergify
Enabling retries for Beats flaky tests (#39174)
0fd5c1e 
* added retries for auditbeat flaky tests

* added retries for filebeat flaky tests

* added retries

* test exitcode

* checged exit status code for retries

* set larger timeout for pytestOpts

* restored timeout for pytests

(cherry picked from commit 02ea29d)
@mergify mergify bot mentioned this pull request May 1, 2024
Merged
6 tasks
dliappis pushed a commit to oakrizan/beats that referenced this pull request May 2, 2024
@oakrizan @dliappis
Enabling retries for Beats flaky tests (elastic#39174)
e985f41
dliappis added a commit to oakrizan/beats that referenced this pull request May 2, 2024
@dliappis
Revert "Enabling retries for Beats flaky tests (elastic#39174)"
7897953 
This reverts commit e985f41.
dliappis pushed a commit that referenced this pull request May 2, 2024
@mergify @oakrizan
Enabling retries for Beats flaky tests (#39174) (#39351)
c63692b 
(cherry picked from commit 02ea29d)

Co-authored-by: Olga Naydyonock <olga.naidjonoka@elastic.co>
dliappis pushed a commit that referenced this pull request May 2, 2024
@mergify @oakrizan
Enabling retries for Beats flaky tests (#39174) (#39350)
6da9e95 
(cherry picked from commit 02ea29d)

Co-authored-by: Olga Naydyonock <olga.naidjonoka@elastic.co>
dliappis added a commit that referenced this pull request May 2, 2024
@mergify @oakrizan @dliappis
[7.17](backport #39174) Enabling retries for Beats flaky tests (#39349)
65ac3af 
(cherry picked from commit 02ea29d)

---------

Co-authored-by: Olga Naydyonock <olga.naidjonoka@elastic.co>
Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dliappis dliappis dliappis approved these changes

@alexsapran alexsapran Awaiting requested review from alexsapran

@pazone pazone Awaiting requested review from pazone

@AndersonQ AndersonQ Awaiting requested review from AndersonQ AndersonQ was automatically assigned from elastic/elastic-agent-data-plane

@leehinman leehinman Awaiting requested review from leehinman leehinman was automatically assigned from elastic/elastic-agent-data-plane

Assignees

@oakrizan oakrizan

Labels
arm Enable builds in the CI for ARM testing backport-v7.17.0 Automated backport with mergify backport-v8.13.0 Automated backport with mergify backport-v8.14.0 Automated backport with mergify macOS Enable builds in the CI for darwin testing Team:Ingest-EngProd :Windows
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@oakrizan @elasticmachine @alexsapran @dliappis