Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is consistently_failing #45402

Closed
can-anyscale opened this issue May 17, 2024 · 38 comments
Closed

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is consistently_failing #45402

can-anyscale opened this issue May 17, 2024 · 38 comments
Assignees
@sven1977
Labels
bug Something that is supposed to be working; but isn't ci-test flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/ ray-test-bot Issues managed by OSS test policy rllib RLlib related issues stability triage Needs triage (eg: priority, bug/not-bug, and owning component)

Comments

@can-anyscale
Copy link
Collaborator

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is consistently_failing. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4505#018f8519-116f-487c-9ce5-fc3ca791106e
- https://buildkite.com/ray-project/postmerge/builds/4505#018f84f9-fb09-4ab5-bdc1-160a45fcec68

DataCaseName-linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack-END
Managed by OSS Test Policy
@can-anyscale can-anyscale added bug Something that is supposed to be working; but isn't ci-test flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/ ray-test-bot Issues managed by OSS test policy rllib RLlib related issues stability triage Needs triage (eg: priority, bug/not-bug, and owning component) weekly-release-blocker Issues that will be blocking Ray weekly releases labels May 17, 2024
@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4509#018f8759-4a71-4479-8d1e-c855cc1f35af

@can-anyscale can-anyscale changed the title CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is consistently_failing CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is flaky May 17, 2024
@can-anyscale can-anyscale reopened this May 17, 2024
@can-anyscale
Copy link
Collaborator Author

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4505#018f8519-116f-487c-9ce5-fc3ca791106e
- https://buildkite.com/ray-project/postmerge/builds/4505#018f84f9-fb09-4ab5-bdc1-160a45fcec68

DataCaseName-linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

passing now

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4511#018f882d-4a77-41c3-b080-13cad6066e88

@can-anyscale can-anyscale reopened this May 17, 2024
@can-anyscale
Copy link
Collaborator Author

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4505#018f8519-116f-487c-9ce5-fc3ca791106e
- https://buildkite.com/ray-project/postmerge/builds/4505#018f84f9-fb09-4ab5-bdc1-160a45fcec68

DataCaseName-linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4517#018f890f-30ac-4ca4-82a0-f5b3d34f5ad8

@can-anyscale can-anyscale reopened this May 18, 2024
@can-anyscale
Copy link
Collaborator Author

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4505#018f8519-116f-487c-9ce5-fc3ca791106e
- https://buildkite.com/ray-project/postmerge/builds/4505#018f84f9-fb09-4ab5-bdc1-160a45fcec68

DataCaseName-linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

probably not a release blocker, remove the tag temporarily to generate a green build, will confirm Monday

@can-anyscale can-anyscale added release-blocker P0 Issue that blocks the release and removed weekly-release-blocker Issues that will be blocking Ray weekly releases labels May 18, 2024
@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4505#018f89a8-8f04-4364-826e-2176d7f53542

@can-anyscale can-anyscale reopened this May 18, 2024
@can-anyscale
Copy link
Collaborator Author

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4520#018f8a1c-003b-4696-8abf-d23042cb6a27
- https://buildkite.com/ray-project/postmerge/builds/4505#018f8519-116f-487c-9ce5-fc3ca791106e
- https://buildkite.com/ray-project/postmerge/builds/4505#018f84f9-fb09-4ab5-bdc1-160a45fcec68

DataCaseName-linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4622#018fbad8-433b-4c38-aa72-f64ac2919dec

@can-anyscale can-anyscale reopened this May 27, 2024
@can-anyscale
Copy link
Collaborator Author

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4613#018fad59-b263-4517-a6a8-45aadab0d627
- https://buildkite.com/ray-project/postmerge/builds/4613#018fad1c-d32e-442b-847a-950039234ed8
- https://buildkite.com/ray-project/postmerge/builds/4607#018faba1-6822-48ad-86ff-d66086da09d4
- https://buildkite.com/ray-project/postmerge/builds/4607#018fab64-50e4-4fec-bf8e-21f126570f87
- https://buildkite.com/ray-project/postmerge/builds/4598#018fa71a-1415-4d03-a5e8-4c57e3ac2ad1

DataCaseName-linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4623#018fbbb3-73c1-4c05-94a0-764d30541fea

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4623#018fbbd4-4d0e-4701-853a-f75f6eb5e482

@can-anyscale can-anyscale reopened this May 28, 2024
@can-anyscale
Copy link
Collaborator Author

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4623#018fbbb3-73be-44ac-9a36-9d228bd9008f
- https://buildkite.com/ray-project/postmerge/builds/4613#018fad59-b263-4517-a6a8-45aadab0d627
- https://buildkite.com/ray-project/postmerge/builds/4613#018fad1c-d32e-442b-847a-950039234ed8
- https://buildkite.com/ray-project/postmerge/builds/4607#018faba1-6822-48ad-86ff-d66086da09d4
- https://buildkite.com/ray-project/postmerge/builds/4607#018fab64-50e4-4fec-bf8e-21f126570f87

DataCaseName-linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack-END
Managed by OSS Test Policy

1 similar comment
@can-anyscale
Copy link
Collaborator Author

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4623#018fbbb3-73be-44ac-9a36-9d228bd9008f
- https://buildkite.com/ray-project/postmerge/builds/4613#018fad59-b263-4517-a6a8-45aadab0d627
- https://buildkite.com/ray-project/postmerge/builds/4613#018fad1c-d32e-442b-847a-950039234ed8
- https://buildkite.com/ray-project/postmerge/builds/4607#018faba1-6822-48ad-86ff-d66086da09d4
- https://buildkite.com/ray-project/postmerge/builds/4607#018fab64-50e4-4fec-bf8e-21f126570f87

DataCaseName-linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is flaky. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4631#018fbffd-e9f9-4d60-9914-8a5b7c6992ff
- https://buildkite.com/ray-project/postmerge/builds/4623#018fbbb3-73be-44ac-9a36-9d228bd9008f
- https://buildkite.com/ray-project/postmerge/builds/4613#018fad59-b263-4517-a6a8-45aadab0d627
- https://buildkite.com/ray-project/postmerge/builds/4613#018fad1c-d32e-442b-847a-950039234ed8
- https://buildkite.com/ray-project/postmerge/builds/4607#018faba1-6822-48ad-86ff-d66086da09d4

DataCaseName-linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack-END
Managed by OSS Test Policy

@can-anyscale can-anyscale added weekly-release-blocker Issues that will be blocking Ray weekly releases and removed release-blocker P0 Issue that blocks the release labels May 28, 2024
@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4649#018fc2c8-29a4-4b86-afda-f450106687b2

@can-anyscale can-anyscale removed the weekly-release-blocker Issues that will be blocking Ray weekly releases label May 29, 2024
@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4652#018fc525-397d-42df-a703-23f891661d28

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4657#018fc600-1518-4341-b437-96084f3cf8fd

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4657#018fc633-cc8b-4725-ad67-78797022c5dd

@can-anyscale can-anyscale changed the title CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is flaky CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is consistently_failing May 30, 2024
@can-anyscale can-anyscale reopened this May 30, 2024
@can-anyscale
Copy link
Collaborator Author

CI test linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack is consistently_failing. Recent failures:
- https://buildkite.com/ray-project/postmerge/builds/4663#018fc80d-6e43-43a5-ac99-51f4e8725acb
- https://buildkite.com/ray-project/postmerge/builds/4663#018fc7e9-a7f1-4e92-9ed4-4caa98ae6d95
- https://buildkite.com/ray-project/postmerge/builds/4631#018fbffd-e9f9-4d60-9914-8a5b7c6992ff

DataCaseName-linux://rllib:learning_tests_cartpole_appo_fake_gpus_old_api_stack-END
Managed by OSS Test Policy

@can-anyscale
Copy link
Collaborator Author

This test is now considered as flaky because it has been failing on postmerge for too long. Flaky tests do not run on premerge.

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4671#018fcbfe-e7d8-4d2b-ae44-cf14b5763a7d

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4672#018fcd0f-56d6-42d7-bc48-051c25ed53d1

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4675#018fcf6b-bc35-4850-874a-3cf5ff54996c

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4679#018fd047-0cab-407c-9dbc-7b8bde9e2bf5

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4683#018fd124-f33b-4440-97f3-498332c49131

@can-anyscale
Copy link
Collaborator Author

Test passed on latest run: https://buildkite.com/ray-project/postmerge/builds/4686#018fd236-56ca-48c0-8845-b2852227811b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sven1977 sven1977

Labels
bug Something that is supposed to be working; but isn't ci-test flaky-tracker Issue created via Flaky Test Tracker https://flaky-tests.ray.io/ ray-test-bot Issues managed by OSS test policy rllib RLlib related issues stability triage Needs triage (eg: priority, bug/not-bug, and owning component)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sven1977 @can-anyscale