Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI Failure (key symptom) in OMBValidationTest.test_common_workload #18419

Open
vbotbuildovich opened this issue May 12, 2024 · 0 comments
Open
Labels
auto-triaged used to know which issues have been opened from a CI job ci-failure ci-rca/infra CI Root Cause Analysis - Infrastructure Issue

Comments

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented May 12, 2024

https://buildkite.com/redpanda/vtools/builds/13658

Module: rptest.redpanda_cloud_tests.omb_validation_test
Class: OMBValidationTest
Method: test_common_workload
test_id:    OMBValidationTest.test_common_workload
status:     FAIL
run time:   841.310 seconds

RemoteCommandError({'ssh_config': {'host': 'ducktape-node-6-hugely-cool-ant', 'hostname': '10.138.0.74', 'user': 'root', 'port': 22, 'password': None, 'identityfile': '/home/ubuntu/.ssh/id_rsa'}, 'hostname': 'ducktape-node-6-hugely-cool-ant', 'ssh_hostname': '10.138.0.74', 'user': 'root', 'externally_routable_ip': '35.203.135.201', '_logger': <Logger rptest.redpanda_cloud_tests.omb_validation_test.OMBValidationTest.test_common_workload-2 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7a2afea9a530>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7a2afea98100>, '_custom_ssh_exception_checks': None}, 'kill -15 3909\n', 1, b'bash: line 1: kill: (3909) - No such process\n')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 103, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/omb_validation_test.py", line 512, in test_common_workload
    self.run_benchmark_with_retries(benchmark, max_retries)
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/omb_validation_test.py", line 93, in run_benchmark_with_retries
    self.handle_failed_attempt(try_count, max_retries, benchmark,
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/omb_validation_test.py", line 116, in handle_failed_attempt
    benchmark.clean()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/service.py", line 326, in clean
    self.clean_node(node, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark.py", line 524, in clean_node
    self.stop_node(node, allow_fail=True)
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark.py", line 513, in stop_node
    self.workers.stop()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/service.py", line 310, in stop
    self.stop_node(node, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark.py", line 129, in stop_node
    node.account.kill_process("openmessaging-benchmark",
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 430, in kill_process
    self.signal(pid, sig, allow_fail=allow_fail)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 418, in signal
    self.ssh(cmd, allow_fail=allow_fail)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 41, in wrapper
    return method(self, *args, **kwargs)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 300, in ssh
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: root@ducktape-node-6-hugely-cool-ant: Command 'kill -15 3909
' returned non-zero exit status 1. Remote error message: b'bash: line 1: kill: (3909) - No such process\n'

JIRA Link: CORE-2927

@vbotbuildovich vbotbuildovich added auto-triaged used to know which issues have been opened from a CI job ci-failure labels May 12, 2024
@piyushredpanda piyushredpanda added the ci-rca/infra CI Root Cause Analysis - Infrastructure Issue label May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-triaged used to know which issues have been opened from a CI job ci-failure ci-rca/infra CI Root Cause Analysis - Infrastructure Issue
Projects
None yet
Development

No branches or pull requests

2 participants