Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build 5 for 8.14 with status FAILURE - Compose up failed, retrying: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana #39076

Open
elasticmachine opened this issue Apr 19, 2024 · 1 comment
Open

Build 5 for 8.14 with status FAILURE - Compose up failed, retrying: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana #39076

elasticmachine opened this issue Apr 19, 2024 · 1 comment
Labels
automation build-failures Build failures in the CI. ci-reported Issues that have been automatically reported from the CI Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Comments

@elasticmachine
Copy link
Collaborator

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2024-04-19T10:12:16.081+0000

  • Duration: 112 min 40 sec

Test stats 🧪

Test Results
Failed 10
Passed 32479
Skipped 2404
Total 34893

Test errors 10

Expand to view the tests failures

Build&Test / x-pack/metricbeat-pythonIntegTest / test_dashboards – x-pack.metricbeat.tests.system.test_xpack_base.Test
    Expand to view the error details

     failed on setup with "Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana"

    Expand to view the stacktrace

     self = <class "test_xpack_base.Test">

    @classmethod
    def setUpClass(self):
        self.beat_name = "metricbeat"
        self.beat_path = os.path.abspath(
            os.path.join(os.path.dirname(__file__), "../../"))
    
        self.template_paths = [
            os.path.abspath(os.path.join(self.beat_path, "../../metricbeat")),
            os.path.abspath(os.path.join(self.beat_path, "../../libbeat")),
        ]
    
>       super(XPackTest, self).setUpClass()

tests/system/xpack_metricbeat.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../metricbeat/tests/system/metricbeat.py:42: in setUpClass
    super().setUpClass()
../../libbeat/tests/system/beat/beat.py:204: in setUpClass
    cls.compose_up_with_retries()
../../libbeat/tests/system/beat/beat.py:222: in compose_up_with_retries
    raise ex
../../libbeat/tests/system/beat/beat.py:218: in compose_up_with_retries
    cls.compose_up()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class "test_xpack_base.Test">

    @classmethod
    def compose_up(cls):
        """
        Ensure *only* the services defined under `COMPOSE_SERVICES` are running and healthy
        """
        if not INTEGRATION_TESTS or not cls.COMPOSE_SERVICES:
            return
    
        if os.environ.get("NO_COMPOSE"):
            return
    
        def print_logs(container):
            print("---- " + container.name_without_project)
            print(container.logs())
            print("----")
    
        def is_healthy(container):
            return container.inspect()["State"]["Health"]["Status"] == "healthy"
    
        project = cls.compose_project()
    
        with disabled_logger("compose.service"):
            project.pull(
                ignore_pull_failures=True,
                service_names=cls.COMPOSE_SERVICES)
    
        project.up(
            strategy=ConvergenceStrategy.always,
            service_names=cls.COMPOSE_SERVICES,
            timeout=30)
    
        # Wait for them to be healthy
        start = time.time()
        while True:
            containers = project.containers(
                service_names=cls.COMPOSE_SERVICES,
                stopped=True)
    
            healthy = True
            for container in containers:
                if not container.is_running:
                    print_logs(container)
                    raise Exception(
                        "Container %s unexpectedly finished on startup" %
                        container.name_without_project)
                if not is_healthy(container):
                    healthy = False
                    break
    
            if healthy:
                break
    
            if cls.COMPOSE_ADVERTISED_HOST:
                for service in cls.COMPOSE_SERVICES:
                    cls._setup_advertised_host(project, service)
    
            time.sleep(1)
            timeout = time.time() - start > cls.COMPOSE_TIMEOUT
            if timeout:
                for container in containers:
                    if not is_healthy(container):
                        print_logs(container)
>               raise Exception(
                    "Timeout while waiting for healthy "
                    "docker-compose services: %s" %
                    ",".join(cls.COMPOSE_SERVICES))
E               Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana

../../libbeat/tests/system/beat/compose.py:102: Exception

Build&Test / x-pack/metricbeat-pythonIntegTest / test_export_config – x-pack.metricbeat.tests.system.test_xpack_base.Test
    Expand to view the error details

     failed on setup with "Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana"

    Expand to view the stacktrace

     self = <class "test_xpack_base.Test">

    @classmethod
    def setUpClass(self):
        self.beat_name = "metricbeat"
        self.beat_path = os.path.abspath(
            os.path.join(os.path.dirname(__file__), "../../"))
    
        self.template_paths = [
            os.path.abspath(os.path.join(self.beat_path, "../../metricbeat")),
            os.path.abspath(os.path.join(self.beat_path, "../../libbeat")),
        ]
    
>       super(XPackTest, self).setUpClass()

tests/system/xpack_metricbeat.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../metricbeat/tests/system/metricbeat.py:42: in setUpClass
    super().setUpClass()
../../libbeat/tests/system/beat/beat.py:204: in setUpClass
    cls.compose_up_with_retries()
../../libbeat/tests/system/beat/beat.py:222: in compose_up_with_retries
    raise ex
../../libbeat/tests/system/beat/beat.py:218: in compose_up_with_retries
    cls.compose_up()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class "test_xpack_base.Test">

    @classmethod
    def compose_up(cls):
        """
        Ensure *only* the services defined under `COMPOSE_SERVICES` are running and healthy
        """
        if not INTEGRATION_TESTS or not cls.COMPOSE_SERVICES:
            return
    
        if os.environ.get("NO_COMPOSE"):
            return
    
        def print_logs(container):
            print("---- " + container.name_without_project)
            print(container.logs())
            print("----")
    
        def is_healthy(container):
            return container.inspect()["State"]["Health"]["Status"] == "healthy"
    
        project = cls.compose_project()
    
        with disabled_logger("compose.service"):
            project.pull(
                ignore_pull_failures=True,
                service_names=cls.COMPOSE_SERVICES)
    
        project.up(
            strategy=ConvergenceStrategy.always,
            service_names=cls.COMPOSE_SERVICES,
            timeout=30)
    
        # Wait for them to be healthy
        start = time.time()
        while True:
            containers = project.containers(
                service_names=cls.COMPOSE_SERVICES,
                stopped=True)
    
            healthy = True
            for container in containers:
                if not container.is_running:
                    print_logs(container)
                    raise Exception(
                        "Container %s unexpectedly finished on startup" %
                        container.name_without_project)
                if not is_healthy(container):
                    healthy = False
                    break
    
            if healthy:
                break
    
            if cls.COMPOSE_ADVERTISED_HOST:
                for service in cls.COMPOSE_SERVICES:
                    cls._setup_advertised_host(project, service)
    
            time.sleep(1)
            timeout = time.time() - start > cls.COMPOSE_TIMEOUT
            if timeout:
                for container in containers:
                    if not is_healthy(container):
                        print_logs(container)
>               raise Exception(
                    "Timeout while waiting for healthy "
                    "docker-compose services: %s" %
                    ",".join(cls.COMPOSE_SERVICES))
E               Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana

../../libbeat/tests/system/beat/compose.py:102: Exception

Build&Test / x-pack/metricbeat-pythonIntegTest / test_export_ilm_policy – x-pack.metricbeat.tests.system.test_xpack_base.Test
    Expand to view the error details

     failed on setup with "Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana"

    Expand to view the stacktrace

     self = <class "test_xpack_base.Test">

    @classmethod
    def setUpClass(self):
        self.beat_name = "metricbeat"
        self.beat_path = os.path.abspath(
            os.path.join(os.path.dirname(__file__), "../../"))
    
        self.template_paths = [
            os.path.abspath(os.path.join(self.beat_path, "../../metricbeat")),
            os.path.abspath(os.path.join(self.beat_path, "../../libbeat")),
        ]
    
>       super(XPackTest, self).setUpClass()

tests/system/xpack_metricbeat.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../metricbeat/tests/system/metricbeat.py:42: in setUpClass
    super().setUpClass()
../../libbeat/tests/system/beat/beat.py:204: in setUpClass
    cls.compose_up_with_retries()
../../libbeat/tests/system/beat/beat.py:222: in compose_up_with_retries
    raise ex
../../libbeat/tests/system/beat/beat.py:218: in compose_up_with_retries
    cls.compose_up()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class "test_xpack_base.Test">

    @classmethod
    def compose_up(cls):
        """
        Ensure *only* the services defined under `COMPOSE_SERVICES` are running and healthy
        """
        if not INTEGRATION_TESTS or not cls.COMPOSE_SERVICES:
            return
    
        if os.environ.get("NO_COMPOSE"):
            return
    
        def print_logs(container):
            print("---- " + container.name_without_project)
            print(container.logs())
            print("----")
    
        def is_healthy(container):
            return container.inspect()["State"]["Health"]["Status"] == "healthy"
    
        project = cls.compose_project()
    
        with disabled_logger("compose.service"):
            project.pull(
                ignore_pull_failures=True,
                service_names=cls.COMPOSE_SERVICES)
    
        project.up(
            strategy=ConvergenceStrategy.always,
            service_names=cls.COMPOSE_SERVICES,
            timeout=30)
    
        # Wait for them to be healthy
        start = time.time()
        while True:
            containers = project.containers(
                service_names=cls.COMPOSE_SERVICES,
                stopped=True)
    
            healthy = True
            for container in containers:
                if not container.is_running:
                    print_logs(container)
                    raise Exception(
                        "Container %s unexpectedly finished on startup" %
                        container.name_without_project)
                if not is_healthy(container):
                    healthy = False
                    break
    
            if healthy:
                break
    
            if cls.COMPOSE_ADVERTISED_HOST:
                for service in cls.COMPOSE_SERVICES:
                    cls._setup_advertised_host(project, service)
    
            time.sleep(1)
            timeout = time.time() - start > cls.COMPOSE_TIMEOUT
            if timeout:
                for container in containers:
                    if not is_healthy(container):
                        print_logs(container)
>               raise Exception(
                    "Timeout while waiting for healthy "
                    "docker-compose services: %s" %
                    ",".join(cls.COMPOSE_SERVICES))
E               Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana

../../libbeat/tests/system/beat/compose.py:102: Exception

Build&Test / x-pack/metricbeat-pythonIntegTest / test_export_index_pattern – x-pack.metricbeat.tests.system.test_xpack_base.Test
    Expand to view the error details

     failed on setup with "Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana"

    Expand to view the stacktrace

     self = <class "test_xpack_base.Test">

    @classmethod
    def setUpClass(self):
        self.beat_name = "metricbeat"
        self.beat_path = os.path.abspath(
            os.path.join(os.path.dirname(__file__), "../../"))
    
        self.template_paths = [
            os.path.abspath(os.path.join(self.beat_path, "../../metricbeat")),
            os.path.abspath(os.path.join(self.beat_path, "../../libbeat")),
        ]
    
>       super(XPackTest, self).setUpClass()

tests/system/xpack_metricbeat.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../metricbeat/tests/system/metricbeat.py:42: in setUpClass
    super().setUpClass()
../../libbeat/tests/system/beat/beat.py:204: in setUpClass
    cls.compose_up_with_retries()
../../libbeat/tests/system/beat/beat.py:222: in compose_up_with_retries
    raise ex
../../libbeat/tests/system/beat/beat.py:218: in compose_up_with_retries
    cls.compose_up()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class "test_xpack_base.Test">

    @classmethod
    def compose_up(cls):
        """
        Ensure *only* the services defined under `COMPOSE_SERVICES` are running and healthy
        """
        if not INTEGRATION_TESTS or not cls.COMPOSE_SERVICES:
            return
    
        if os.environ.get("NO_COMPOSE"):
            return
    
        def print_logs(container):
            print("---- " + container.name_without_project)
            print(container.logs())
            print("----")
    
        def is_healthy(container):
            return container.inspect()["State"]["Health"]["Status"] == "healthy"
    
        project = cls.compose_project()
    
        with disabled_logger("compose.service"):
            project.pull(
                ignore_pull_failures=True,
                service_names=cls.COMPOSE_SERVICES)
    
        project.up(
            strategy=ConvergenceStrategy.always,
            service_names=cls.COMPOSE_SERVICES,
            timeout=30)
    
        # Wait for them to be healthy
        start = time.time()
        while True:
            containers = project.containers(
                service_names=cls.COMPOSE_SERVICES,
                stopped=True)
    
            healthy = True
            for container in containers:
                if not container.is_running:
                    print_logs(container)
                    raise Exception(
                        "Container %s unexpectedly finished on startup" %
                        container.name_without_project)
                if not is_healthy(container):
                    healthy = False
                    break
    
            if healthy:
                break
    
            if cls.COMPOSE_ADVERTISED_HOST:
                for service in cls.COMPOSE_SERVICES:
                    cls._setup_advertised_host(project, service)
    
            time.sleep(1)
            timeout = time.time() - start > cls.COMPOSE_TIMEOUT
            if timeout:
                for container in containers:
                    if not is_healthy(container):
                        print_logs(container)
>               raise Exception(
                    "Timeout while waiting for healthy "
                    "docker-compose services: %s" %
                    ",".join(cls.COMPOSE_SERVICES))
E               Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana

../../libbeat/tests/system/beat/compose.py:102: Exception

Build&Test / x-pack/metricbeat-pythonIntegTest / test_export_index_pattern_migration – x-pack.metricbeat.tests.system.test_xpack_base.Test
    Expand to view the error details

     failed on setup with "Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana"

    Expand to view the stacktrace

     self = <class "test_xpack_base.Test">

    @classmethod
    def setUpClass(self):
        self.beat_name = "metricbeat"
        self.beat_path = os.path.abspath(
            os.path.join(os.path.dirname(__file__), "../../"))
    
        self.template_paths = [
            os.path.abspath(os.path.join(self.beat_path, "../../metricbeat")),
            os.path.abspath(os.path.join(self.beat_path, "../../libbeat")),
        ]
    
>       super(XPackTest, self).setUpClass()

tests/system/xpack_metricbeat.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../metricbeat/tests/system/metricbeat.py:42: in setUpClass
    super().setUpClass()
../../libbeat/tests/system/beat/beat.py:204: in setUpClass
    cls.compose_up_with_retries()
../../libbeat/tests/system/beat/beat.py:222: in compose_up_with_retries
    raise ex
../../libbeat/tests/system/beat/beat.py:218: in compose_up_with_retries
    cls.compose_up()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class "test_xpack_base.Test">

    @classmethod
    def compose_up(cls):
        """
        Ensure *only* the services defined under `COMPOSE_SERVICES` are running and healthy
        """
        if not INTEGRATION_TESTS or not cls.COMPOSE_SERVICES:
            return
    
        if os.environ.get("NO_COMPOSE"):
            return
    
        def print_logs(container):
            print("---- " + container.name_without_project)
            print(container.logs())
            print("----")
    
        def is_healthy(container):
            return container.inspect()["State"]["Health"]["Status"] == "healthy"
    
        project = cls.compose_project()
    
        with disabled_logger("compose.service"):
            project.pull(
                ignore_pull_failures=True,
                service_names=cls.COMPOSE_SERVICES)
    
        project.up(
            strategy=ConvergenceStrategy.always,
            service_names=cls.COMPOSE_SERVICES,
            timeout=30)
    
        # Wait for them to be healthy
        start = time.time()
        while True:
            containers = project.containers(
                service_names=cls.COMPOSE_SERVICES,
                stopped=True)
    
            healthy = True
            for container in containers:
                if not container.is_running:
                    print_logs(container)
                    raise Exception(
                        "Container %s unexpectedly finished on startup" %
                        container.name_without_project)
                if not is_healthy(container):
                    healthy = False
                    break
    
            if healthy:
                break
    
            if cls.COMPOSE_ADVERTISED_HOST:
                for service in cls.COMPOSE_SERVICES:
                    cls._setup_advertised_host(project, service)
    
            time.sleep(1)
            timeout = time.time() - start > cls.COMPOSE_TIMEOUT
            if timeout:
                for container in containers:
                    if not is_healthy(container):
                        print_logs(container)
>               raise Exception(
                    "Timeout while waiting for healthy "
                    "docker-compose services: %s" %
                    ",".join(cls.COMPOSE_SERVICES))
E               Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana

../../libbeat/tests/system/beat/compose.py:102: Exception

Build&Test / x-pack/metricbeat-pythonIntegTest / test_export_template – x-pack.metricbeat.tests.system.test_xpack_base.Test
    Expand to view the error details

     failed on setup with "Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana"

    Expand to view the stacktrace

     self = <class "test_xpack_base.Test">

    @classmethod
    def setUpClass(self):
        self.beat_name = "metricbeat"
        self.beat_path = os.path.abspath(
            os.path.join(os.path.dirname(__file__), "../../"))
    
        self.template_paths = [
            os.path.abspath(os.path.join(self.beat_path, "../../metricbeat")),
            os.path.abspath(os.path.join(self.beat_path, "../../libbeat")),
        ]
    
>       super(XPackTest, self).setUpClass()

tests/system/xpack_metricbeat.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../metricbeat/tests/system/metricbeat.py:42: in setUpClass
    super().setUpClass()
../../libbeat/tests/system/beat/beat.py:204: in setUpClass
    cls.compose_up_with_retries()
../../libbeat/tests/system/beat/beat.py:222: in compose_up_with_retries
    raise ex
../../libbeat/tests/system/beat/beat.py:218: in compose_up_with_retries
    cls.compose_up()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class "test_xpack_base.Test">

    @classmethod
    def compose_up(cls):
        """
        Ensure *only* the services defined under `COMPOSE_SERVICES` are running and healthy
        """
        if not INTEGRATION_TESTS or not cls.COMPOSE_SERVICES:
            return
    
        if os.environ.get("NO_COMPOSE"):
            return
    
        def print_logs(container):
            print("---- " + container.name_without_project)
            print(container.logs())
            print("----")
    
        def is_healthy(container):
            return container.inspect()["State"]["Health"]["Status"] == "healthy"
    
        project = cls.compose_project()
    
        with disabled_logger("compose.service"):
            project.pull(
                ignore_pull_failures=True,
                service_names=cls.COMPOSE_SERVICES)
    
        project.up(
            strategy=ConvergenceStrategy.always,
            service_names=cls.COMPOSE_SERVICES,
            timeout=30)
    
        # Wait for them to be healthy
        start = time.time()
        while True:
            containers = project.containers(
                service_names=cls.COMPOSE_SERVICES,
                stopped=True)
    
            healthy = True
            for container in containers:
                if not container.is_running:
                    print_logs(container)
                    raise Exception(
                        "Container %s unexpectedly finished on startup" %
                        container.name_without_project)
                if not is_healthy(container):
                    healthy = False
                    break
    
            if healthy:
                break
    
            if cls.COMPOSE_ADVERTISED_HOST:
                for service in cls.COMPOSE_SERVICES:
                    cls._setup_advertised_host(project, service)
    
            time.sleep(1)
            timeout = time.time() - start > cls.COMPOSE_TIMEOUT
            if timeout:
                for container in containers:
                    if not is_healthy(container):
                        print_logs(container)
>               raise Exception(
                    "Timeout while waiting for healthy "
                    "docker-compose services: %s" %
                    ",".join(cls.COMPOSE_SERVICES))
E               Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana

../../libbeat/tests/system/beat/compose.py:102: Exception

Build&Test / x-pack/metricbeat-pythonIntegTest / test_index_management – x-pack.metricbeat.tests.system.test_xpack_base.Test
    Expand to view the error details

     failed on setup with "Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana"

    Expand to view the stacktrace

     self = <class "test_xpack_base.Test">

    @classmethod
    def setUpClass(self):
        self.beat_name = "metricbeat"
        self.beat_path = os.path.abspath(
            os.path.join(os.path.dirname(__file__), "../../"))
    
        self.template_paths = [
            os.path.abspath(os.path.join(self.beat_path, "../../metricbeat")),
            os.path.abspath(os.path.join(self.beat_path, "../../libbeat")),
        ]
    
>       super(XPackTest, self).setUpClass()

tests/system/xpack_metricbeat.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../metricbeat/tests/system/metricbeat.py:42: in setUpClass
    super().setUpClass()
../../libbeat/tests/system/beat/beat.py:204: in setUpClass
    cls.compose_up_with_retries()
../../libbeat/tests/system/beat/beat.py:222: in compose_up_with_retries
    raise ex
../../libbeat/tests/system/beat/beat.py:218: in compose_up_with_retries
    cls.compose_up()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class "test_xpack_base.Test">

    @classmethod
    def compose_up(cls):
        """
        Ensure *only* the services defined under `COMPOSE_SERVICES` are running and healthy
        """
        if not INTEGRATION_TESTS or not cls.COMPOSE_SERVICES:
            return
    
        if os.environ.get("NO_COMPOSE"):
            return
    
        def print_logs(container):
            print("---- " + container.name_without_project)
            print(container.logs())
            print("----")
    
        def is_healthy(container):
            return container.inspect()["State"]["Health"]["Status"] == "healthy"
    
        project = cls.compose_project()
    
        with disabled_logger("compose.service"):
            project.pull(
                ignore_pull_failures=True,
                service_names=cls.COMPOSE_SERVICES)
    
        project.up(
            strategy=ConvergenceStrategy.always,
            service_names=cls.COMPOSE_SERVICES,
            timeout=30)
    
        # Wait for them to be healthy
        start = time.time()
        while True:
            containers = project.containers(
                service_names=cls.COMPOSE_SERVICES,
                stopped=True)
    
            healthy = True
            for container in containers:
                if not container.is_running:
                    print_logs(container)
                    raise Exception(
                        "Container %s unexpectedly finished on startup" %
                        container.name_without_project)
                if not is_healthy(container):
                    healthy = False
                    break
    
            if healthy:
                break
    
            if cls.COMPOSE_ADVERTISED_HOST:
                for service in cls.COMPOSE_SERVICES:
                    cls._setup_advertised_host(project, service)
    
            time.sleep(1)
            timeout = time.time() - start > cls.COMPOSE_TIMEOUT
            if timeout:
                for container in containers:
                    if not is_healthy(container):
                        print_logs(container)
>               raise Exception(
                    "Timeout while waiting for healthy "
                    "docker-compose services: %s" %
                    ",".join(cls.COMPOSE_SERVICES))
E               Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana

../../libbeat/tests/system/beat/compose.py:102: Exception

Build&Test / x-pack/metricbeat-pythonIntegTest / test_start_stop – x-pack.metricbeat.tests.system.test_xpack_base.Test
    Expand to view the error details

     failed on setup with "Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana"

    Expand to view the stacktrace

     self = <class "test_xpack_base.Test">

    @classmethod
    def setUpClass(self):
        self.beat_name = "metricbeat"
        self.beat_path = os.path.abspath(
            os.path.join(os.path.dirname(__file__), "../../"))
    
        self.template_paths = [
            os.path.abspath(os.path.join(self.beat_path, "../../metricbeat")),
            os.path.abspath(os.path.join(self.beat_path, "../../libbeat")),
        ]
    
>       super(XPackTest, self).setUpClass()

tests/system/xpack_metricbeat.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../metricbeat/tests/system/metricbeat.py:42: in setUpClass
    super().setUpClass()
../../libbeat/tests/system/beat/beat.py:204: in setUpClass
    cls.compose_up_with_retries()
../../libbeat/tests/system/beat/beat.py:222: in compose_up_with_retries
    raise ex
../../libbeat/tests/system/beat/beat.py:218: in compose_up_with_retries
    cls.compose_up()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class "test_xpack_base.Test">

    @classmethod
    def compose_up(cls):
        """
        Ensure *only* the services defined under `COMPOSE_SERVICES` are running and healthy
        """
        if not INTEGRATION_TESTS or not cls.COMPOSE_SERVICES:
            return
    
        if os.environ.get("NO_COMPOSE"):
            return
    
        def print_logs(container):
            print("---- " + container.name_without_project)
            print(container.logs())
            print("----")
    
        def is_healthy(container):
            return container.inspect()["State"]["Health"]["Status"] == "healthy"
    
        project = cls.compose_project()
    
        with disabled_logger("compose.service"):
            project.pull(
                ignore_pull_failures=True,
                service_names=cls.COMPOSE_SERVICES)
    
        project.up(
            strategy=ConvergenceStrategy.always,
            service_names=cls.COMPOSE_SERVICES,
            timeout=30)
    
        # Wait for them to be healthy
        start = time.time()
        while True:
            containers = project.containers(
                service_names=cls.COMPOSE_SERVICES,
                stopped=True)
    
            healthy = True
            for container in containers:
                if not container.is_running:
                    print_logs(container)
                    raise Exception(
                        "Container %s unexpectedly finished on startup" %
                        container.name_without_project)
                if not is_healthy(container):
                    healthy = False
                    break
    
            if healthy:
                break
    
            if cls.COMPOSE_ADVERTISED_HOST:
                for service in cls.COMPOSE_SERVICES:
                    cls._setup_advertised_host(project, service)
    
            time.sleep(1)
            timeout = time.time() - start > cls.COMPOSE_TIMEOUT
            if timeout:
                for container in containers:
                    if not is_healthy(container):
                        print_logs(container)
>               raise Exception(
                    "Timeout while waiting for healthy "
                    "docker-compose services: %s" %
                    ",".join(cls.COMPOSE_SERVICES))
E               Exception: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana

../../libbeat/tests/system/beat/compose.py:102: Exception

Build&Test / x-pack/metricbeat-pythonIntegTest / test_health – x-pack.metricbeat.module.enterprisesearch.test_enterprisesearch.Test
    Expand to view the error details

     failed on setup with "compose.project.ProjectError: Encountered errors while bringing up the project."

    Expand to view the stacktrace

     self = <class "test_enterprisesearch.Test">

    @classmethod
    def setUpClass(self):
        self.beat_name = "metricbeat"
        self.beat_path = os.path.abspath(
            os.path.join(os.path.dirname(__file__), "../../"))
    
        self.template_paths = [
            os.path.abspath(os.path.join(self.beat_path, "../../metricbeat")),
            os.path.abspath(os.path.join(self.beat_path, "../../libbeat")),
        ]
    
>       super(XPackTest, self).setUpClass()

tests/system/xpack_metricbeat.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../metricbeat/tests/system/metricbeat.py:42: in setUpClass
    super().setUpClass()
../../libbeat/tests/system/beat/beat.py:204: in setUpClass
    cls.compose_up_with_retries()
../../libbeat/tests/system/beat/beat.py:222: in compose_up_with_retries
    raise ex
../../libbeat/tests/system/beat/beat.py:218: in compose_up_with_retries
    cls.compose_up()
../../libbeat/tests/system/beat/compose.py:66: in compose_up
    project.up(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <compose.project.Project object at 0x703b2ab26ed0>
service_names = ["enterprise_search"], start_deps = True
strategy = <ConvergenceStrategy.always: 2>, do_build = <BuildAction.none: 0>
timeout = 30, detached = False, remove_orphans = False, ignore_orphans = False
scale_override = {}, rescale = True, start = True, always_recreate_deps = False
reset_container_image = False, renew_anonymous_volumes = False, silent = False
cli = False, one_off = False, attach_dependencies = False
override_options = None

    def up(self,
           service_names=None,
           start_deps=True,
           strategy=ConvergenceStrategy.changed,
           do_build=BuildAction.none,
           timeout=None,
           detached=False,
           remove_orphans=False,
           ignore_orphans=False,
           scale_override=None,
           rescale=True,
           start=True,
           always_recreate_deps=False,
           reset_container_image=False,
           renew_anonymous_volumes=False,
           silent=False,
           cli=False,
           one_off=False,
           attach_dependencies=False,
           override_options=None,
           ):
    
        self.initialize()
        if not ignore_orphans:
            self.find_orphan_containers(remove_orphans)
    
        if scale_override is None:
            scale_override = {}
    
        services = self.get_services_without_duplicate(
            service_names,
            include_deps=start_deps)
    
        for svc in services:
            svc.ensure_image_exists(do_build=do_build, silent=silent, cli=cli)
        plans = self._get_convergence_plans(
            services,
            strategy,
            always_recreate_deps=always_recreate_deps,
            one_off=service_names if one_off else [],
        )
    
        services_to_attach = filter_attached_for_up(
            services,
            service_names,
            attach_dependencies,
            lambda service: service.name)
    
        def do(service):
            return service.execute_convergence_plan(
                plans[service.name],
                timeout=timeout,
                detached=detached or (service not in services_to_attach),
                scale_override=scale_override.get(service.name),
                rescale=rescale,
                start=start,
                reset_container_image=reset_container_image,
                renew_anonymous_volumes=renew_anonymous_volumes,
                override_options=override_options,
            )
    
        def get_deps(service):
            return {
                (self.get_service(dep), config)
                for dep, config in service.get_dependency_configs().items()
            }
    
        results, errors = parallel.parallel_execute(
            services,
            do,
            operator.attrgetter("name"),
            None,
            get_deps,
        )
        if errors:
>           raise ProjectError(
                "Encountered errors while bringing up the project."
            )
E           compose.project.ProjectError: Encountered errors while bringing up the project.

/opt/venv/lib/python3.11/site-packages/compose/project.py:705: ProjectError

Build&Test / x-pack/metricbeat-pythonIntegTest / test_stats – x-pack.metricbeat.module.enterprisesearch.test_enterprisesearch.Test
    Expand to view the error details

     failed on setup with "compose.project.ProjectError: Encountered errors while bringing up the project."

    Expand to view the stacktrace

     self = <class "test_enterprisesearch.Test">

    @classmethod
    def setUpClass(self):
        self.beat_name = "metricbeat"
        self.beat_path = os.path.abspath(
            os.path.join(os.path.dirname(__file__), "../../"))
    
        self.template_paths = [
            os.path.abspath(os.path.join(self.beat_path, "../../metricbeat")),
            os.path.abspath(os.path.join(self.beat_path, "../../libbeat")),
        ]
    
>       super(XPackTest, self).setUpClass()

tests/system/xpack_metricbeat.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../metricbeat/tests/system/metricbeat.py:42: in setUpClass
    super().setUpClass()
../../libbeat/tests/system/beat/beat.py:204: in setUpClass
    cls.compose_up_with_retries()
../../libbeat/tests/system/beat/beat.py:222: in compose_up_with_retries
    raise ex
../../libbeat/tests/system/beat/beat.py:218: in compose_up_with_retries
    cls.compose_up()
../../libbeat/tests/system/beat/compose.py:66: in compose_up
    project.up(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <compose.project.Project object at 0x703b2ab26ed0>
service_names = ["enterprise_search"], start_deps = True
strategy = <ConvergenceStrategy.always: 2>, do_build = <BuildAction.none: 0>
timeout = 30, detached = False, remove_orphans = False, ignore_orphans = False
scale_override = {}, rescale = True, start = True, always_recreate_deps = False
reset_container_image = False, renew_anonymous_volumes = False, silent = False
cli = False, one_off = False, attach_dependencies = False
override_options = None

    def up(self,
           service_names=None,
           start_deps=True,
           strategy=ConvergenceStrategy.changed,
           do_build=BuildAction.none,
           timeout=None,
           detached=False,
           remove_orphans=False,
           ignore_orphans=False,
           scale_override=None,
           rescale=True,
           start=True,
           always_recreate_deps=False,
           reset_container_image=False,
           renew_anonymous_volumes=False,
           silent=False,
           cli=False,
           one_off=False,
           attach_dependencies=False,
           override_options=None,
           ):
    
        self.initialize()
        if not ignore_orphans:
            self.find_orphan_containers(remove_orphans)
    
        if scale_override is None:
            scale_override = {}
    
        services = self.get_services_without_duplicate(
            service_names,
            include_deps=start_deps)
    
        for svc in services:
            svc.ensure_image_exists(do_build=do_build, silent=silent, cli=cli)
        plans = self._get_convergence_plans(
            services,
            strategy,
            always_recreate_deps=always_recreate_deps,
            one_off=service_names if one_off else [],
        )
    
        services_to_attach = filter_attached_for_up(
            services,
            service_names,
            attach_dependencies,
            lambda service: service.name)
    
        def do(service):
            return service.execute_convergence_plan(
                plans[service.name],
                timeout=timeout,
                detached=detached or (service not in services_to_attach),
                scale_override=scale_override.get(service.name),
                rescale=rescale,
                start=start,
                reset_container_image=reset_container_image,
                renew_anonymous_volumes=renew_anonymous_volumes,
                override_options=override_options,
            )
    
        def get_deps(service):
            return {
                (self.get_service(dep), config)
                for dep, config in service.get_dependency_configs().items()
            }
    
        results, errors = parallel.parallel_execute(
            services,
            do,
            operator.attrgetter("name"),
            None,
            get_deps,
        )
        if errors:
>           raise ProjectError(
                "Encountered errors while bringing up the project."
            )
E           compose.project.ProjectError: Encountered errors while bringing up the project.

/opt/venv/lib/python3.11/site-packages/compose/project.py:705: ProjectError

Steps errors 4

Expand to view the steps failures

x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 27 min 57 sec . View more details here
  • Description: mage pythonIntegTest
x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 20 min 0 sec . View more details here
  • Description: mage pythonIntegTest
x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 20 min 46 sec . View more details here
  • Description: mage pythonIntegTest
Error signal
  • Took 0 min 0 sec . View more details here
  • Description: Error "hudson.AbortException: script returned exit code 1"

@elasticmachine elasticmachine added automation build-failures Build failures in the CI. ci-reported Issues that have been automatically reported from the CI Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Apr 19, 2024
@elasticmachine
Copy link
Collaborator Author

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@pierrehilbert pierrehilbert changed the title Build 5 for 8.14 with status FAILURE Build 5 for 8.14 with status FAILURE - Compose up failed, retrying: Timeout while waiting for healthy docker-compose services: elasticsearch,kibana Apr 22, 2024
@pierrehilbert pierrehilbert mentioned this issue Apr 22, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
automation build-failures Build failures in the CI. ci-reported Issues that have been automatically reported from the CI Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@elasticmachine