Testsuite fails intermittently

[yurivict]

In most test runs many tests fail.
But every once in a while al tests succeed.
It looks like there is a dependency on the order tests are run. (PyTest runs them in parallel.)

[olofk]

I have not been able to reproduced this myself. The problem might still exist, but there's not much I can do about it without more information. If you have any logs from a failed run, please add them here. Otherwise I will have to close this without action

[yurivict]

For example, I got this error:

=============================================================================================== FAILURES ================================================================================================
_____________________________________________________________________________________________ test_modelsim _____________________________________________________________________________________________

self = <edalize.modelsim.Modelsim object at 0x91b546b50>, cmd = 'gmake', args = [], quiet = True

    def _run_tool(self, cmd, args=[], quiet=False):
        logger.debug("Running " + cmd)
        logger.debug("args  : " + " ".join(args))
    
        capture_output = quiet and not (self.verbose or self.stdout or self.stderr)
        try:
>           cp = run(
                [cmd] + args,
                cwd=self.work_root,
                stdin=subprocess.PIPE,
                stdout=self.stdout,
                stderr=self.stderr,
                capture_output=capture_output,
                check=True,
            )

edalize/edatool.py:460: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = None, check = True, popenargs = (['gmake'],)
kwargs = {'cwd': '/tmp/pytest-of-yuri/pytest-70/test_modelsim0/0', 'stderr': None, 'stdin': -1, 'stdout': None}, process = <Popen: returncode: 2 args: ['gmake']>, stdout = None, stderr = None
retcode = 2

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['gmake']' returned non-zero exit status 2.

/usr/local/lib/python3.9/subprocess.py:528: CalledProcessError

During handling of the above exception, another exception occurred:

make_edalize_test = <function make_edalize_test.<locals>._fun at 0x91b35b550>

    def test_modelsim(make_edalize_test):
        tool_options = {
            "vcom_options": ["various", "vcom_options"],
            "vlog_options": ["some", "vlog_options"],
            "vsim_options": ["a", "few", "vsim_options"],
        }
    
        # FIXME: Add VPI tests
        tf = make_edalize_test("modelsim", tool_options=tool_options)
    
        tf.backend.configure()
    
        tf.compare_files(["Makefile", "edalize_build_rtl.tcl", "edalize_main.tcl"])
    
        orig_env = os.environ.copy()
        try:
            os.environ["MODEL_TECH"] = os.path.join(tests_dir, "mock_commands")
    
>           tf.backend.build()

tests/test_modelsim.py:24: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
edalize/edatool.py:241: in build
    self.build_main()
edalize/edatool.py:252: in build_main
    self._run_tool("gmake", [] if target is None else [target], quiet=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <edalize.modelsim.Modelsim object at 0x91b546b50>, cmd = 'gmake', args = [], quiet = True

    def _run_tool(self, cmd, args=[], quiet=False):
        logger.debug("Running " + cmd)
        logger.debug("args  : " + " ".join(args))
    
        capture_output = quiet and not (self.verbose or self.stdout or self.stderr)
        try:
            cp = run(
                [cmd] + args,
                cwd=self.work_root,
                stdin=subprocess.PIPE,
                stdout=self.stdout,
                stderr=self.stderr,
                capture_output=capture_output,
                check=True,
            )
        except FileNotFoundError:
            _s = "Command '{}' not found. Make sure it is in $PATH".format(cmd)
            raise RuntimeError(_s)
        except subprocess.CalledProcessError as e:
            _s = "'{}' exited with an error: {}".format(e.cmd, e.returncode)
            logger.debug(_s)
    
            if e.stdout:
                logger.info(e.stdout.decode())
            if e.stderr:
                logger.error(e.stderr.decode())
                logger.debug("=== STDERR ===")
                logger.debug(e.stderr)
    
>           raise RuntimeError(_s)
E           RuntimeError: '['gmake']' exited with an error: 2

edalize/edatool.py:483: RuntimeError
----------------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------------
gmake[1]: Entering directory '/tmp/pytest-of-yuri/pytest-70/test_modelsim0/0'
gmake[1]: Leaving directory '/tmp/pytest-of-yuri/pytest-70/test_modelsim0/0'

[yurivict]

Another error:

______________________________________________________________________________________________ test_oxide _______________________________________________________________________________________________

self = <edalize.oxide.Oxide object at 0x92381a460>, cmd = 'gmake', args = [], quiet = True

    def _run_tool(self, cmd, args=[], quiet=False):
        logger.debug("Running " + cmd)
        logger.debug("args  : " + " ".join(args))
    
        capture_output = quiet and not (self.verbose or self.stdout or self.stderr)
        try:
>           cp = run(
                [cmd] + args,
                cwd=self.work_root,
                stdin=subprocess.PIPE,
                stdout=self.stdout,
                stderr=self.stderr,
                capture_output=capture_output,
                check=True,
            )

edalize/edatool.py:460: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = None, check = True, popenargs = (['gmake'],)
kwargs = {'cwd': '/tmp/pytest-of-yuri/pytest-72/test_oxide0/0', 'stderr': None, 'stdin': -1, 'stdout': None}, process = <Popen: returncode: 2 args: ['gmake']>, stdout = None, stderr = None, retcode = 2

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['gmake']' returned non-zero exit status 2.

/usr/local/lib/python3.9/subprocess.py:528: CalledProcessError

During handling of the above exception, another exception occurred:

make_edalize_test = <function make_edalize_test.<locals>._fun at 0x922fa20d0>

    def test_oxide(make_edalize_test):
        tool_options = {
            "device": "LIFCL-40-9BG400CES",
            "yosys_synth_options": ["some", "yosys_synth_options"],
            "nextpnr_options": ["a", "few", "nextpnr_options"],
        }
        tf = make_edalize_test(
            "oxide", param_types=["vlogdefine", "vlogparam"], tool_options=tool_options
        )
    
>       run_oxide_test(tf)

tests/test_oxide.py:27: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_oxide.py:13: in run_oxide_test
    tf.backend.build()
edalize/edatool.py:241: in build
    self.build_main()
edalize/edatool.py:252: in build_main
    self._run_tool("gmake", [] if target is None else [target], quiet=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <edalize.oxide.Oxide object at 0x92381a460>, cmd = 'gmake', args = [], quiet = True

    def _run_tool(self, cmd, args=[], quiet=False):
        logger.debug("Running " + cmd)
        logger.debug("args  : " + " ".join(args))
    
        capture_output = quiet and not (self.verbose or self.stdout or self.stderr)
        try:
            cp = run(
                [cmd] + args,
                cwd=self.work_root,
                stdin=subprocess.PIPE,
                stdout=self.stdout,
                stderr=self.stderr,
                capture_output=capture_output,
                check=True,
            )
        except FileNotFoundError:
            _s = "Command '{}' not found. Make sure it is in $PATH".format(cmd)
            raise RuntimeError(_s)
        except subprocess.CalledProcessError as e:
            _s = "'{}' exited with an error: {}".format(e.cmd, e.returncode)
            logger.debug(_s)
    
            if e.stdout:
                logger.info(e.stdout.decode())
            if e.stderr:
                logger.error(e.stderr.decode())
                logger.debug("=== STDERR ===")
                logger.debug(e.stderr)
    
>           raise RuntimeError(_s)
E           RuntimeError: '['gmake']' exited with an error: 2

edalize/edatool.py:483: RuntimeError
----------------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------------
gmake[1]: Entering directory '/tmp/pytest-of-yuri/pytest-72/test_oxide0/0'
yosys -l yosys.log -p 'tcl edalize_yosys_template.tcl'

 /----------------------------------------------------------------------------\
 |                                                                            |
 |  yosys -- Yosys Open SYnthesis Suite                                       |
 |                                                                            |
 |  Copyright (C) 2012 - 2020  Claire Xenia Wolf <claire@yosyshq.com>         |
 |                                                                            |
 |  Permission to use, copy, modify, and/or distribute this software for any  |
 |  purpose with or without fee is hereby granted, provided that the above    |
 |  copyright notice and this permission notice appear in all copies.         |
 |                                                                            |
 |  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES  |
 |  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF          |
 |  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR   |
 |  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES    |
 |  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN     |
 |  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF   |
 |  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.            |
 |                                                                            |
 \----------------------------------------------------------------------------/

 Yosys 0.26 (git sha1 7e588664e7e, c++ 13.0.0 -O2 -fstack-protector-strong -fno-strict-aliasing -fPIC -Os)


-- Running command `tcl edalize_yosys_template.tcl' --
[TCL: yosys -import] Command name collision: found pre-existing command `cd' -> skip.
[TCL: yosys -import] Command name collision: found pre-existing command `eval' -> skip.
[TCL: yosys -import] Command name collision: found pre-existing command `exec' -> skip.
[TCL: yosys -import] Command name collision: found pre-existing command `read' -> skip.
[TCL: yosys -import] Command name collision: found pre-existing command `trace' -> skip.
ERROR: Can't open input file `sv_file.sv' for reading: No such file or directory
gmake[1]: Leaving directory '/tmp/pytest-of-yuri/pytest-72/test_oxide0/0'
----------------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------------
ERROR: TCL interpreter returned an error: Yosys command produced an error
gmake[1]: *** [Makefile:6: test_oxide_0.json] Error 1
_____________________________________________________________________________________________ test_ghdl_03 ______________________________________________________________________________________________

self = <edalize.ghdl.Ghdl object at 0x9220374f0>, cmd = 'gmake', args = [], quiet = True

    def _run_tool(self, cmd, args=[], quiet=False):
        logger.debug("Running " + cmd)
        logger.debug("args  : " + " ".join(args))
    
        capture_output = quiet and not (self.verbose or self.stdout or self.stderr)
        try:
>           cp = run(
                [cmd] + args,
                cwd=self.work_root,
                stdin=subprocess.PIPE,
                stdout=self.stdout,
                stderr=self.stderr,
                capture_output=capture_output,
                check=True,
            )

edalize/edatool.py:460: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = None, check = True, popenargs = (['gmake'],)
kwargs = {'cwd': '/tmp/pytest-of-yuri/pytest-72/test_ghdl_030/0', 'stderr': None, 'stdin': -1, 'stdout': None}, process = <Popen: returncode: 2 args: ['gmake']>, stdout = None, stderr = None
retcode = 2

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['gmake']' returned non-zero exit status 2.

/usr/local/lib/python3.9/subprocess.py:528: CalledProcessError

During handling of the above exception, another exception occurred:

make_edalize_test = <function make_edalize_test.<locals>._fun at 0x923817af0>

    def test_ghdl_03(make_edalize_test):
        tf = make_edalize_test(
            "ghdl",
            ref_dir="test03",
            test_name="test_ghdl_03",
            param_types=["generic"],
            files=LOCAL_FILES,
            tool_options={
                "analyze_options": ["--std=08", "--ieee=synopsys"],
                "run_options": ["a", "few", "run_options"],
            },
        )
    
        for vhdl_file in ["vhdl_file.vhd", "vhdl_lfile", "vhdl2008_file"]:
            with open(os.path.join(tf.work_root, vhdl_file), "a"):
                os.utime(os.path.join(tf.work_root, vhdl_file), None)
    
        tf.backend.configure()
    
        tf.compare_files(["Makefile"])
    
>       tf.backend.build()

tests/test_ghdl.py:87: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
edalize/edatool.py:241: in build
    self.build_main()
edalize/edatool.py:252: in build_main
    self._run_tool("gmake", [] if target is None else [target], quiet=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <edalize.ghdl.Ghdl object at 0x9220374f0>, cmd = 'gmake', args = [], quiet = True

    def _run_tool(self, cmd, args=[], quiet=False):
        logger.debug("Running " + cmd)
        logger.debug("args  : " + " ".join(args))
    
        capture_output = quiet and not (self.verbose or self.stdout or self.stderr)
        try:
            cp = run(
                [cmd] + args,
                cwd=self.work_root,
                stdin=subprocess.PIPE,
                stdout=self.stdout,
                stderr=self.stderr,
                capture_output=capture_output,
                check=True,
            )
        except FileNotFoundError:
            _s = "Command '{}' not found. Make sure it is in $PATH".format(cmd)
            raise RuntimeError(_s)
        except subprocess.CalledProcessError as e:
            _s = "'{}' exited with an error: {}".format(e.cmd, e.returncode)
            logger.debug(_s)
    
            if e.stdout:
                logger.info(e.stdout.decode())
            if e.stderr:
                logger.error(e.stderr.decode())
                logger.debug("=== STDERR ===")
                logger.debug(e.stderr)
    
>           raise RuntimeError(_s)
E           RuntimeError: '['gmake']' exited with an error: 2

edalize/edatool.py:483: RuntimeError
----------------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------------
gmake[1]: Entering directory '/tmp/pytest-of-yuri/pytest-72/test_ghdl_030/0'
Creating libraries directories
mkdir -p libx
ghdl -i --std=08 --ieee=synopsys -P./libx  vhdl_file.vhd
gmake[1]: Leaving directory '/tmp/pytest-of-yuri/pytest-72/test_ghdl_030/0'
----------------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------------
vhdl_file.vhd:1:1: design file is empty (no design unit found)
ghdl: importation has failed due to compilation error
gmake[1]: *** [Makefile:26: work-obj08.cf] Error 1
------------------------------------------------------------------------------------------- Captured log call -------------------------------------------------------------------------------------------
WARNING  edalize.ghdl:ghdl.py:53 Analyze option --std=08 given, will override any vhdlSource-xxxx specification
_____________________________________________________________________________________________ test_ghdl_01 ______________________________________________________________________________________________

self = <edalize.ghdl.Ghdl object at 0x924744790>, cmd = 'gmake', args = [], quiet = True

    def _run_tool(self, cmd, args=[], quiet=False):
        logger.debug("Running " + cmd)
        logger.debug("args  : " + " ".join(args))
    
        capture_output = quiet and not (self.verbose or self.stdout or self.stderr)
        try:
>           cp = run(
                [cmd] + args,
                cwd=self.work_root,
                stdin=subprocess.PIPE,
                stdout=self.stdout,
                stderr=self.stderr,
                capture_output=capture_output,
                check=True,
            )

edalize/edatool.py:460: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = None, check = True, popenargs = (['gmake'],)
kwargs = {'cwd': '/tmp/pytest-of-yuri/pytest-72/test_ghdl_010/0', 'stderr': None, 'stdin': -1, 'stdout': None}, process = <Popen: returncode: 2 args: ['gmake']>, stdout = None, stderr = None
retcode = 2

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['gmake']' returned non-zero exit status 2.

/usr/local/lib/python3.9/subprocess.py:528: CalledProcessError

During handling of the above exception, another exception occurred:

make_edalize_test = <function make_edalize_test.<locals>._fun at 0x923817e50>

    def test_ghdl_01(make_edalize_test):
        tf = make_edalize_test(
            "ghdl",
            ref_dir="test01",
            param_types=["generic"],
            tool_options={
                "analyze_options": ["some", "analyze_options"],
                "run_options": ["a", "few", "run_options"],
            },
        )
    
        for vhdl_file in ["vhdl_file.vhd", "vhdl_lfile", "vhdl2008_file"]:
            with open(os.path.join(tf.work_root, vhdl_file), "a"):
                os.utime(os.path.join(tf.work_root, vhdl_file), None)
    
        tf.backend.configure()
    
        tf.compare_files(["Makefile"])
    
>       tf.backend.build()

tests/test_ghdl.py:24: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
edalize/edatool.py:241: in build
    self.build_main()
edalize/edatool.py:252: in build_main
    self._run_tool("gmake", [] if target is None else [target], quiet=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <edalize.ghdl.Ghdl object at 0x924744790>, cmd = 'gmake', args = [], quiet = True

    def _run_tool(self, cmd, args=[], quiet=False):
        logger.debug("Running " + cmd)
        logger.debug("args  : " + " ".join(args))
    
        capture_output = quiet and not (self.verbose or self.stdout or self.stderr)
        try:
            cp = run(
                [cmd] + args,
                cwd=self.work_root,
                stdin=subprocess.PIPE,
                stdout=self.stdout,
                stderr=self.stderr,
                capture_output=capture_output,
                check=True,
            )
        except FileNotFoundError:
            _s = "Command '{}' not found. Make sure it is in $PATH".format(cmd)
            raise RuntimeError(_s)
        except subprocess.CalledProcessError as e:
            _s = "'{}' exited with an error: {}".format(e.cmd, e.returncode)
            logger.debug(_s)
    
            if e.stdout:
                logger.info(e.stdout.decode())
            if e.stderr:
                logger.error(e.stderr.decode())
                logger.debug("=== STDERR ===")
                logger.debug(e.stderr)
    
>           raise RuntimeError(_s)
E           RuntimeError: '['gmake']' exited with an error: 2

edalize/edatool.py:483: RuntimeError
----------------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------------

[olofk]

Thank you. This provide some additional clues. It looks like your tests are trying to run the actual commands, rather than the mock commands from https://github.com/olofk/edalize/tree/main/tests/mock_commands which are supposed to override the ordinary binaries.
I also notice that your tests run gmake instead of plain make. Is this a change you have made directly in the Edalize source or some clever alias? Given your FreeBSD affiliation I guess it's to use GNU Make over BSD Make. And in that case, I also wonder if there are some assumptions made about the path environment that doesn't hold true for BSD

[yurivict]

0.5.3 has the similar failuses, see the log.

It appears that many failures are due to formatting differencres.
It isn't clear why formatting tests are a part of the testsuite. Sholudn't formatting be fixed at the development stage? Run-time results are not supposed to depend on code formatting.

gmake is the executable for GNU Make on BSDs. We apply these patches in the port:

        @${GREP} -rl '"make"' ${WRKSRC} | ${XARGS} ${REINPLACE_CMD} -i '' -e 's|"make"|"${GMAKE}"|'
        @${GREP} -rl '"python3"' ${WRKSRC} | ${XARGS} ${REINPLACE_CMD} -i '' -e 's|"python3"|"${PYTHON_CMD}"|'
        @${GREP} -rl 'shell python3 ' ${WRKSRC} | ${XARGS} ${REINPLACE_CMD} -i '' -e 's|shell python3 |shell ${PYTHON_CMD} |'
        @${GREP} -rl '$$(shell which python3)' ${WRKSRC} | ${XARGS} ${REINPLACE_CMD} -i '' -e 's|$$(shell which python3)|${PYTHON_CMD}|'
        @${REINPLACE_CMD} -i '' -e 's|/usr/bin/env python3|${PYTHON_CMD}|' ${WRKSRC}/tests/mock_commands/vcs

Ideally make and python3 should be configurable at the build time.

[olofk]

I'm also very confused about why it tries to do formatting as part of the test suite. I don't think I have seen that before. It's only supposed to run pytest. Manually running black locally, I also see a lot of errors, so we might as well just fix them. I created a freebsd branch where I have reformatted all files so that it's a bit easier to see what remaining issues we actually have. Is it possible for you to rerun the checks against that branch to see if we can get the number of errors down. There also seems to be a couple more errors and I'll try to look into those as well.

And point taken about make and python3. With the new flow API, it's a bit easier to change this centrally, so I'll look into doing that once we have the other stuff taken care of.