Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--show-failed-logs if binary file used as log file #2826

Open
votti opened this issue Apr 19, 2024 · 0 comments
Open

--show-failed-logs if binary file used as log file #2826

votti opened this issue Apr 19, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@votti
Copy link
Contributor

votti commented Apr 19, 2024

Snakemake version
main banch

Describe the bug
If a binary file is set as log output and --show-failed-logs (or other options triggering logs to be shown) is activated, if a job crashes the whole workflow execution crashes as snakemake tries to print the binary log file as text.

Logs


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake/executors/local.py", line 259, in _callback
    raise ex
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake/executors/local.py", line 245, in cached_or_run
    run_func(*args)
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake/executors/local.py", line 232, in spawn_job
    raise SpawnedJobError()
snakemake.exceptions.SpawnedJobError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/concurrent/futures/_base.py", line 340, in _invoke_callbacks
    callback(self)
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake/executors/local.py", line 266, in _callback
    self.report_job_error(job_info)
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake_interface_executor_plugins/executors/base.py", line 92, in report_job_error
    self.print_job_error(job_info, msg, **kwargs)
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake_interface_executor_plugins/executors/base.py", line 113, in print_job_error
    job_info.job.log_error(msg, **kwargs)
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake/jobs.py", line 1023, in log_error
    logger.job_error(**self.get_log_error_info(msg, indent, aux_logs, **kwargs))
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake/logging.py", line 418, in job_error
    self.handler(msg)
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake/logging.py", line 359, in handler
    handler(msg)
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake/logging.py", line 589, in text_handler
    self.logger.error("\n".join(map(indent, job_error())))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake/logging.py", line 584, in job_error
    yield from show_logs(msg["log"])
  File "/project/home/vizano/mambaforge/envs/snakemake8/lib/python3.12/site-packages/snakemake/logging.py", line 505, in show_logs
    content = open(f, "r").read()
              ^^^^^^^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

Minimal example
Snakefile:

rule broken_rule:
    log:
        'log.bin'
    run:
        import pickle
        with open(log[0], 'wb') as f:
            pickle.dump(log, f)
        asd

Run with:

snakemake --jobs 1 --show-failed-logs

Additional context
I encountered this issue when running a workflow that uses the log to store R environment when the rule is run (I guess to facilitate debugging).
For me this triggered a whole Slurm execution to fail and it took me quite a while to figure out what was going on.

@votti votti added the bug Something isn't working label Apr 19, 2024
votti pushed a commit to FroeseLab/snakemake that referenced this issue Apr 19, 2024
This replicates snakemake#2826

Unfortunately this test seems to currently crush the pytest test execution
votti pushed a commit to FroeseLab/snakemake that referenced this issue Apr 19, 2024
Previously a binary log file would cause `show_logs` to crash the whole
snakemake workflow execution with a `UnicodeDecodeError`.

Closes snakemake#2826
votti pushed a commit to FroeseLab/snakemake that referenced this issue Apr 19, 2024
This replicates snakemake#2826

Unfortunately this test seems to currently crush the pytest test execution
votti pushed a commit to FroeseLab/snakemake that referenced this issue Apr 19, 2024
Previously a binary log file would cause `show_logs` to crash the whole
snakemake workflow execution with a `UnicodeDecodeError`.

Closes snakemake#2826
votti pushed a commit to FroeseLab/snakemake that referenced this issue Apr 19, 2024
This replicates snakemake#2826

Unfortunately this test seems to currently crush the pytest test execution
votti pushed a commit to FroeseLab/snakemake that referenced this issue Apr 19, 2024
Previously a binary log file would cause `show_logs` to crash the whole
snakemake workflow execution with a `UnicodeDecodeError`.

Closes snakemake#2826
votti pushed a commit to FroeseLab/snakemake that referenced this issue Apr 22, 2024
This replicates snakemake#2826

Unfortunately this test seems to currently crush the pytest test
execution when failing. Given that it should not fail any more
I think this may be acceptable.
votti pushed a commit to FroeseLab/snakemake that referenced this issue Apr 22, 2024
Previously a binary log file would cause `show_logs` to crash the whole
snakemake workflow execution with a `UnicodeDecodeError`.

Closes snakemake#2826
johanneskoester pushed a commit that referenced this issue Apr 28, 2024
…-failed-logs` (#2827)

### Description

As described in #2826 currently the whole snakemake execution crashes if
a log file is binary and `show-failed-logs` is used.

This fix prevents the crash by capturing the `UnicodeDecodeError` and
returning an error message that the log file is not a text instead.

### QC

* [x] The PR contains a test case for the changes or the changes are
already covered by an existing test case.
* [x] The documentation (`docs/`) is updated to reflect the changes or
this is not necessary (e.g. if the change does neither modify the
language nor the behavior or functionalities of Snakemake).

---------

Co-authored-by: Vito Zanotelli <vito.zanotelli@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant