chore(main): release 8.0.0

[github-actions]

🤖 I have created a release beep boop

8.0.0 (2022-10-07)

⚠ BREAKING CHANGES

• template rendering integration (yte and jinja2), require Python 3.6 as minimum version (f-string support) (feat!: template rendering integration (yte and jinja2), require Python 3.6 as minimum version (f-string support) snakemake/snakemake#1410)

Features

• Add shadow: "copy-minimal" directive (#1155) (1803f0b)
• add Bash script functionality (#1821) (c4cf8fd)
• add flag --draft-notebook for generating a skeleton notebook for manual editing (e.g. in VSCode). (#1284) (d279322)
• add flag ensure that allows to annotate that certain output files should be non-empty or agree with a given checksum (#1651) (76f69d9)
• add gitfile option to make it possible to use local git repos when importing modules (#1376) (1a3b91f)
• Add more informative errors when evaluation of --default-resources fails (#1192) (b3c4e68)
• add pyproject.toml to use setuptools features (#1725) (454bfd1)
• Add support for rust scripts (enabling directly integrated ad-hoc robust high performance scripting) (#1053) (f0e8fa2)
• Added timestamp to each log message (#1304) (a5769f0)
• Adding --cluster-cancel and --cluster-cancel-nargs (#1395) (0593de1)
• adding default_target directive for declaring default target rules that are not the first rule in the workflow. (#1358) (638ec1a)
• adding flux executor (#1810) (40d2bd0)
• adding service jobs, i.e. the ability to define jobs that provide a resource for consumers (like a shared memory device or a database), and will be automatically terminated by Snakemake once all consumers are finished. (#1413) (a471adb)
• Allow paramspace to separate filename params with custom separator (#1299) (8236e80)
• allow prefix definition in module statements (#1310) (29e6540)
• Allow to mark all output files as temp with --all-temp (#1097) (0ac3b38)
• autoconvert Path objects to str when passing to R or Julia scripts (80ec513)
• automatically rerun jobs if parameters, code, input file set, or software stack changed (thanks to @cclienty and @timtroendle). This also increases performance of DAG building by handling job "needrun" updates level wise, while avoiding to perform a full check for those jobs that are already downstream of a job that has been determined to require a rerun. (#1663) (4c11893)
• cluster sidecar (#1397) (b992cd1)
• enable restricted access support in zenodo remote provider (#1623) (692caf9)
• enable the definition of conda pin files in order to freeze an environment. This can drastically increase the robustness because it allows to freeze an environment at a working state. (#1667) (53972bf)
• fail with an error if snakemake cannot write job metadata. (#1273) (cd968cd)
• for small files, compare checksums to determine if job needs to run if input file is newer than output file (#1568) (1ae85c6)
• Group resources (#1218) (a8014d0)
• implement support for removing GFAL remote files (#1103) (25943e5)
• improved reports: more interactive and modern interface, ability to define a label based representation of files (#1470) (d09df0c)
• LockException (#1276) (f5e6fa6)
• make it possible to exclude rules that will be imported when using 'use rule' statement (#1717) (d9e0611)
• new directive "retries" for annotating the number of times a job shall be restarted after a failure (#1649) (c8d81d0)
• print reason summary in case of dryrun (#1778) (bd2a68b)
• require at least Python 3.7 (fd5daae)
• specify conda environments via their name (#1340) (735ab23)
• Support conda environment definitions to be passed as function pointers, similar to input, params, and resources (#1300) (6f582f1)
• support for group local jobs by enabling optional groupid consideration in input functions (#1418) (5d45493)
• Support for machine_type for kubernetes executor (#1291) (12d6f67)
• support for post deploy scripts (#1325) (e5dac4f)
• support XRootD as a default remote provider (#1017) (fe03157)
• template rendering integration (yte and jinja2), require Python 3.6 as minimum version (f-string support) (#1410) (e1cbde5)
• vim syntax updates (#1584) (b8c77f6)
• Zenodo remote provider for transparent storage on and retrieval from Zenodo (#1455) (4586ef7)

Bug Fixes

• --conda-frontend value not passed on to cluster jobs (#1317) (df46ddb)
• add about page to report, including embedded packages and licenses (#1511) (142a452)
• Add back logging of run directives (#1883) (a65559c)
• add local marker for input files in cufflinks example. fixes issue #1362 (90bc88b)
• add lock free mechanism for avoiding race conditions when writing persistence information; consider corrupt metadata records as non-existent (#1745) (71fe952)
• add missing report.templates.components module to setup.py (cb4e3fe)
• add quotes to each item of the wait_for_files list (#1160) (72856ed)
• add_stylesheet() was deprecated in Sphing v1.8 (#1071) (#1076) (0a79b33)
• Adds fixes for the first two MREs in #823 (#1215) (cfd2f89)
• allocation of local ssds in k8s tests (#1870) (d0de4dc)
• allow apptainer as a successor to singularity. (#1706) (bcbdb0b)
• allow labels function to return None (#1565) (fef74d6)
• allow script directive to take pathlib Path (#1869) (12cdc96)
• allow to use {wildcards} for group jobs in cluster config (#1555) (f0ec73d)
• always deploy conda envs in main process when assuming a shared file system (fixes issue #1463) (#1472) (79788eb)
• always recalculate job resources before job is scheduled as input might have changed or not have been present initially (#1552) (44aacdb)
• always wait for input files before starting jobs, also upon local execution and within group jobs. This should add further robustness against NFS latency issues. (#1486) (cab2adb)
• AmbiguousRuleException bug caused by weak ordering of rules (#1124) (7f54c39)
• async_run to allow nested event loops. (#1170) (5dc6bbd)
• atomic job error display (#1326) (aa2c265)
• avoid "Admin" prompt when using conda on windows (#1046) (552fadf)
• avoid erroneous too early deletion of parent directories in case of failed jobs (thanks to @SichongP). (#1601) (b0917e6)
• avoid incomplete remote files in case of errors and automatically retry download and upload (#1432) (8fc23ed)
• avoid mutable default argument (#1330) (978cc93)
• avoid superfluous calls of conda info that have slowed down Snakemake since 6.4.1. (#1099) (e990927)
• better error message in case of failing to create conda env (#1526) (e7a461c)
• better error messages in case of missing files after latency period (#1528) (5b394c0)
• bug in pipe group handling that led to multiple assignments of the same group id to different groups; bug that accidentally added already running groups of the list of ready jobs (issue #1331) (#1332) (1a9b483)
• Bugfix tes add files (#1133) (8892bf2)
• caching process (#1225) (0825a29)
• catch errors in remote.AUTO provider list (#1834) (c613ed2)
• check for skipped rules in case of local rule inheritance (#1631) (9083ac1)
• cleaned up and rewritten execution backend structure, (fixing #1475, #860, #1007, #1008) (PR #1491) (e87cc97)
• conda python interpreter path on Windows (#1711) (155c9d6)
• consider post-deploy script for env hashing (#1363) (d50efd9)
• consistently use text output in conda shell commands and various little fixes for failing test cases due to conda package changes (#1864) (4234fe7)
• convert conda env to string before checks (#1382) (7a8da9f)
• correct handling of exceptions in input functions that are generators (#1536) (d9a56aa)
• declare associative arrays (#1844) (90ae449)
• disable mtime retrieval from github api for now. This quickly exceeds rate limits. (1858bb9)
• Disable Persistence cache for snakemake jobs (#1159) (7110f9d)
• display change warnings only for jobs that won't be executed otherwise (086f60f)
• display wrapper or external script code in report #1393 (#1404) (a007bd1)
• do not apply module prefix in case of remote files (5645b3f)
• do not pass SNAKEMAKE_PROFILE into cluster-submit (#1398) (#1407) (7189183)
• do not remove existing temp files in case of dryrun (#1543) (e820f97)
• do not require --cores or --jobs to be set when --cleanup-metadata is used. (#1429) (9c73907)
• do not skip local conda env creation per se when having no shared FS, because it is still needed for local jobs. Instead, decide for each env whether it is needed locally or not. (#1490) (3f03c5d)
• do not wait for named or containerized conda envs (#1473) (6b1d09c)
• do not wrap whitespace in result info headers of reports (653d0d0)
• don't raise WorkflowError when entry is empty (#1368) (1fc6f7b)
• Draft notebook filename with wildcards and params. (#1352) (11d4dc8)
• dummy release (e4dca50)
• efficient job status checking when using DRMAA API (this should yield much better parallelization and performance when using --drmaa) (#1156) (ac004cb)
• enable usage of job grouping in GLS (#1054) (d243c22)
• ensure that rule inheritance considers the same globals and other settings as parent module (#1621) (104cab9)
• ensures that REncoder also checks for numpy.bool_ in encode_value (#1749) (10a6e1d)
• env file usage after changes to source file handling (inspired by #1233 and #1211). (#1236) (3ac8e85)
• fail with error if conda installation is not set to strict channel priorities (#1672) (f1ffbf2)
• failure to properly apply default remote prefix in combination with the unpack marker (#1448) (82666f1)
• fix --immediate-submit (#1851) (e358372)
• fix a bug leading to duplicate conda env initializations; fix display of jobs and output files with changes (994b151)
• fix assertion error in conda env file spec when applying wildcards (thanks @ddesvillechabrol) (#1377) (6200652)
• Fix case of multiple scattergather processes (#1799) (417aad4)
• fix errors occurring when refering to input func via rules..input (#1669) (28a4795)
• fix falsely triggered reruns if input files are obtained via workflow.source_path() (#1862) (2dc2e6a)
• fix function call when creating report and hashes for between workflow caching (#1198) (a4f6836)
• fix issue when generating unit tests for rules with directory output (#1385) (7db614f)
• fix None type error when invoking Workflow object manually (#1366) (fca3895)
• fix path returned by get_source() (ee05315)
• fix pepfile handling in case of module usage (#1387) (f097a76)
• fix regression in workflow source acquisition of google life science executor (#1773) (c07732e)
• fix singularity logging messages causing conda fail (#1523) (7797595)
• fix source cache handling for remote source files retrieved via github() or gitlab() tags. (#1322) (6e2ecd2)
• fix source retrieval during between workflow caching and report generation (2394ca4)
• Fix technical bugs in resource-scope documentation (#1784) (878420c)
• fixed bug in missing input file handling for cluster jobs (#1544) (40e2eb2)
• fixed bug in needrun computation of jobs downsteam of checkpoints (#1704) (c634b78)
• fixed code change detection (#1513) (67298c6)
• fixed code change detection when using modules (#1264) (b571e09)
• fixed conda frontend detection and checking to also work with latest mambaforge (#1781) (225e68c)
• fixed detection of norun rules inside of modules (#1566) (d2223d4)
• fixed display of any exceptions and errors from within a workflow definition (23d40d9)
• fixed handling of input functions and unpack when using the prefix setting of module definitions (#1553) (d561e04)
• fixed parsing of subsequent use rule statements directly beneath each other (#1548) (77d5a08)
• fixed permission issues when using zenodo remote provider to access restricted depositions (#1634) (510f534)
• fixed temp file deletion for group jobs (#1487) (d030443)
• fixed typos (#1847) (a1e49b6)
• Ga4gh tes bugfixes (#1127) (af21d6c)
• handle benmark bug that arise with singularity (#1671) (10ef7c4)
• handle config file extension/overwriting more explicitly (#1251) (d0a7bf2)
• Handle temp files for all jobs in a group. (#1779) (d28b893)
• handling of remaining jobs when using --keep-going (#1693) (87e4303)
• implement lock-free source file caching. This avoids hangs on network file systems like NFS. (#1464) (9520e98)
• improve robustness when retrieving remote source files, fixed usage of local git repos as wrapper prefixes (in collaboration with @cokelaer and @Smeds) (#1495) (e16531d)
• improved display of percentage of done jobs (1fee8c0)
• improved error handling for cluster status scripts and smarter job selector choice in case of cluster submission (use greedy for single jobs). (#1142) (48d2dd9)
• improved error message in case of target rule misspecification (83b1f5b)
• improved provenance trigger info (#1720) (29d959d)
• in google live science backend, save multiple logs per rule name and overwrite existing logs (#1504) (9e92d63)
• in rules from imported modules, exclude modified paths from module prefixing (#1494) (1e73db0)
• Initialize assignments dictionary when setting rule-based resources (#1154) (68c13fd)
• iRODS functionality - issue #1510 (#1611) (9c3767d)
• issue #1069 (#1077) (7e34fda)
• Issue #1253 (problems editing Jupyter Notebooks) (#1255) (3398ddf)
• issue #1257 (missing logfile failure when using shadow directory) (#1258) (426d92f)
• issue 1615 - Switch formatting condition for dictionary (#1617) (0771062)
• issue with duplicated prefix for checkpoints on cloud (#1294) (8ed0c8c)
• issue1460 intermediate files existing (or not) (#1541) (1b3ede1)
• k8s container volume mounts as list (#1868) (5c54df3)
• keep empty output and input dirs of --draft-notebook job (f1181bd)
• keep flags with apply_wildcards on cloned IOFile (#1416) (23c943f)
• key error when handling FileNotFoundError in input functions. (#1138) (d25f04d)
• limit filename length of temporary files generated by the persistence backend (metadata, incomplete markers, etc.) (#1780) (59053e7)
• linting of remote snakefiles (#1131) (2104e10)
• merging of pipe groups when multiple rules are chained together via pipes (#1173) (de91d2c)
• modify dag and workflow display in report to also work for big DAGs (#1517) (1364dfb)
• more comprehensive error reporting for RuleExceptions (#1802) (1cd9512)
• more details on input and output exceptions (missing input, protected output, etc.) (#1453) (8d64af2)
• more informative nothing to be done message (#1234) (368d265)
• more robust calculation of number of jobs until ready for execution (#1691) (fdfc717)
• more robust handling of incompletely evaluated parameters (any interaction with them will result in a string now). (#1525) (3d4c768)
• more robust place for runtime source file cache (#1436) (2681f6f)
• move max_status_checks_per_second attribute setting before the wait thread of cluster backends is started to avoid missing attribute errors (#1775) (a48e9d0)
• Move quiet default after profile parsing (#1764) (6ade76d)
• mtime inventory for google storage was accidentally setting a float instead of a proper mtime object (#1484) (7c762c7)
• multiext prefix computation in case it is used within a module that defines an additional prefix (#1609) (fc6dfc6)
• None type error when invoking Workflow object manually (#1731) (dc45ccb)
• obtaining conda prefix when using in combination with singularity (#1535) (99b22d3)
• Only --bind Snakemake when we're working with a Python script (#1206) (1d79f62)
• only consider context of shell command for technical switches if called from snakemake rules. (#1213) (4816a58)
• only display a warning in case of non-strict channel priorities (#1752) (b84fa33)
• Open Snakefile for reading with explicit encoding specified (#1146) (ec1d859)
• parsing error when combining single line directive with multi-line directive in use rule statements (#1662) (26e57d6)
• pass triggers and resources to subworkflow (#1733) (fa7fb75)
• potential memory corruption caused by Google storage objects accessed from different threads (#1174) (41a5071)
• preserve dtypes across paramspace (#1578) (70ce6a0)
• preserve empty names input or output file lists in params or resource functions (0d19ab0)
• propagate rerun trigger info to cluster jobs; fix a bug leading to software stack trigger generating false positives in case of conda environments; fixed display of info message in case of provenance triggered reruns (#1686) (503c70c)
• proper error message if resource types do not match (#1556) (1112321)
• proper error message in case of missing git when checking for source files (#1534) (92887a3)
• proper error message when defining cache eligibility for rules with multiple output files and no multiext declaration. (#1357) (47b5096)
• properly use configfiles specified via CLI also if configfile specified via configfile directive is not present (1e0649a)
• properly use retry mechanism in source cache (#1564) (624a83d)
• provide details on error when failing to evaluate default resources (#1430) (04f39a9)
• provide proper error when using immediate submit in combination with checkpoint jobs. (#1437) (865cf0f)
• provide temporary IPYTHONDIR for notebook execution in order to avoid race conditions in https://github.com/ipython/ipython/blob/master/IPython/paths.py#L20 upon execution of multiple notebooks at the same time. (#1280) (4d70da1)
• quote jobid passed to status script to support multi-cluster Slurm setup (#1459) (0232201)
• quote workdir in job exec prefix to allow to spaces in the workdir (#1547) (c3a593e)
• R encoding of pathlib.Path objects (#1201) (bd516e9)
• reduce resource requirements for kubernetes tests (#1876) (cb4b78a)
• remove accidental pdb statement (9c935f1)
• remove deprecated and add missing arguments to internal functions (93a7e39)
• remove raise that limits using --config with dicts (#1341) (bd65057)
• remove redundant print (#1608) (cc7e0e3)
• remove superfluous comma causing TypeError in conda-frontend error message (#1804) (87b013c)
• remove whitespace before `` (#1067) (bd7951a)
• render empty caption if nothing defined in report flag (013a6e8)
• Repair MREs from #823 (#1203) (b007979)
• Replaced pathlib relative_to with os.relpath (#1505) (dc65e29)
• report error and possible cause if metadata cleanup fails (#1554) (6866134)
• request disk_mb resource from k8s (#1858) (f68f166)
• respect shebang lines in post-deploy scripts (see deployment docs) (#1841) (c26c4b6)
• rule inheritance within modules (did previously lead to key errors) (#1292) (603e0a8)
• run dependencies with non-existent ancient files before the consuming job (#1202) (84d1f64), closes #946
• SameFileError #1153 (#1220) (ede313d)
• set channel priority in container system wide (#1690) (41175b3)
• set default number of nodes to 1 in test cases (#1288) (f6e12b4)
• set mtime for cached source files [WIP] (#1443) (dd27209)
• skip global report caption when using a module (#1379) (a755cee)
• small bug in snakemake.executors (#1440) (6e64292)
• small changes to make docs checkpoint example functional (#1714) (1d4909e)
• snakemake API using only 1 job as default (#1283) (e92ad48)
• status cmd repeats until killed by 11 different signals (#1207) (8b28b57)
• support multiple input files for template_engine rules (#1571) (aee7cf2)
• throw error message in case of target rule that depends on a pipe. (#1532) (b9e9a7e)
• typo in sourcecache use (#1229) (8b54bc5)
• typos in code examples (#1324) (60010e4)
• update for minimum of Python 3.7 (#1509) (62024e2)
• Use 'snakemake.utils.update_config' instead of 'dict.update' (#1126) (2658027)
• use mambaforge for snakemake container image (#1595) (b7e6906)
• warn on non-file-modification-date changes like params, code, or input files (#1419) (b5f53f0)
• wms monitor arg parsing now accepts any kind of value (#1181) (313de93)
• work around segfault with >100 jobs in google life sciences backend (#1451) (2c0fee2)
• XRootDHelper.exists supports non posix filesystem (object store) (#1348) (7a3ad2f)

Performance Improvements

• improve job selection performance in case of potential ambiguity that is resolved by comprehensive ruleorder statements. (#1147) (921f4f7)
• more extensive caching of source files, including wrappers. (#1182) (bdb75f8)

Documentation

• add install info of development (git) version to docs (#1477) (2a2d6cd)
• add sentence about workflow template to docs (#1369) (5fabffb)
• add workflows.community metadata (#1736) (8a42afc)
• checkpoint documentation (#1562) (4cbfb47)
• Clarification of --cluster-stats docs & elaborating on the situation where job ids are not passed to the status script (#1221) (ed0e4a2)
• clarify namespacing when using modules. (dbed4a3)
• Clarify the use of conda with notebook directive (#1515) (aefb1eb)
• Combine CHANGELOG.rst with CHANGELOG.md (#1228) (19f5a43)
• Command line arguments for configuration files (#1343) (ad8aaa4)
• details on benchmarked results (64fea09)
• display rust-script env. (950d8ba)
• explain automatic decompression strategy for http remote provider (e6826b6)
• explain relative path interpretation (#1428) (add9a05)
• explain SNAKEMAKE_PROFILE environment variable (2b32bba)
• fix broken link in executor_tutorial/tutorial.rst (#1360) (c9be764)
• fix contributing executors link (#1112) (4bb58d1)
• fix grammar in the intro (#1859) (774bc6a)
• fix list display in docs (3724367)
• fix list display in docs (2dd0e91)
• fix long description type for pypi (set to markdown) (d8d9b8f)
• Fix problems with code blocks and broken internal link. (#1424) (5d4e7d8)
• fix small type in FAQ (#1832) (914172b)
• Fix syntax error in docs/conf.py and update sphinx.ext.napoleon import (#1084) (3e3fac2)
• fix tutorial setup instructions for MacOS. (#1383) (b57b749)
• fix typo (#1843) (6572ad9)
• Fix typo and grammar mistake in scatter-gather section. (#1441) (f218aaa)
• Fix typo in file path in remote files documentation (#1110) (9ce294f)
• fix typo in installation.rst (#1344) (c45d47a)
• Fix typo in rules.rst (—draft-notebook) (#1290) (f5c42cf)
• link to list of dependencies from installation (#1336) (99d7bfe)
• Mention required openssl dep for rust-script (#1216) (fc8c5f6)
• move note (75a544b)
• move psutil import into benchmark methods to avoid needing it as a dependency for doc building (6ffe38d)
• polish (47a7b62)
• require sphinx>=3 (1773875)
• separate api docs (ded7da9)
• separate api docs (#1499) (5cf275a)
• short tutorial updates (#1286) (b653a44)
• singularity sometimes uses system /tmp explanation (#1588) (170c1d9)
• skip lazy property (2883718)
• small tweaks to flux documentation (#1886) (f29b371)
• temaplte rendering examples and available variables (#1431) (5995e9e)
• tutorial formatting (594f5fb)
• Unpin docutils version (#1230) (15a82bf)
• update contribution docs (09a5595)
• update copyright year (#1427) (6b9f5da)
• update URL to emacs snakemake-mode (#1339) (dae7b8f)
• various little fixes (#1875) (b93f8e3)
• zenodo example (76159ae)

---

This PR was generated with Release Please. See documentation.