chore(main): release 8.0.0

[github-actions]

🤖 I have created a release beep boop

8.0.0 (2023-12-06)

⚠ BREAKING CHANGES

• redesigned Snakemake API. It now uses a modern, dataclass based approach (#2403)

Features

• ability to encode paramspaces into a single wildcard, via the newly introduced single_wildcard argument of Paramspace. (#2069) (728ab3c)
• add keyword to gridftp remote provide to specify the number or disable usage of multiple data stream (#1974) (3e6675d)
• add ability to inject conda environments into running Snakefile (#2479) (6140e29)
• add Bash script functionality (#1821) (c4cf8fd)
• add cleanup containers option (#2088) (053e3b3)
• add flag ensure that allows to annotate that certain output files should be non-empty or agree with a given checksum (#1651) (76f69d9)
• add functionality for deploying sources if no shared FS is assumed (#2486) (76eac3c)
• add gitfile option to make it possible to use local git repos when importing modules (#1376) (1a3b91f)
• add option to control software deployment mode (shared or non shared FS) (#2525) (04ec2c0)
• add pyproject.toml to use setuptools features (#1725) (454bfd1)
• Add support for Google Service Accounts and GCE VM network configuration (#2318) (2b754aa)
• add support for Kubernetes service account name spec (#2254) (3370426)
• Add support for tokens in the XRootD interface (#2076) (bca013a)
• add tes token (#1966) (59a8fa0)
• Add token auth to GitLab/GitHub hosting providers (#1761) (e03a3b4), closes #1301
• added localrule directive (#2180) (9c990b0)
• adding flux executor (#1810) (40d2bd0)
• allow config files to be processed with YTE (#2269) (8e1c22f)
• allow detailed configuration of shared FS usage (#2528) (0d34be9)
• allow environment variables in string values of profile (e.g. paths may now contain elements like $USER). (58dc70c)
• allow for human friendly resource definitions (e.g. mem="5GB", runtime="1d") (#1861) (24610ac)
• allow input, output, and params to be used in functions passed to report mark arguments (#2081) (93ff8b6)
• Allow paramspace to separate filename params with custom separator (#1299) (8236e80)
• allow profiles to be YTE templates; adapt to eido 2.0 (#2325) (67d9ff2)
• allow python expressions in --set-resources (#2521) (022a31e)
• Allow the environment variable SNAKEMAKE_CONDA_PREFIX to be present without --use-conda (#2263) (e4eba8d)
• allow to define the cache mode per rule (this enables to exclude software envs from the caching hash value, which can be handy e.g. for download rules where the software version does not affect the result) (#1933) (715e618)
• allow to set latency_wait in executor test suite (c0bca0b)
• automatically rerun jobs if parameters, code, input file set, or software stack changed (thanks to @cclienty and @timtroendle). This also increases performance of DAG building by handling job "needrun" updates level wise, while avoiding to perform a full check for those jobs that are already downstream of a job that has been determined to require a rerun. (#1663) (4c11893)
• automatically upload workflow sources to default storage provider if no shared FS is used (a450c49)
• Azbatch executor (#1953) (#2246) (0f9c49f)
• changed report layout to display menu always left of the results. For fullscreen, one can still hide the menu, which leads to automatic growth of the results (#2116) (d771b1b)
• enable restricted access support in zenodo remote provider (#1623) (692caf9)
• enable the definition of conda pin files in order to freeze an environment. This can drastically increase the robustness because it allows to freeze an environment at a working state. (#1667) (53972bf)
• Faster ci test setup (#2489) (4798e8a)
• first try to match output files against input files while persisting wildcard values from the consuming job. This can dramatically reduce ambiuity problems. Thanks to @descostesn! (#1939) (d093907)
• for small files, compare checksums to determine if job needs to run if input file is newer than output file (#1568) (1ae85c6)
• Group resources (#1218) (a8014d0)
• implement precommand (#2482) (ff0f979)
• introduce --workflow-profile for additional workflow specific profiles that overwrite global profiles; add ability to define key-value CLI flags like --set-threads or --set-resources as multi-level dictionaries in profile config yaml files (#2310) (9675c17)
• k8s: add --k8s-cpu-scalar (#1857) (a067a1b)
• limit the number of input/output files in job properties (#2149) (d93f091)
• LockException (#1276) (f5e6fa6)
• make it possible to exclude rules that will be imported when using 'use rule' statement (#1717) (d9e0611)
• new directive "retries" for annotating the number of times a job shall be restarted after a failure (#1649) (c8d81d0)
• prepare for storage plugins (#2454) (353a71c)
• print reason summary in case of dryrun (#1778) (bd2a68b)
• provide information about temp, pipe, and service files in --summary (#1977) (c7c7776)
• Publish docker images for amd64 & arm64 (#2105) (4c898f5)
• redesigned Snakemake API. It now uses a modern, dataclass based approach (#2403) (2be3bfa)
• SLURM executor (#1015) (c7ea059)
• Support conda environment definitions to be passed as function pointers, similar to input, params, and resources (#1300) (6f582f1)
• support for external executor plugins (#2305) (c9eaa4e)
• Tes auth (#2169) (3326a6f)
• version specific profile config files (profile/config.v8+.yaml with profile/config.yaml as fallback that matches any version) (#2498) (47e5811)
• vim syntax updates (#1584) (b8c77f6)

Bug Fixes

• 🐛 - fix hyperlink (#2046) (9519d31)
• #2130 by patching the protect() method so the path of files in subdirectories is properly resolved during write-protection (#2131) (1a754fd)
• sre_constants import because of deprecation (#2139) (3b326db)
• 2142 log directive in default target rule (#2191) (86e9624)
• adapt linting rule to Python 3.11 (a3a5c58)
• adapt to changes in snakemake-interface-executor-plugins (635c68a)
• Add back logging of run directives (#1883) (a65559c)
• add lock free mechanism for avoiding race conditions when writing persistence information; consider corrupt metadata records as non-existent (#1745) (71fe952)
• add missing spaces between lines that get concatenated. (#2268) (7238458)
• add storage provider args to deploy sources command (67178e3)
• addressing #2197 by allowing 256 character account names in slurm (#2198) (ab58c65)
• allocation of local ssds in k8s tests (#1870) (d0de4dc)
• allow apptainer as a successor to singularity. (#1706) (bcbdb0b)
• allow labels function to return None (#1565) (fef74d6)
• allow report generation to handle pathlib objects (#1904) (7c34656)
• allow script directive to take pathlib Path (#1869) (12cdc96)
• allow to use {wildcards} for group jobs in cluster config (#1555) (f0ec73d)
• also inherit rule proxies if there is no rulename modifier specified in a use rule statement (#2440) (1570289)
• always make sure that the original path of source cached files is properly passed into metadata persistence records (#2179) (8bacbd0)
• always recalculate job resources before job is scheduled as input might have changed or not have been present initially (#1552) (44aacdb)
• assume at most 8GB memory for default resources. This way, we avoid exploding memory requirements for large input files that are very unlikely to be put entirely into memory by any tool. (11c2ecc)
• assume shared filesystem by default when running with --flux (#2075) (4bec2fd)
• avoid "Admin" prompt when using conda on windows (#1046) (552fadf)
• avoid erroneous too early deletion of parent directories in case of failed jobs (thanks to @SichongP). (#1601) (b0917e6)
• avoid logfile writing in case of dryrun; better hints in case of incomplete checkpoints (#1994) (a022705)
• batch collect jobs for scancel (#2114) (0b1fe31)
• better job status queries for slurm executor (#2136) (a4df38c)
• better message about profile usage upon execution (#2391) (cf8aea5)
• Bump yte from >=1.0,<2.0 to >=1.5.1,<2.0 (#2275) (8c0b34f)
• catch errors in remote.AUTO provider list (#1834) (c613ed2)
• Catch missing error stream in Slurm executor (#2063) (c21fc7e)
• Change ratelimiter dependency to throttler (#1958) (50b8f16)
• change source cache entries to keep the original name and folder structure, such that imports from e.g. scripts also work with remote modules (if specified as additional input files with workflow.source_path) (#1936) (c34f3f6)
• check for skipped rules in case of local rule inheritance (#1631) (9083ac1)
• Comparison of rules and non-rule instances (#1894) (bf01ece)
• comparison to float in scheduler (ef44d84)
• conda create --no-shortcuts absent on Linux/MacOS (regression from #1046) (#1916) (8a86a1e)
• conda env inside script (#1812) (49cac6a)
• conda python interpreter path on Windows (#1711) (155c9d6)
• Consider source cache when setting search path for python scripts. This allows to import from Python modules next to scripts while deploying the workflow as a snakemake module, even from remote locations. (#1940) (27be1d4)
• consistently use text output in conda shell commands and various little fixes for failing test cases due to conda package changes (#1864) (4234fe7)
• correctly parse empty values in config cli (#2032) (1b0689d)
• Correctly parse UserDicts in executors (#2016) (e3926fa)
• declare associative arrays (#1844) (90ae449)
• delay evaluation of tmpdir to actual job execution, and not submission. This way, tmpdir can be dependent on the node context. (#1860) (4203556)
• detect job paths that leave and then enter a group. Such paths are invalid because then the group depends on itself. (#2527) (5383a4d)
• Detect pandas availability to select serializer (#2300) (e08a771)
• do not allow setting benchmark and between-workflow caching for the same rule. The reason is that when the result is taken from cache, there is no way to fill the benchmark file with any reasonable values. (#2335) (e2d64fa)
• do not overwrite default resources setting in azure batch executor (#2395) (4aef3b9)
• do not remove existing temp files in case of dryrun (#1543) (e820f97)
• do not wrap whitespace in result info headers of reports (653d0d0)
• Enable values with an = sign in default_resources (#2340) (c1c9229)
• ensure lazy evaluation of resource functions/callables (this also entails, for now, a removal of the thread statistics in the yellow job stats table); further, added some clarifying sentences about resource function evaluation to the docs (#2356) (4c591b7)
• ensure that auto deployment of default storage provider works in containers with read-only root home. (1a347ff)
• ensure that rule inheritance considers the same globals and other settings as parent module (#1621) (104cab9)
• ensure that rule name string instead of object is passed to tabulate package (#1898) (f9ff157)
• ensure that targetjob is always forced. This fixes a bug causing run-directive rules to not being executed even when enforced via e.g. -R. (#2448) (b2a60d5)
• ensure user and group rw permissions for metadata files and source cache (#2132) (cc51faa)
• ensures that REncoder also checks for numpy.bool_ in encode_value (#1749) (10a6e1d)
• Escape workdir paths for potential spaces in paths (#2196) (9261f7e)
• fail with error if conda installation is not set to strict channel priorities (#1672) (f1ffbf2)
• fix --immediate-submit (#1851) (e358372)
• fix cache handling and unlock handling (2f4d5e1)
• Fix case of multiple scattergather processes (#1799) (417aad4)
• fix errors occurring when refering to input func via rules.<rulename>.input (#1669) (28a4795)
• fix false rerun triggering downstream of checkpoints due to spurious parameter, code or software env changes (638ea86)
• fix false reruns after checkpoints (#1907) (dc5af12)
• fix falsely triggered reruns if input files are obtained via workflow.source_path() (#1862) (2dc2e6a)
• Fix handling of --jobs in no-exec state (#2029) (e8e8222)
• Fix inconsistencies between detailed summary and normal summary (#2218) (d903123)
• fix index out of bounds error raised by usage of workflow.source_path called from input or params functions (thanks @AKBrueggemann) (#2170) (cf8e6e8)
• fix missed wildcard constraints when using local rule inheritance (#2242) (8e94785)
• fix nargs definition for --deploy-sources (fc252c8)
• fix pandas import handling in metadata persistence (27f7b40)
• fix path handling when detective profiles (fe63881)
• Fix race condition when creating lock directory (#2225) (66ea4d1)
• fix regression in workflow source acquisition of google life science executor (#1773) (c07732e)
• fix scrolling behavior in landing page of report for large workflows (63c0c31)
• fix storage handling on windows by converting all paths to posix paths (#2519) (7864a76)
• Fix technical bugs in resource-scope documentation (#1784) (878420c)
• fix typo in line display of exceptions (#1912) (55e38a6)
• fixed bug in missing input file handling for cluster jobs (#1544) (40e2eb2)
• fixed bug in needrun computation of jobs downsteam of checkpoints (#1704) (c634b78)
• fixed conda frontend detection and checking to also work with latest mambaforge (#1781) (225e68c)
• fixed detection of norun rules inside of modules (#1566) (d2223d4)
• fixed handling of input functions and unpack when using the prefix setting of module definitions (#1553) (d561e04)
• fixed parsing of subsequent use rule statements directly beneath each other (#1548) (77d5a08)
• fixed permission issues when using zenodo remote provider to access restricted depositions (#1634) (510f534)
• fixed problem with leaked modifications when inheriting multiple times from the same rule (#1957) (2475cbc)
• fixed typos (#1847) (a1e49b6)
• forwarding --keep-incomplete to cluster executor (#1951) (2894c7d)
• ga4gh executor resources (#2042) (ad6eaef)
• get python version for script environment in a backwards compatible way that works down to python 2.7 (#2161) (44e59b9)
• handle benmark bug that arise with singularity (#1671) (10ef7c4)
• handle case where zenodo deposition does not return files (#2004) (b63c4a7)
• handle different f-string tokens in py3.12 (#2485) (f2c7613)
• handle non-PEP440 versions of apptainer/singulariy (#2337) (dea6ba8)
• Handle temp files for all jobs in a group. (#1779) (d28b893)
• handling of group jobs when obtaining temp input files (71be1de)
• handling of remaining jobs when using --keep-going (#1693) (87e4303)
• import (#2402) (2c831f1)
• improved default resources parsing (also allowing to deactivate a default resource via setting it to None) (#2006) (e6cdb32)
• improved provenance trigger info (#1720) (29d959d)
• iRODS functionality - issue #1510 (#1611) (9c3767d)
• is_run error with local, group jobs (#2133) (31bfcd5)
• issue #1882 WorkflowError: Metadata can't be created as it already exists (Windows) (#1971) (d4484e6)
• issue 1615 - Switch formatting condition for dictionary (#1617) (0771062)
• issue 1846 (#1888) (da2dfbd)
• issue1460 intermediate files existing (or not) (#1541) (1b3ede1)
• json validation error with markdown cells (#1986) (6c26f75)
• k8s container volume mounts as list (#1868) (5c54df3)
• lexicographically sorted rule display with --list, and trimmed rule docstrings (#1880) (32128ae)
• limit filename length of temporary files generated by the persistence backend (metadata, incomplete markers, etc.) (#1780) (59053e7)
• limit length of failed logs decorations (#2125) (6fc9243)
• make --show-failed-logs handle empty log files (#2039) (683c6f2), closes #2023
• make python version check more robust (#2058) (e685621)
• more comprehensive error reporting for RuleExceptions (#1802) (1cd9512)
• more robust calculation of number of jobs until ready for execution (#1691) (fdfc717)
• more robust encoding of params in persistent metadata storage. This way, pandas parameters do not lead to spurious rerun triggers. (#2080) (106a4c3)
• more robust parsing of sacct output in slurm executor (#2036) (fe651f8)
• move max_status_checks_per_second attribute setting before the wait thread of cluster backends is started to avoid missing attribute errors (#1775) (a48e9d0)
• Move quiet default after profile parsing (#1764) (6ade76d)
• multiext prefix computation in case it is used within a module that defines an additional prefix (#1609) (fc6dfc6)
• None type error when invoking Workflow object manually (#1731) (dc45ccb)
• only consider global wildcard_constraints from the same module (#2235) (c412b71)
• only deactivate conda inject envs upon workflow tear down (#2503) (e6dfdd4)
• only display a warning in case of non-strict channel priorities (#1752) (b84fa33)
• Open Snakefile for reading with explicit encoding specified (#1146) (ec1d859)
• Panoptes --wms-monitor-arg (#2444) (98d2bdf)
• parsing error when combining single line directive with multi-line directive in use rule statements (#1662) (26e57d6)
• parsing error when last line is comment (#2054) (a928dd4)
• pass triggers and resources to subworkflow (#1733) (fa7fb75)
• Postprocess job groups in toposorted order for correct touch times (#2073) (10b5849)
• preserve dtypes across paramspace (#1578) (70ce6a0)
• prevent overriding of retries when set to 0 (#2053) (a328f3e)
• prevents DeprecationWarning caused by using old draft of json schema (#2152) (9791ffb)
• print exceptions when job is not a shell job (#2385) (8a37b85)
• propagate attempt count from group to subjobs (#2052) (da3f1c0)
• propagate rerun trigger info to cluster jobs; fix a bug leading to software stack trigger generating false positives in case of conda environments; fixed display of info message in case of provenance triggered reruns (#1686) (503c70c)
• proper error message if resource types do not match (#1556) (1112321)
• proper reuse of rule proxies when importing several times from the same module (#2404) (e867dda)
• properly handle NA values for paramspaces (#2098) (6b6a880)
• properly use configfiles specified via CLI also if configfile specified via configfile directive is not present (1e0649a)
• properly use retry mechanism in source cache (#1564) (624a83d)
• quote paths given to singularity in order to ensure that it does not fail when paths contain whitespace (#2190) (a572fb7)
• quote workdir in job exec prefix to allow to spaces in the workdir (#1547) (c3a593e)
• raise error if callable is passed to expand. (#2171) (1f28476)
• reduce resource requirements for kubernetes tests (#1876) (cb4b78a)
• regression ValueError introduced with 7.17.2 (#1947) (53a4fca)
• remote GS builds too many inventories; io:collect_mtime always uses uncached mtime (#2266) (bad9115)
• remote-azblob-sasToken-Authorization (#1800) (bc854a7)
• remove overflow from rulegraph div in report (9a0aaa7)
• remove redundant dot in expand call in multiext documentation (#1921) (278beaa)
• remove redundant print (#1608) (cc7e0e3)
• remove superfluous comma causing TypeError in conda-frontend error message (#1804) (87b013c)
• remove superfluous dependency (aad61a0)
• removed distutils from snakemake (#2312) (9b8c362)
• report error and possible cause if metadata cleanup fails (#1554) (6866134)
• report spacing (f3954b3)
• request disk_mb resource from k8s (#1858) (f68f166)
• require python >=3.7 again (the python 3.9 dependency was unnecessary) (#2372) (0d0e9c4)
• require toposort >= 1.10 (#2145) (3cb54b8)
• respect shebang lines in post-deploy scripts (see deployment docs) (#1841) (c26c4b6)
• Restore backward compatibility for Google Life Sciences executor (#2461) (5e3a464)
• rounding for batch calculation (#2064) (cbdbf9b)
• set channel priority in container system wide (#1690) (41175b3)
• shadow "full" mode ignore symlinks (#2516) (1d58120)
• show failed logs in executor testcases (92f7bf4)
• show input files on job error (#1949) (ad21631)
• skip type checks of missing dir in touch mode (#2051) (ae00c25)
• slurm batch job status queries (#2167) (0bb69e4)
• slurm default_resources quoting (#2043) (47d3fc3)
• small changes to make docs checkpoint example functional (#1714) (1d4909e)
• Solve apptainer version issue (#2333) (a876e0f)
• sort report (sub-)categories in lexicographical order (#2449) (d0705ad)
• support multiple input files for template_engine rules (#1571) (aee7cf2)
• SyntaxWarnings due to non-raw regex pattern strings (#2359) (a08c0b0)
• unnecessary set Snakefile in AzBatch executor (#2397) (78e6d6e)
• Update init.py to move "file" param to "print" (#2291) (92352b6)
• Update list of python versions in classifiers (#2020) (7a98100)
• update minimum snakemake-interface-storage-plugins version (0ef7226)
• updating of non-dict config values gives error (#2364) (b33aeec)
• Use job.rule.name attribute to fill rule field in summary (#2217) (837c3fd)
• use mambaforge for snakemake container image (#1595) (b7e6906)
• use short argument name for --chdir for compatibility with Slurm <=v17 (#2040) (a9ed3ec)
• use temporary directory (faster, more likely local, always writable) for persistence and source cache in case of remote execution without shared fs (#2502) (c8fa7ba)
• use text/markdown for long_description_content_type (#2112) (0241075)
• wait for logs before showing them on error (a4ff328)
• wms-monitor now gets data in correct json format (#2347) (7fafa7a)
• wrong rule names when nesting module imports (#1817) (65c79a4)

Performance Improvements

• Average NamedList getitem performance improvement (#1825) (10451b7)
• avoid superflous mtime checks when the same file is referred to by multiple jobs (#2284) (eb6e2e1)
• cached os.pathconf() call in _record_path() (#1920) (551badb)
• Gfal2 remote provider using gfal2-python instead of gfal2-utils. (#2128) (0b9bfe5)
• Improve execution speed of cleanup_workdir (in dag) (#2103) (1fbc5f5)

Documentation

• add workflows.community metadata (#1736) (8a42afc)
• added changelog info for >v7.19.1 parsing error of "hh:mm:ss" time format in runtime resource (#2189) (2889f38)
• basics.rst: suggest VS Code instead of deprecated Atom as IDE (#2368) (1357316)
• Change snakemake-tutorial download link to always be the latest (#2183) (ae8a8f4)
• checkpoint documentation (#1562) (4cbfb47)
• clarify minimum Snakemake version for profiles (86dc277)
• clarify the channel priority in environment definition deployment.rst (#2352) (76aa964)
• document name directive with example (cd49c34)
• explain SNAKEMAKE_PROFILE environment variable (2b32bba)
• fix a copy&paste (?) mistake (#2386) (d878847)
• fix formatting (087fe63)
• fix grammar in the intro (#1859) (774bc6a)
• fix link (4889c93)
• fix long description type for pypi (set to markdown) (d8d9b8f)
• fix minor typos in a linting rule (#2162) (71e1171)
• fix small type in FAQ (#1832) (914172b)
• fix statement about logging (#2252) (56c24b6)
• fix syntax in cluster example (#2460) (64e9645)
• fix typo (e1c3cc6)
• fix typo (#1843) (6572ad9)
• fix typo (stackoverflow issue) (#2365) (f770984)
• Fix typo in SLURM help text (#2049) (79b7025)
• fix typos (e45b9e6)
• fix typos (151095d)
• fix typos in --help (#2182) (09f0cbe)
• format table (4180a1b)
• Improve error message when rule contains multiple run/shell/script/notebook/wrapper/template_engine/cwl keywords (#2186) (cd5a3c4)
• mention XDG_CACHE_HOME (#2057) (ec2ef45)
• note on using checkpoint mechanism only for input function, not for params or resources. (#2353) (4be2f9d)
• notes on arm based machines in tutorial docs (0586f04)
• polished text and table display (413356c)
• replace nosetest with pytest (#2211) (f6b3c47)
• rust: Fix typo on rust-script version (#2488) (a79dd94)
• singularity sometimes uses system /tmp explanation (#1588) (170c1d9)
• small tweaks to flux documentation (#1886) (f29b371)
• update CHANGELOG.md: add minimum Python version bump (#2370) (48e934d)
• update contribution docs (09a5595)
• update docs for azbatch and dockerhub ref (#2298) (908dbf1)
• update misc/vim/Readme with info for packer.nvim (#2095) (32166a7)
• Update modularization.rst (#2137) (16954c7)
• Update workflow syntax with priority directive (#2188) (af10db5)
• various little fixes (#1875) (b93f8e3)

---

This PR was generated with Release Please. See documentation.