Viash 0.9.0-RC4

Viash 0.9.0-RC4 (2024-05-29): Improvements for CI

These are mainly improvements for issues highlighted by running Viash in a CI environment.
Additionally, implemented a proper way of caching dependency repositories. The cache is stored under ~/.viash/repositories.

NEW FUNCTIONALITY

• dependencies: GitHub and ViashHub repositories now get properly cached (PR #699).
  The cache is stored in the ~/.viash/repositories directory using sparse-checkout to only fetch the necessary files.
  During a build, the cache is checked for the repository and if it is found and still up-to-date, the repository is not cloned again and instead the cache is copied to a temporary folder where the files are checked out from the sparse-checkout.

• resources_test: Add a resources_test field to the _viash.yaml to specify resources that are needed during testing (PR #709).
  Currently it is up to the user or CI to make sure these resources are available in the resources_test directory during testing.

BUG FIXES

dependencies: Fix resolving of dependencies of dependencies (PR #701). The stricter build config was now lacking the necessary information to resolve dependencies of dependencies.
We added it back as .build_info.dependencies in a more structured, anonymized way.

dependencies: Fix the name field of repositories possibly being outputted in the build config (PR #703).

symlinks: Allow following of symlinks when finding configs (PR #704). This improves symlink functionality for viash ns ... and dependency resolving.

Viash 0.9.0-RC3

Viash 0.9.0-RC3 (2024-04-26): Various bug fixes and minor improvements

Mainly fixes for code changes from previous release candidates. Some additional minor fixes and QoL improvements are included.

BREAKING CHANGES

• computational requirements: Use 1000-base units instead of 1024-base units for memory (PR #686). Additionally, the memory units kib, mib, gib, tib, and pib are added to support 1024-base definitions.

MINOR CHANGES

• error message: Improve the error message when using an invalid field in the config (#PR #662). The error message now includes the field names that are not valid if that happens to be the case or otherwise a more general error message.

• config mods: Improve the displayed error message when a config mod could not be applied because of an invalid path (PR #672).

BUG FIXES

• export json_schema: Fix minor inconsistencies and make the strict schema stricter by adapting to what Viash will effectively return (PR #666).

• deprecation & removal warning: Improve the displayed warning where a deprecated or removed field could display a double '.' when it field was located at the root level (PR #671).

• resource path: Don't finalize the path field of a resource until it's written as part of building a component (PR #668).

• requirements: Improve the error message when a Python or R requirement uses a single quote in the .script field (PR #675).

• viash test: Fix Docker id between build and test components not being consistent when using a custom Docker registry (PR #679).

• MainNSBuildNativeSuite: Capture the error message when reading the configs so we can capture the expected warning message (PR #688).
  While almost all tests were already cleanly capturing their expected warning/error messages, this one was still remaining, resulting in warnings being shown in the output.

• runners & engines: When applying a filter on empty runners or engines, the fallback default native engine and executable runner respectively are set before applying the filter (PR #691).

Viash 0.8.6

Viash 0.8.6 (2024-04-26): Bug fixes and improvements for CI

Fix some issues in some edge cases.
Add options for testing in a CI environment. Given that these options are not meant for general use, they are hidden from the help message.
Some improvements are made to run in Nextflow Fusion.

DOCUMENTATION

• docker setup strategy: Fix inconsistencies in the documentation (PR #657).

• repositories: Fix uri -> repo in the repositories documentation (PR #682).

NEW FUNCTIONALITY

• viash test and viash ns test: Add a hidden --dry_run option to build the tests without executing them (PR #676).

• viash test and viash ns test: Add a hidden --deterministic_working directory argument to use a fixed directory path (PR #683).

• component names: Verify that component namespace and name combinations are unique (PR #685).

BUG FIXES

• NextflowPlatform: Fix publishing state for output arguments with multiple: true (#638, PR #639).

• Executable: Check whether a multiple output file argument contains a wildcard (PR #639).

• NextflowPlatform: Fix a possible cause of concurrency issues (PR #669).

• Resources: Fix an issue where if the first resource is not a script, the resource is silently dropped (PR #670).

• Docker automount: Prevent adding a trailing slash to an automounted folder (PR #673).

• NextflowPlatform: Change the at-runtime generated nextflow process from an in-memory to an on-disk temporary file, which should cause less issues with Nextflow Fusion (PR #681).

Viash 0.9.0-RC2

Viash 0.9.0-RC2 (2024-02-23): Restructure the config and change some default values

The .functionality layer has been removed from the config and all fields have been moved to the top layer.
The default multiple_sep has been changed from : to ; to avoid conflicts with paths like s3://foo/bar.

BREAKING CHANGES

• arguments: Change default multiple_sep from : to ; to avoid conflicts with paths like s3://foo/bar (PR #645).
  The previous behaviour of using multiple_sep: ":" can be achieved by adding a config mod to the _viash.yaml:

  config_mods: |
    .functionality.argument_groups[true].arguments[.multiple == true].multiple_sep := ":"

• functionality: Remove the functionality layer from the config and move all fields to the top layer (PR #649).

MINOR CHANGES

• package config: Renamed project config to package config (PR #636). Now that we start using the config more, we came to the conclusion that "package" was better suited that "project".

• ns exec: Added an extra field {name} to replace {functionality-name} (PR #649). No immediate removal of the old field is planned, but it is deprecated.

• BashWrapper: Added meta-data field meta_name as a replacement for meta_functionality_name (PR #649). No immediate removal of the old field is planned, but it is deprecated.

BUG FIXES

• schema: Don't require undocumented fields to set default values and add the links and reference fields to functionality as they were not meant only to be in the project config (PR #636).

Viash 0.8.5

Viash 0.8.5 (2024-02-21): Bug fixes and documentation improvements

Fix a bug when building a test docker container which requires a test resource. Additional improvements for the website documentation and support for the latest version of Nextflow are added.

BUG FIXES

• nextflow runner: Fix an issue with current nextflow-latest (24.01.0-edge) where our supporting library passes a GString instead of a String and results in a type mismatch (PR #640).

• test resources: Make non-script test resources available during building of a docker container for viash test (PR #652).

DOCUMENTATION

• repositories: Improve the repository documentation with links and a overview table with links (PR #643).

Viash 0.9.0-RC1

Viash 0.9.0-RC1 (2024-01-26): Restructure platforms into runners and engines

This release restructures the platforms field into runners and engines.
Additionally changes are made to sanitize the built config output and include additional relevant meta data.

BREAKING CHANGES

• runners and engines: The usage of platforms is deprecated and instead these are split into runners and engines (PR #510).
  The platforms field is still supported but will be removed in a future release.
  In brief, the native platform became a native engine and docker platform became a docker engine.
  Additionally, the native platform and docker platform became a executable runner, nextflow platform became a nextflow runner.
  The fields of docker platform is split between docker engine and docker runner: port, workdir, setup_strategy, and run_args are captured by the runner as they define how the component is run. The other fields are captured by the engine as they define the environment in which the component is run. One exception is chown which is rarely set to false and is now always enabled.

• arguments: Merge arguments into argument_groups during a json decode prepare step (PR #574). The --parse_argument_groups option from ns list and config view is deprecated as it is now always enabled.

NEW FUNCTIONALITY

• export json_schema: Add a --strict option to output a subset of the schema representing the internal structure of the Viash config (PR #564).

• config view and ns list: Do not output internal functionality fields (#564). Additionally, add a validation that no internal fields are present when reading a Viash config file.

• project config: Add fields in the project config to specify default values for component config fields (PR #612). This allows for a more DRY approach to defining the same values for multiple components.

MINOR CHANGES

• testbenches: Add testbenches for local dependencies (PR #565).

• testbenches: Refactor testbenches helper functions to uniformize them (PR #565).

• logging: Preserve log order of StdOut and StdErr messages during reading configs in namespaces (PR #571).

• Java 21 support: Update Scala to 2.13.12 and update dependencies (PR #602).

• project config: Output the project config under the default name ProjectConfig instead of Project during schema export (PR #631). This is now important as the project config is now part of the component config. Previously this was overridden as the class name was ViashProject which was less descriptive.

BUG FIXES

• __merge__: Handle invalid yaml during merging (PR #570). There was not enough error handling during this operation. Switched to the more advanced Convert.textToJson helper method.

• config: Anonymize paths in the config when outputting the config (PR #625).

Viash 0.8.4 (2024-01-15): Bug fix

Fix building components with dependencies that have symlinks in their paths.

BUG FIXES

• dependencies: Fix dependencies with paths using symlinks (PR #621). The resolution for the .build.vsh was changed to use the toRealPath previously, so dependency resolution must take account possible resolved symlinks too.

Viash 0.8.3 (2024-01-08): Bug fixes

Viash 0.8.3 (2024-01-08): Bug fixes

Fix some edge cases and improve usability.

BUG FIXES

• NextflowPlatform: properly resolve paths when a nextflow workflow has another nextflow
  workflow as dependency and the worktree contains a directory that is a symlink (PR #614).

• Main: Fixes a bug added by #294 which causes Viash to print a stacktrace instead of a helpful error message when viash is run without any arguments (#617, PR #618).
  Thanks @mberacochea for pointing out this oversight!

• Dependency: When an alias is defined, pass the alias as a key to the .run() function (#601, PR #606).

Viash 0.8.2

Viash 0.8.2 (2023-12-14): Minor changes and bug fixes

This release fixes a few bugs regarding dependencies and how the Nextflow platform handles Paths.

MINOR CHANGES

• NextflowTestHelper: Do not hardcode a version of Nextflow in the testbench,
  but use the version of Nextflow that is installed on the system (PR #593).

• GitHub Actions: Test different versions of Nextflow (22.04.5, latest, and latest-edge) (PR #593).
  Testing the latest Edge version of Nextflow will allow us to catch notice changes in Nextflow earlier.

• Updates to the documentation and templates in the Git repo (#598, PR #600):

  • Add contributing guidelines.

  • Add issue templates.

  • Reworked the pull request template.

BUG FIXES

• config: Fix the main level of a component config not enforcing strict mode and instead allowing any field to be specified (PR #585).

• dependencies: Allow the user to define a local dependency with specifying repository: local as sugar syntax (PR #609). A local repository is the default value so it's not required to be filled in, but allowing it with a sensible sugar syntax makes sense.

• Repositories: Fix a structural issue where a repository defined directly in a dependency would require the name field to be set (PR #607). Repository variants are created with and without the name field. Repositories under .functionality.dependencies[] use repositories without the name field, while repositories under .functionality.repositories[] use repositories with the name field.

• NextflowPlatform: Do not resolve remote paths relative to the --param_list file (PR #592).

• NextflowPlatform: Allow finding .build.yaml file when worktree contains a directory that is a symlink (PR #611).

Viash 0.8.1

Viash 0.8.1 (2023-11-20): Minor bug fix to Nextflow workflows

This release fixes a bug in the Nextflow platform where calling a workflow with the .run() function without specifying the fromState argument would result in an error when the input channel contained tuples with more than two elements.

BUG FIXES

NextflowPlatform: Fix error when using .run() without using fromState and the input channel holds tuples with more than two elements (PR #587).