Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add the Loop extensions #190

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

add the Loop extensions #190

wants to merge 1 commit into from

Conversation

mr-c
Copy link
Contributor

@mr-c mr-c commented Feb 3, 2023 •
edited

No luck at https://workbench2.pirca.arvadosapi.com/processes/pirca-xvhdp-qr0weevqxf9pngw

arvados-cwl-runner --match-submitter-images  --submit-runner-image arvados/jobs:2.6.0.dev20230203163323_loops tests/loop/multi-source-loop.cwl tests/loop/single-var-loop-job.yml

INFO /home/michael/src/arvados/env3.10/bin/arvados-cwl-runner 2.6.0.dev20230203163323, arvados-python-client 2.5.0, cwltool 3.1.20221008225030
INFO Resolved 'tests/loop/multi-source-loop.cwl' to 'file:///home/michael/cwltool/tests/loop/multi-source-loop.cwl'
tests/loop/multi-source-loop.cwl:60:7: object id `tests/loop/multi-source-loop.cwl#loop/i1` previously defined
WARNING Workflow checker warning:
tests/loop/multi-source-loop.cwl:61:11: Source 'osmall' of type {"type": "array", "items": ["null",
                                        "int"]} may be incompatible
tests/loop/multi-source-loop.cwl:15:5:    with sink 'o1' of type {"type": "array", "items": "int"}
                                          source has linkMerge method merge_flattened
                                          pickValue is: all_non_null
tests/loop/multi-source-loop.cwl:61:19: Source 'obig' of type {"type": "array", "items": ["null",
                                        "int"]} may be incompatible
tests/loop/multi-source-loop.cwl:15:5:    with sink 'o1' of type {"type": "array", "items": "int"}
                                          source has linkMerge method merge_flattened
                                          pickValue is: all_non_null
INFO Git provenance
INFO   gitCommit: f79bce163721e352b50621d68769699252ab320b
INFO   gitDate: Sat, 8 Oct 2022 18:50:30 -0400
INFO   gitCommitter: GitHub <noreply@github.com>
INFO   gitBranch: HEAD
INFO   gitOrigin: git@github.com:common-workflow-language/cwltool.git
INFO   gitDescribe: 3.1.20221008225030
INFO   gitPath: tests/loop/multi-source-loop.cwl
INFO Using cluster pirca (https://workbench2.pirca.arvadosapi.com/)
tests/loop/multi-source-loop.cwl:60:7: object id `tests/loop/multi-source-loop.cwl#loop/i1` previously defined
INFO Uploading workflow dependencies
INFO Upload local files: "multi-source-loop.cwl"
INFO Uploaded to 8496081f0f1a879983bd820350c391ed+69 (pirca-4zz18-8rpol3cb9urj72y)
INFO Upload local files: "multi-source-loop.cwl"
INFO Uploaded to 8496081f0f1a879983bd820350c391ed+69 (pirca-4zz18-0tmwn569qdpu3r8)
INFO Upload local files: "multi-source-loop.cwl"
INFO Uploaded to 8496081f0f1a879983bd820350c391ed+69 (pirca-4zz18-cw3cbw3rsnv8963)
tests/loop/multi-source-loop.cwl:60:7: object id `tests/loop/multi-source-loop.cwl#loop/i1` previously defined
INFO Using collection cache size 256 MiB
tests/loop/multi-source-loop.cwl:60:7: object id `tests/loop/multi-source-loop.cwl#loop/i1` previously defined
INFO [container multi-source-loop.cwl (3.1.20221008225030)] submitted container_request pirca-xvhdp-qr0weevqxf9pngw
INFO Monitor workflow progress at https://workbench2.pirca.arvadosapi.com/processes/pirca-xvhdp-qr0weevqxf9pngw
INFO [container multi-source-loop.cwl (3.1.20221008225030)] pirca-xvhdp-qr0weevqxf9pngw is Final
ERROR [container multi-source-loop.cwl (3.1.20221008225030)] (pirca-dz642-32ee2lghp5x26if) error log:

  2023-02-03T16:58:02.823871300Z stderr INFO   gitDate: Sat, 8 Oct 2022 18:50:30 -0400
  2023-02-03T16:58:02.823871300Z stderr INFO   gitCommitter: GitHub <noreply@github.com>
  2023-02-03T16:58:02.823871300Z stderr INFO   gitBranch: HEAD
  2023-02-03T16:58:02.823871300Z stderr INFO   gitOrigin: git@github.com:common-workflow-language/cwltool.git
  2023-02-03T16:58:02.823871300Z stderr INFO   gitDescribe: 3.1.20221008225030
  2023-02-03T16:58:02.823871300Z stderr INFO   gitPath: tests/loop/multi-source-loop.cwl
  2023-02-03T16:58:02.833091865Z stderr INFO Using cluster pirca (https://workbench2.pirca.arvadosapi.com/)
  2023-02-03T16:58:02.835312666Z stderr INFO Uploading workflow dependencies
  2023-02-03T16:58:02.837468818Z stderr INFO Upload local files: "workflow.json"
  2023-02-03T16:58:03.037883236Z stderr INFO Uploaded to 05cad3385458ba6f2a3f1d938d7fdf6b+61 (pirca-4zz18-xlwkg30tr4h6nmp)
  2023-02-03T16:58:03.040241342Z stderr INFO Upload local files: "workflow.json"
  2023-02-03T16:58:03.205269548Z stderr INFO Uploaded to 05cad3385458ba6f2a3f1d938d7fdf6b+61 (pirca-4zz18-wkqwbzs9w6j8jk1)
  2023-02-03T16:58:03.214737914Z stderr INFO Upload local files: "workflow.json"
  2023-02-03T16:58:03.384116424Z stderr INFO Uploaded to 05cad3385458ba6f2a3f1d938d7fdf6b+61 (pirca-4zz18-65aucfnlwh2b2pa)
  2023-02-03T16:58:03.455522756Z stderr INFO Using collection cache size 256 MiB
  2023-02-03T16:58:03.482503709Z stderr INFO Running inside container pirca-dz642-32ee2lghp5x26if
  2023-02-03T16:58:03.617923824Z stderr INFO [workflow workflow.json#main (3.1.20221008225030)] start
  2023-02-03T16:58:03.618067770Z stderr INFO [workflow workflow.json#main (3.1.20221008225030)] starting step loop
  2023-02-03T16:58:05.328734890Z stderr INFO [step loop] start
  2023-02-03T16:58:05.329627441Z stderr INFO [workflow loop] start
  2023-02-03T16:58:05.329870356Z stderr INFO [workflow loop] starting step big_values
  2023-02-03T16:58:05.340676698Z stderr INFO [step big_values] will be skipped
  2023-02-03T16:58:05.340687590Z stderr INFO [step big_values] completed skipped
  2023-02-03T16:58:05.340785230Z stderr INFO [workflow loop] starting step small_values
  2023-02-03T16:58:05.353339742Z stderr INFO [step small_values] start
  2023-02-03T16:58:05.364155348Z stderr INFO [step small_values] completed success
  2023-02-03T16:58:05.364338369Z stderr INFO [workflow loop] completed success
  2023-02-03T16:58:05.364622131Z stderr INFO [step loop] Iteration 1 completed success
  2023-02-03T16:58:05.364782285Z stderr WARNING Failed to evaluate expression:
  2023-02-03T16:58:05.364782285Z stderr Value for file:///var/lib/cwl/workflow.json#main/loop/i1 not specified
  2023-02-03T16:58:05.506490459Z stderr ERROR [step small_values] Output is missing expected field file:///var/lib/cwl/workflow.json#main/loop/run/small_values/o1
  2023-02-03T16:58:05.762850244Z stderr WARNING [step small_values] completed permanentFail
  2023-02-03T16:58:05.936479138Z stderr INFO [workflow loop] completed permanentFail
  2023-02-03T16:58:05.936669733Z stderr WARNING [step loop] Iteration 2 completed permanentFail
  2023-02-03T16:58:06.120741674Z stderr ERROR Workflow execution failed:
  2023-02-03T16:58:06.120741674Z stderr Value for file:///var/lib/cwl/workflow.json#main/loop/i1 not specified
  2023-02-03T16:58:06.285046365Z stderr ERROR Workflow error, try again with --debug for more information:
  2023-02-03T16:58:06.285046365Z stderr Workflow did not return a result.
  2023-02-03T16:58:06.778839605Z crunch-run Container exited with status code 1
  2023-02-03T16:58:07.041021540Z crunch-run Complete
ERROR Overall process status is permanentFail
INFO Final output collection 5551f5d7ec57a8aa9758d14a976e87e9+57
INFO Output at https://workbench2.pirca.arvadosapi.com/collections/5551f5d7ec57a8aa9758d14a976e87e9+57
{}
WARNING Final process status is permanentFail

Test files from https://github.com/common-workflow-language/cwltool/tree/3.1.20221008225030/tests )

@sonarcloud
Copy link

sonarcloud bot commented Feb 3, 2023

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot E 1 Security Hotspot
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

@mr-c mr-c marked this pull request as draft February 3, 2023 17:01
@mr-c
Copy link
Contributor Author

mr-c commented Feb 3, 2023

Running the same test in the same virtualen/terminal using just cwltool --enable-ext produces the expected output

$ cwltool --enable-ext tests/loop/multi-source-loop.cwl tests/loop/single-var-loop-job.yml
INFO /home/michael/src/arvados/env3.10/bin/cwltool 3.1.20221008225030
INFO Resolved 'tests/loop/multi-source-loop.cwl' to 'file:///home/michael/cwltool/tests/loop/multi-source-loop.cwl'
URI prefix 'cwltool' of 'cwltool:loop' not recognized, are you missing a $namespaces section?
tests/loop/multi-source-loop.cwl:60:7: object id `tests/loop/multi-source-loop.cwl#loop/i1` previously defined
WARNING Workflow checker warning:
tests/loop/multi-source-loop.cwl:61:11: Source 'osmall' of type {"type": "array", "items": ["null",
                                        "int"]} may be incompatible
tests/loop/multi-source-loop.cwl:15:5:    with sink 'o1' of type {"type": "array", "items": "int"}
                                          source has linkMerge method merge_flattened
                                          pickValue is: all_non_null
tests/loop/multi-source-loop.cwl:61:19: Source 'obig' of type {"type": "array", "items": ["null",
                                        "int"]} may be incompatible
tests/loop/multi-source-loop.cwl:15:5:    with sink 'o1' of type {"type": "array", "items": "int"}
                                          source has linkMerge method merge_flattened
                                          pickValue is: all_non_null
INFO [workflow ] start
INFO [workflow ] starting step loop
INFO [step loop] start
INFO [workflow loop] start
INFO [workflow loop] starting step big_values
INFO [step big_values] will be skipped
INFO [step big_values] completed skipped
INFO [workflow loop] starting step small_values
INFO [step small_values] start
INFO [step small_values] completed success
INFO [workflow loop] completed success
INFO [step loop] Iteration 1 completed success
INFO [step loop] start
INFO [workflow loop_2] start
INFO [workflow loop_2] starting step big_values_2
INFO [step big_values_2] will be skipped
INFO [step big_values_2] completed skipped
INFO [workflow loop_2] starting step small_values_2
INFO [step small_values_2] start
INFO [step small_values_2] completed success
INFO [workflow loop_2] completed success
INFO [step loop] Iteration 2 completed success
INFO [step loop] start
INFO [workflow loop_3] start
INFO [workflow loop_3] starting step big_values_3
INFO [step big_values_3] will be skipped
INFO [step big_values_3] completed skipped
INFO [workflow loop_3] starting step small_values_3
INFO [step small_values_3] start
INFO [step small_values_3] completed success
INFO [workflow loop_3] completed success
INFO [step loop] Iteration 3 completed success
INFO [step loop] start
INFO [workflow loop_4] start
INFO [workflow loop_4] starting step big_values_4
INFO [step big_values_4] will be skipped
INFO [step big_values_4] completed skipped
INFO [workflow loop_4] starting step small_values_4
INFO [step small_values_4] start
INFO [step small_values_4] completed success
INFO [workflow loop_4] completed success
INFO [step loop] Iteration 4 completed success
INFO [step loop] start
INFO [workflow loop_5] start
INFO [workflow loop_5] starting step big_values_5
INFO [step big_values_5] start
INFO [step big_values_5] completed success
INFO [workflow loop_5] starting step small_values_5
INFO [step small_values_5] will be skipped
INFO [step small_values_5] completed skipped
INFO [workflow loop_5] completed success
INFO [step loop] Iteration 5 completed success
INFO [step loop] start
INFO [workflow loop_6] start
INFO [workflow loop_6] starting step big_values_6
INFO [step big_values_6] start
INFO [step big_values_6] completed success
INFO [workflow loop_6] starting step small_values_6
INFO [step small_values_6] will be skipped
INFO [step small_values_6] completed skipped
INFO [workflow loop_6] completed success
INFO [step loop] Iteration 6 completed success
INFO [step loop] start
INFO [workflow loop_7] start
INFO [workflow loop_7] starting step big_values_7
INFO [step big_values_7] start
INFO [step big_values_7] completed success
INFO [workflow loop_7] starting step small_values_7
INFO [step small_values_7] will be skipped
INFO [step small_values_7] completed skipped
INFO [workflow loop_7] completed success
INFO [step loop] Iteration 7 completed success
INFO [step loop] start
INFO [workflow loop_8] start
INFO [workflow loop_8] starting step big_values_8
INFO [step big_values_8] start
INFO [step big_values_8] completed success
INFO [workflow loop_8] starting step small_values_8
INFO [step small_values_8] will be skipped
INFO [step small_values_8] completed skipped
INFO [workflow loop_8] completed success
INFO [step loop] Iteration 8 completed success
INFO [step loop] start
INFO [workflow loop_9] start
INFO [workflow loop_9] starting step big_values_9
INFO [step big_values_9] start
INFO [step big_values_9] completed success
INFO [workflow loop_9] starting step small_values_9
INFO [step small_values_9] will be skipped
INFO [step small_values_9] completed skipped
INFO [workflow loop_9] completed success
INFO [step loop] Iteration 9 completed success
INFO [step loop] completed success
INFO [workflow ] completed success
{
    "o1": [
        2,
        3,
        4,
        5,
        8,
        11,
        14,
        17,
        20
    ]
}
INFO Final process status is success

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@mr-c