IPython2CWL DEMO

This demo requires ipython2cwl

$ pip install ipython2cwl

This example is based on demo1.ipynb.

• You can clone the repo or,
• You can execute:
  • jupyter-repo2cwl https://github.com/giannisdoukas/ipython2cwl-demo -o .
  • That command will generate a docker image & a file named demo1.cwl in your current working directory
  • You can still execute the Jupyter Notebook by executing: $ docker run -p 127.0.0.1:8888:8888 THE_NAME_OF_THE_GENERATED_IMAGE
  • You can find the generated image id or in the logs produced by the jupyter-ipython2cwl or in the generated cwl file.

---

Users can use typing hints in string format or by importing the library.

!echo "Hello World 1" > message1.txt
file_input1: 'CWLFilePathInput' = 'message1.txt'
with open(file_input1, 'r') as f:
    data1 = f.read()    

!echo "Hello World 2" > message2.txt
from ipython2cwl.iotypes import CWLFilePathInput
file_input2: CWLFilePathInput = 'message2.txt'
with open(file_input2, 'r') as f:
    data2 = f.read()

For each variable which has the typing hints the tool will remove the presented assigment and it will generify it and expose it as a command line argument.

Currently the supported type hints are:

• CWLFilePathInput
• CWLBooleanInput
• CWLStringInput
• CWLIntInput
• CWLFilePathOutput

These types can be also combined with List & Optional.

import os
from typing import List
from ipython2cwl.iotypes import CWLStringInput
messages1: List[CWLStringInput] = ["message1:", "hello", "world", '!']
messages2: List['CWLStringInput'] = ["message2:", "hello", "world", "!!"]
messages3: 'List[CWLStringInput]' = ["message3:", "hello", "world", "!!!"]
messages4: 'Optional[CWLStringInput]' = "messages4"

print(' '.join(messages1))
print(' '.join(messages2))
print(' '.join(messages3))
if messages4 is None:
    print('='*20)
    print("Message 4 is None")
    print('='*20)
else:
    print(messages4)

message1: hello world !
message2: hello world !!
message3: hello world !!!
messages4

To store the results to a file we have to define a variable and give the hint CWLFilePathOutput.

output: 'CWLFilePathOutput' = 'output.txt'
with open(output, 'w') as f:
    f.write(' '.join(messages1))
    f.write(os.linesep)
    f.write('-'*10)
    f.write(os.linesep)
    f.write(' '.join(messages2))
    f.write(os.linesep)
    f.write('-'*10)
    f.write(os.linesep)
    f.write(' '.join(messages3))

!cat output.txt

message1: hello world !
----------
message2: hello world !!
----------
message3: hello world !!!

The auto-generated CWL file will look like:

# !cat ../out/demo1.cwl

arguments:
- --
baseCommand: /app/cwl/bin/demo1
class: CommandLineTool
cwlVersion: v1.1
hints:
  DockerRequirement:
    dockerImageId: r2d-2fvar-2ffolders-2fk8-2f800hfw-5fn2md-5f2zb44lhhtqqr0000gn-2ft-2frepo2cwl-5fixi144l4-2frepo1593555577
inputs:
  file_input1:
    inputBinding:
      prefix: --file_input1
    type: File
  file_input2:
    inputBinding:
      prefix: --file_input2
    type: File
  messages1:
    inputBinding:
      prefix: --messages1
    type: string[]
  messages2:
    inputBinding:
      prefix: --messages2
    type: string[]
  messages3:
    inputBinding:
      prefix: --messages3
    type: string[]
  messages4:
    inputBinding:
      prefix: --messages4
    type: string?
outputs:
  output:
    outputBinding:
      glob: output.txt
    type: File

and the auto generated script looks like:

$ docker run -it r2d-2fvar-2ffolders-2fk8-2f800hfw-5fn2md-5f2zb44lhhtqqr0000gn-2ft-2frepo2cwl-5fixi144l4-2frepo1593555577 cat /app/cwl/bin/demo1 # this command will display the auto-generated script to the terminal

#!/usr/bin/env ipython
"""
DO NOT EDIT THIS FILE
THIS FILE IS AUTO-GENERATED BY THE ipython2cwl.
FOR MORE INFORMATION CHECK https://github.com/giannisdoukas/ipython2cwl
"""


def main(file_input1, file_input2, messages1, messages2, messages3, messages4):
    get_ipython().system('echo "Hello World 1" > message1.txt')
    with open(file_input1, 'r') as f:
        data1 = f.read()
    get_ipython().system('echo "Hello World 2" > message2.txt')
    with open(file_input2, 'r') as f:
        data2 = f.read()
    import os
    from typing import List
    print(' '.join(messages1))
    print(' '.join(messages2))
    print(' '.join(messages3))
    if messages4 is None:
        print('=' * 20)
        print('Message 4 is None')
        print('=' * 20)
    else:
        print(messages4)
    output = 'output.txt'
    with open(output, 'w') as f:
        f.write(' '.join(messages1))
        f.write(os.linesep)
        f.write('-' * 10)
        f.write(os.linesep)
        f.write(' '.join(messages2))
        f.write(os.linesep)
        f.write('-' * 10)
        f.write(os.linesep)
        f.write(' '.join(messages3))
    get_ipython().system('cat output.txt')


if __name__ == '__main__':
    import argparse
    import pathlib
    parser = argparse.ArgumentParser()
    parser.add_argument('--file_input1', type=pathlib.Path, required=True)
    parser.add_argument('--file_input2', type=pathlib.Path, required=True)
    parser.add_argument('--messages1', type=str, required=True, nargs='+')
    parser.add_argument('--messages2', type=str, required=True, nargs='+')
    parser.add_argument('--messages3', type=str, required=True, nargs='+')
    parser.add_argument('--messages4', type=str, required=False, default=None)
    args = parser.parse_args()
    main(file_input1=args.file_input1, file_input2=args.file_input2,
        messages1=args.messages1, messages2=args.messages2, messages3=args.
        messages3, messages4=args.messages4)

You can use try execute the generated workflow with the files in.yaml & in2.yaml

$ cwltool demo1.cwl in.yaml

$ cwltool demo1.cwl in1.yaml