Programmatically cancelling a pipelinerunner.run

[qwertimer]

I am looking at setting up a pipeline programmatically in python and am trying to figure out whether there is a way to run the runner and cancel it mid process. I am looking at using it for running experiment tasks and calling a cancel button when needed, this would stop the current runner in the same way as would occur on the CLI. Is this at all possible.

[yaythomas]

hi @qwertimer ,

You might want to investigate making your pipelines so that they will exit at certain points... This is a cleaner way of shutting down the process, which might matter or not depending on exactly what you're doing and what systems pypyr is interacting with.

For this sort of thing, you might want to check pypyr.steps.stop and then conditionally letting the stop step run:

steps:
  - name: pypyr.steps.set
    in:
      set:
        stop_me: !py input("type 1 ENTER to stop, anything else will continue:\n")

  - name: pypyr.steps.stop
    description: -> will stop here if stop_me is 1, true or TRUE
    run: '{stop_me}'

  - name: pypyr.steps.echo
    comment: you won't see this if pypyr stopped at the previous step
    in:
      echoMe: continuing rather than stopping

This is just a super simplistic example to give you some ideas.

If you can't write your pipelines to have "stop" points in them, you could launch pypyr as a subprocess as follows:

import subprocess
import time

process = subprocess.Popen(['pypyr', 'my-pipeline', 'a=b', 'c=d'])

while True:
    process.poll()
    rc = process.returncode
    if rc is None:
        print("still running")
    else:
        if rc == 0:
            print("success done")
        else:
            print("done with err")

        break

    cancel_command = input(
        "x and ENTER to terminate, any other key to continue\n")

    if cancel_command == 'x':
        print("cancelling")
        process.terminate()
        break

    time.sleep(2)

print("done")

I just arbitrarily mocked something like what you're asking for using a simple CLI where if the user types x enter it will exit the loop where it's waiting for the pypyr process to finish. It's not the tidiest example in the world, but hopefully enough to get you started. This spawns a subprocess to run pypyr in.

If you want to use the API and launch the pypyr API and cancel/kill it from the same python process, this gets more tricky - you're going to have to delve into the lower-level multithreading/multiprocessing libraries for that. Just asyncio in itself is not going to do what you want, because you won't have any way of cancelling the running Future.