Combination of click and switch command in poe

[amirhessam88]

Hello @nat-n
I have a quick question and appreciate your thoughts. I am using the example below to describe my current issue.

I am currently using poe to invoke pipeline runs. My pipelines use click for parameterization and I am trying to combine switch command in poe with dynamic parameters (so far no luck).

Let's assume I have the following files in my pipeline namespace:

.
├── src
 │   └── pipelines
 │       ├── p1.py
 │       └── p2.py
└── pyproject.toml

# p1.py

import click


@click.command()
@click.option(
    "--foo",
    type=str,
)
@click.option(
    "--bar",
    type=str,
)
def main(foo: str, bar: str) -> int:
    s = f"pipeline ran with {foo=} and {bar=} ..."
    print(s)

    return 0


if __name__ == "__main__":
    raise SystemExit(main())

and

# p2.py

import click


@click.command()
@click.option(
    "--baz",
    type=str,
)
def main(baz: str) -> int:
    s = f"pipeline ran with {baz=} ..."
    print(s)

    return 0


if __name__ == "__main__":
    raise SystemExit(main())

So, normally if I wanna invoke the above pipelines in terminal, I would do:

# p1 invoke
poetry run python -m pipelines.p1 --foo="FOO" --bar="BAR"

# p2 invoke
poetry run python -m pipelines.p2 --baz="BAZ"

Previously, I was using script command in poe, but in this case I am not sure how can I have one general poe run command that covers all different params cases (something like **params)

I am looking to figure out a way to propagate the params into the switch commands; for example

# pyproject.toml

[tool.poe.tasks.run]
control.expr = "pipeline"
args = ["pipeline", "foo", "bar"]

[[tool.poe.tasks.run.switch]]
case = "p1"
# how to pass the args from terminal here ?
cmd = "python -m pipelines.p1"

So passing poe run --pipeline="p1" --foo="FOO" --bar="BAR" would be like

Poe <= pipeline
Poe => python -m pipelines.p1
pipeline ran with foo=None and bar=None ...

Now, I dont know how can I have a switch command that can handle both p1 and p2 and ... pn params hypothetically.

Appreciate your thoughts ....

[nat-n]

Hi @amirhessam88,

It sounds like maybe what you really want is two separate script tasks, rather an a single switch task?

I haven't tested it but I suspect the following example will work as you intend because extra arguments will be set as sys.argv when running the main function, and thereby get parsed by click.

[tool.poe.tasks.run-p1]
script = "pipelines.p1:main"  

Alternatively if you declare the arguments on the task then you'll benefit from the generated documentation in poe, though I'm not sure how this will interact with the the click decorated main function.

[tool.poe.tasks.run-p2]
script = "pipelines.p2:main()"
args = ["baz"]

If you primarily intend to run these pipeline tasks through poe then maybe you don't need click at all? Since poe supports all the basics in terms of declaring CLI arguments.

One another potential benefit of having multiple tasks is that they'll work with autocomplete in most shells.

---

That said if for some reason you really do want to use a switch task here, then I think you can just pass them in the command or script call like in either of the tasks in this example:

[tool.poe.tasks.run]
control.expr = "pipeline"
args = ["pipeline", "foo", "bar"]

[[tool.poe.tasks.run.switch]]
case = "p1"
cmd = "python -m pipelines.p1 --foo ${foo} --bar ${bar}"

[[tool.poe.tasks.run.switch]]
case = "p2"
script = "pipelines.p2:main(baz=bar)" # if this doesn't work try "pipelines.p2:main(baz=environ['bar'])"

[amirhessam88]

Oh cool @nat-n ; yeah, I hear you; At first, I started it with click and I have poe every where; so, let me try the recommended patterns and I get back to you with my thoughts. Thank you again

[amirhessam88]

@nat-n Thanks; your solution almost worked and there are some nuances on how to deal with special characters when we pass as args.

For example in the example above; I can simply run poetry run python -m pipeline.p1 --foo="f o o" --bar="b ar"
However, when I use poe I should escape space with \ which is fine.
This gets kinda silly when you wanna pass * for example as part of a cron expression like 0 0 0 * * *; I know we can turn off global expansions but I wanna know your opinion on any other solutions here;

# pyproject.toml based on your solution 

[tool.poe.tasks.run]
control.expr = "pipeline"
args = ["pipeline", "foo", "bar", "baz"]

[[tool.poe.tasks.run.switch]]
case = "p1"
cmd = "python -m pipelines.p1 --foo=${foo} --bar=${bar}"

[[tool.poe.tasks.run.switch]]
case = "p2"
cmd = "python -m pipelines.p1 --baz=${baz}"

Now, let's invoke them for some examples

poe run --pipeline="p1" --foo="foo" --bar="bar" # this works
poe run --pipeline="p1" -- foo="foo foo" # this did not work
poe run --pipeline="p1" --foo="foo\ foo" # this works
poe run --pipeline="p1" --foo="0 0 0 * * *" # this did not work with any escaping pattern i tried

I have to mention that I did also try to use single quotes (literal) in cmd part poe task and wrapped the env-vars inside a double quotes like below which fixed the space issue;

[[tool.poe.tasks.run.switch]]
case = "p1"
cmd = 'python -m pipelines.p1 --foo="${foo}" --bar="${bar}"'

then

poe run --pipeline="p1" -- foo="foo        foo" # this also works now

ref: https://www.youtube.com/watch?v=VIUoHnFwEH4

[nat-n]

Hey @amirhessam88,

I would expect using double quotes in the task definition to generally do what you want. The cmd task is intended to closely emulate bash syntax. There are some edge cases where it fails (because currently it relies on shlex which is too simplistic), but I'm actually working on a new parser at the moment to make it more robust and shell like.

Is there anything you can't achieve by defining the tasks like so?

# pyproject.toml based on your solution 

[tool.poe.tasks.run]
control.expr = "pipeline"
args = ["pipeline", "foo", "bar", "baz"]

[[tool.poe.tasks.run.switch]]
case = "p1"
cmd = """python -m pipelines.p1 --foo "${foo}" --bar "${bar}" """

[[tool.poe.tasks.run.switch]]
case = "p2"
cmd = """python -m pipelines.p1 --baz "${baz}" """

Notice that I removed the = characters to get around a silly limitation in the parsing and escaping logic. One of the edge cases I mentioned above, is that quotes only work if they're around a whole word surrounded by whitespace. You could also keep the = and put the quotes around the whole option k like: "--option=${value}", but that's a bit less conventional.

---

EDIT: the issue with partially quoted words in cmd tasks is now fixed with v0.23.0

[amirhessam88]

Thanks @nat-n for all the support and your answer did solve my current problem!

[RoyerRamirez]

Thanks for your help @nat-n!