Need help setting progress reports with the shell plugin

[Kiskadee-dev]

Hi, first of all, this project is absolutely amazing and i'm really mad i just found out about it by accident this week..

I'm trying to setup progress reports with python but it just won't read and update the values in the UI, tried both the json and the shorthand..

Here's a simple test script

import time
import sys

MAX = 30
for i in range(MAX):
    print(f"{i}...")
    progress = {"progress": i / MAX}
    print(progress, file=sys.stdout)
    print("...")
    time.sleep(2)

ran it from the shell plugin:

#!/bin/sh
abs/python/path/python abs/script/path/xyz.py

Logs doesn't show until the end of the execution, neither the progress is reported.. i've enabled the "interpret json in output" in the task settings, the task runs fine.

[jhuckaby]

Hi there! Thank you for using Cronicle!

Okay, so you have two problems here:

First, Python buffers all output by default (it only actually sends STDOUT in batches at intervals), which we absolutely do NOT want in this case. We want it to output EVERY line as you print it. So you need to disable output auto-buffering. There are several methods to do this. The easiest way is to just add the -u flag to Python when you run your script:

python -u your_script.py

The second problem is, if you want to send progress updates from the Shell Plugin, you need to format the output exactly right. Specifically, you need a percent symbol (%) after the number, and you need to format your number as a percentage from 0 to 100.

I am no Python expert by any means, but it looks like your progress numbers will be between 0 and 1, so you need to multiply that final number by 100, and then you need to append a % symbol at the end. You want the output to look like this:

1%
2%
3%
4%
5%

...and so on.

Alternatively, if you check the "Interpret JSON in Output" checkbox in your event using the Shell Plugin, you can output NDJSON to report progress updates that way. See docs here for details. I am not sure how to do this in Python, sorry.

Good luck!

[Kiskadee-dev]

Hi! Thanks for the reply! indeed python does buffer the output, after investigating a bit further i've noticed the simplified progress doesn't accept floating numbers, only integers, i got the wrong impression from the NDJSON examples,
{"progress": 0.5}

So, after unbuffering the outputs and using the correct format the simplified version is working fine now and i'm very happy with how it goes, but the NDJSON way is still a mystery for me

[jhuckaby]

JSON is a very picky file format. Everything has to be EXACTLY correct. The problem I see in your screenshot above is that all your quotes around the keys are 'single' quotes. All key quotes have to be "double" quotes in JSON:

Like this: {"progress": 0.5} ✅

NOT like this: {'progress': 0.5} ❌