tests: update snapd testing tools with spread log filter

[sergiocazzolato]

This change includes a new tool used in snapd-testing-tools used to filter spread logs to be sent to COS.

This new tool reads the input from spread output and filters the data following defined rules. Currently it is filtering data to avoid sending the complete logs to COS.

The new tool sends to the stdout the full imput .

[bboozzoo]

does spread use stderr for regular logs or just stdout? iow. do we need |& ?

[sergiocazzolato]

the output goes to stdout iirc

[bboozzoo]

newline

[bboozzoo]

please use black to format the script

[bboozzoo]

why upcase?

[sergiocazzolato]

I'll revert that

[bboozzoo]

I think I'd restructure this a bit. You could have something like:

regexes = {
    ERROR: compile_rules(filter_rules, ERROR),
    DEBUG: compile_rules(filter_rules, DEBUG),
    FAILED: compile_rules(filter_rules, FAILED),
    WARNING: compile-rules(filter_rules, WARNING),
}

with open(output_file, "w") as myfile:
    for line in sys.stdin:
        print_line(line)

        kind = is_detail_start(line)
        # kind is one of [DEBUG, ERROR, FAILED, WARNING]

        if kind in exclude_lines:
            line = skip_details(sys.stdin)
            continue
            
         line = process_detail(sys.stdin, line, regexes[kind], myfile)
...

[bboozzoo]

hm wasn't the exclusion supposed to filter out the lines from stdout and write them to a file? isn't this operating in reverese?

[sergiocazzolato]

well, those tools are for different things, this tool is needed to filter the output to send to the observability stack, the other tool is used to filter the output displayed in github page.
So, it will work like this -> spread ... | filter_1 | filter_2
The filter 1 will save to a new file the filtered output and will send to stdout the stdin
The filter 2 will save to files the debug output and will send to stdout the contect without the debug output

[zyga]

I did a quick pass with some suggestions.

Please reformat with black as Maciej suggested and lint the code with mypy --strictl. With what we depend on it should not need any loopholes to validate.

[zyga]

I think this should not be capitalized

[zyga]

How about something like this?

def process_detail_line(line: str, regex_list: list[re.Pattern[str], to_stream: typing.TextIO) -> None:
    for regex in regex_list:
        for match in regex.findall(line):
            write_line(match, to_stream)
        break
    else:
        write_line(line, to_stream)

[zyga]

I've added type hints and fixed a few bugs. I'll push a patch after the planning call.