promql: extend test scripting language to support asserting on expected error message

[charleskorn]

This PR extends the PromQL test scripting language to support asserting that a query fails with a particular error message or an error message matching a regexp (rather than just asserting that the query fails for any reason).

[machine424]

how about fail_message or fail_msg to be more explicit?

[charleskorn]

I've changed it to expected_fail_message, how's that?

[beorn7]

Note for a future PR: It would be good to also be able to assert annotations.

[zenador]

The nhcb branch lets us assert warning annotations, but not yet specific ones. After this is merged, when I fix the conflicts I can look into doing something similar to assert specific annotations. Might be slightly more complicated though because you can only have maximum 1 error but you may have multiple warning annotations.

[bboreham]

Why do we need both an explicit string and a regex, given the former is a subset of the latter?

Incidentally "pattern" did not immediately suggest "regex" to me, maybe because I spent too much time using LogQL.

[charleskorn]

Why do we need both an explicit string and a regex, given the former is a subset of the latter?

Many error messages contain characters that are special in regexes (eg. { or [), so it's often clearer to use a simple string match than try to escape all the special characters in a regex.

However, an exact match isn't great for every case - sometimes a regex is more convenient or clearer (eg. if only parts of the error message are important).

Incidentally "pattern" did not immediately suggest "regex" to me, maybe because I spent too much time using LogQL.

Fair point, I'll change this.

[charleskorn]

I've rebased this to resolve the conflicts with #13999.

[charleskorn]

Any chance I could get a review on this? Would be great to get this merged.

[machine424]

The PR has diverged, with the introduction of regexes, since my initial review.

However, an exact match isn't ideal for every case - sometimes a regex is more convenient or clearer (for instance, if only parts of the error message are significant).

Could you provide examples?
When adding a new test, given that promtool will return the precise failure message, users can simply copy and paste it (if they agree with it).

Moreover, we should clarify that we do not guarantee the stability of these messages, which might make these checks challenging to maintain.
Also, users might not need to perform such checks and can trust promql to emit a good failure message for the failing step, it's up to promql's tests to ensure this.
Could tell us more about the use cases of these fail message checks?

[charleskorn]

However, an exact match isn't ideal for every case - sometimes a regex is more convenient or clearer (for instance, if only parts of the error message are significant).

Could you provide examples? When adding a new test, given that promtool will return the precise failure message, users can simply copy and paste it (if they agree with it).

Moreover, we should clarify that we do not guarantee the stability of these messages, which might make these checks challenging to maintain. Also, users might not need to perform such checks and can trust promql to emit a good failure message for the failing step, it's up to promql's tests to ensure this. Could tell us more about the use cases of these fail message checks?

The use case for this is not for promtool test, but for testing the PromQL engine itself and verifying that the error messages returned in failure cases are as expected.