[Console output] http_req_failed : "succMark" & "failMark" seems inversed?

[placaze]

Brief summary

     data_received..................: 7.2 MB 240 kB/s
     data_sent......................: 498 kB 17 kB/s
[...]
     http_req_failed................: 56.68% ✓ 403      ✗ 308
[...]
     http_reqs......................: 711    23.61168/s

I think ✓ and ✗ are inversed here. There should be more failed than succeed.

k6 version

0.35.0

OS

Windows 10

Docker version and image (if applicable)

No response

Steps to reproduce the problem

Do a simple run of k6. Nothing fancy.

Expected behaviour

http_req_failed................: 56.68% ✓ 308      ✗ 403

Actual behaviour

http_req_failed................: 56.68% ✓ 403      ✗ 308

[na--]

Thanks for reporting this! Technically, since the http_req_failed metric is tracking the failures, it's technically correct. But yeah, that's super confusing 😞

So, I'm not sure if we should just reverse the marks, but maybe we should hide them or turn them into text labels? 🤔

http_req_failed................: 56.68% failed: 403      successful: 308

Also, I am not sure if this should be a hack, a check just for the http_req_failed metric in here:

k6/js/summary.js

Lines 233 to 238 in 804203b

	case 'rate':
	return [
	humanizeValue(metric.values.rate, metric, timeUnit),
	succMark + ' ' + metric.values.passes,
	failMark + ' ' + metric.values.fails,
	]

Or should it be a more universal solution? Maybe Rate metrics should have an additional parameter that allows users to specify that they are inverted? Similarly to how Trend metrics have a parameter that specifies if they are tracking time or not?

For now, as a workaround, you can define your own handleSummary() function to do whatever you want with it, but yeah, we need to fix this in k6.

[NevilleC]

I got confused by this, too, while discovering k6.

turn them into text labels

To me, text labels would remove any ambiguity 👍

[f0ff886f]

Hi, so just to clarify, is the %age shown the % of requests that failed? So 0% http_req_failed means 100% requests succeeded (either by the default comparison to 200 status code or overridden by the user)?

So a checkmark in the output defines the amount of failed requests, and the x defines the amount of successful requests?

[na--]

Yes 😞

[Totalus]

+1, that is overly confusing

[mariaduran-ecr]

how is it going to affect thresholds? Should the http_req_failed: ['rate '] be reversed as well?

[na--]

@mariaduran-ecr, it shouldn't. The problem is that the end-of-test summary assumes that all Rate metrics are about measuring "positive" events, i.e. that a rate of 100% is a good thing. That assumption doesn't hold for http_req_failed, where we want the rate of failed requests to be 0%. And it probably also doesn't hold for some custom Rate metrics that users might have defined in their scripts.

And IIRC, http_req_failed is negative precisely because it's easier to reason about it that way when it comes to thresholds. For example, the constraint of "I want under 1% of requests to fail" can be directly expressed as the http_req_failed: ['rate < 0.01'] threshold definition.

[nthornton2010]

I'm trying out k6 now and this definitely confused the heck out of me.

[mstoykov]

Another proposal that I came up with in #3656 was to actually use words to say how many of the given rate were "truthy"

This way it seems like there is no ambiguity.