Stats: marking known-useless runs

[Ekleog]

Assume a fuzzer that looks like:

#[test]
fn fuzzer() {
    check!().for_each(|i: &[u8]| {
        if let Ok(i) = parse_input(i) {
            the_thing_we_actually_want_to_fuzz(i)
        }
    })
}

Basically, any run that does not result in parse_input returning Ok is useless, as the goal of this fuzzer is not semantically to fuzz parse_input (which may be ad-hoc for this fuzzer).

In turn, I'm thinking it might make sense to add a way to report to cargo-bolero that the input was actually (un)interesting, so that it could report stats (eg. "60% of all runs were uninteresting, consider improving parse_int to increase the proportion of valid inputs").

Or even, having the API ending up as something like:

#[test]
fn fuzzer() {
    check!(min_interesting_input_rate=0.6).for_each(|interesting: &mut BoleroMetadata, i: &[u8]| {
        if let Ok(i) = parse_input(i) {
            interesting.is_interesting();
            the_thing_we_actually_want_to_fuzz(i)
        }
    })
}

And then bolero when run in cargo-test mode would run for 1000 times and assert that at least 600 of the runs hit the interesting.is_interesting() line.

This would be bells and whistles, so definitely not essential, but WDYT about it?

[camshaft]

Yeah that would be really nice to have. Although I'd have to think about the API a bit.

One thing to note is you could implement your parse_input function as a ValueGenerator, which has the ability to reject generated values by returning None. At that point, all we would need is for bolero to report on the ratio of Some to None.

[Ekleog]

Maybe it'd make sense to have a .preprocess_input(Fn(T) -> Option<U>) on the builder, that'd hide the creation of the ValueGenerator?
With this setup, bolero reporting the ratio of Some to None would be enough indeed I think :)