Security bugs & fuzzing

[stevenjohnstone]

I've been experimenting with fuzzing yarp. I have a found a few bugs which are perhaps security-relevant including

• double-frees
• heap overflow writes
• use-after-free
• infinite loops on input (with memory exhaustion)

How bad these are is dependent on the context in which yarp is used.

What's the best way to proceed? Open public bugs? Report privately?

I'd like to contribute the fuzzing code I've been using: is there interest?

[rafaelfranca]

As this is early in the development and YARP isn't being used in security sensitive context just yet I believe reporting the bugs publicly is fine

[kddnewton]

@stevenjohnstone what are you using to fuzz test? You've found some pretty incredible stuff so far. I'm wondering if it would be possible to automate some of it on a cron daily or something similar.

[stevenjohnstone]

I'm using AFL++. I can put something together in a PR if you like? I've been holding off until I got something which was robust enough to share. From previous experience, these things tend to rot when they are too noisy, hard to bootstrap, melt the user's CPU etc.

Long term, I think that (eventually), there'd be value in integrating with OSS-Fuzz.

I think it'd be really interesting to use this as a launchpad to differential fuzz testing against other parsers. Something like this was suggested in #827.

[kddnewton]

A PR would be wonderful. Once we get the compiler fully operational, I would love to fuzz this parser and the current parser to ensure we generate the same instructions for the same inputs.

At the moment I'm super happy with the fuzzing you've done so far because it has given me more confidence that we're not leaking all over the place. So in that vein it's great for bugs as well.

OSS-Fuzz would be great, as you said in the long term. You're clearly turning up enough stuff that we shouldn't invest too much into that just yet.

[stevenjohnstone]

stevenjohnstone@bdb0b88 WIP. It would be cool if someone could give this a spin with make fuzz/run/parse. No idea how this stuff plays out on a mac or windows, but in theory it should work if docker is there.

There's a hang bug in 4c48235 along the lines of

0000<<``=<<``\


=<<``\000

where the first and last four bytes are inputs the function under test is not meant to access. This would be a good thing to try out the tools on to see if it can be found again.

[kddnewton]

@stevenjohnstone thank you for putting together the PR, I'll review it over the next couple of days.

I was wondering if you had any experience with grammar-based fuzzing? I've been looking at a couple of tools (Grammarinator, Superion, Nautilus, AFLSmart) and the general approach seems like it could be really good for this project.

[stevenjohnstone]

I've found grammar and structure-aware fuzzing useful to get past code like parsers and lexers to reach further into language runtimes.

I think you'll get far with finding bugs with coverage guidance and cmplog. It may be beneficial to run grammar-mutator in parallel to feed the fuzzer with new, valid ruby syntax to mutate.

I suspect grammar-aware fuzzing will come into its own when doing differential fuzzing of yarp against other parsers, where you need many valid programs to compare.

I'll add the plumbing for grammar-mutator to the PR and a make target to run two afl instances together: one doing the usual mutations and the other doing grammar mutations. In that setup, the fuzzers will be able to take interesting inputs from each other, hopefully finding interesting things.