Idea about test infrastructure

[vsoch]

I was talking with @mplegendre about an idea for a testing setup for Smeagle. The basic idea is that if Smeagle can turn facts into assembly instructions, we could actually compare binaries, which is a better ground truth than hard coding some set of checks that we either have to painfully manually do (and get wrong) or just painfully manually do for many cases.

So to restate the above in more concrete terms, right now we would have Smeagle look at the function, print out the location information it found, and then compare that to a ground truth we've derived (e.g. hard coded in the test). The problem is that the ground truth is hard to derive - we would need to dis-assemble assembly instructions, and we might just get it wrong thinking we know the right answer (and we are wrong).

But if Smeagle can generate output for assembly, we could feed that to an assembler, compiler it, and generate code that runs. The code that runs would do things like make assertions that the arguments passed are the right content, types, etc. We could also just compare binaries directly. Then writing tests is just writing test functions that assert their output. What do you think @hainest?