Distribute compiletest binary

[sfackler]

Now that syntax extensions can be defined by third party code, cfail tests are useful outside of rustc's test suite. It may need to be renamed to something with "rust" in it.

[huonw]

More and more people are writing external syntax extensions, so having compiletest relatively easily usable is become more and more useful.

[japaric]

Now that we have cargo, we could easily distribute compiletest as a package. However, it'd be great if we could tweak it a bit first, to require less flags (by providing defaults) to make it easier to use outside of the rustc bootstrap context.

[brson]

Compiletest would definitely need a big cleanup to be usable out-of-tree.

[SimonSapin]

In rust-lang/cargo#1194 (comment) , @alexcrichton wrote:

I'm also fairly hesitant about doing with as this means we have a stability commitment to our error messages at that point.

What if, until we’re ready to stabilize error messages (or possibly error codes, or something), only binary "triggered a build error or not" testing was available to beta and stable release channels? I think it would still be useful.

[sfackler]

Covered by the compiletest-rs crate.

[OJFord]

Covered by the compiletest-rs crate.

Link for the lazy.

[sunjay]

Sorry to resurrect an old issue (I can open a new one if needed), but are there any plans to add support for this type of compile testing in std? It would be really cool to just be able to add should_not_compile to things like doctests (similarly to how we add no_run now).

When documenting how something doesn't compile, it's convenient to be able to test that it actually never compiles. :)

[Mark-Simulacrum]

I think there's some support for compile_fail in doctests and maybe some support for indicating the error code that's expected. @GuillaumeGomez might know more, see https://github.com/GuillaumeGomez/rust/blob/eb7664b445d4c2a44e31b6557931dda88e7dd071/src/librustc_borrowck/diagnostics.rs#L20 for what I'm talking about.