Switch `run-make` tests from Makefiles to rust #40713

TimNN · 2017-03-21T20:44:34Z

Rust currently has a suite of run-make tests, which generally test specific rustc invocations or behaviour, or require external tools (eg. grep or nm).

The goal of this issue is to rewrite these run-make tests, which are currently written as Makefiles, in rust to a) get rid of the dependency on external tools such as make and b) make them more accessible to rust contributors by not requiring arcane knowledge of the make tool.

The transition will require at least the following steps:

Survey the existing run-make tests with regard to what they actually do / which programs they use (and how). I assume the programs will likely fall into one of three categories:
- Rust tools (rustc, rustdoc)
- Utilities easily replaceable by rust code, eg. grep
- Complex utilities, eg. nm, which cannot be easily rewritten in rust

Design and (partially) implement a support library, which makes the actions identified above easily possible. For example I imagine a test should look something like this:

extern crate support;

fn main() {
    let s = support::init();
    
    let lib = s.rustc().compile("file1.rs").output_rlib();
    
    assert!(s.nm(lib).filter_lines("some_symbol").count() == 2);
}

Add a new mode to compile-test, maybe run-rmake, which runs the new rust-based run-make tests. This will either include compiling the support library, or receiving the support library from a previous build stage.
Start porting the actual run-make tests, which may involve adding additional functionality to the support library. At that time this issue, or another, will track the state of all the existing tests and include some detailed instruction to allow people to easily contributing by porting one of the existing tests.

There are some open questions:

Should the support library allow execution of arbitrary commands? Limiting commands to only those explicitly added to the support library would mean that there is a single place which lists the external tools we depend on.
Is the proposed integration with compiletest the correct choice? For example the support library itself may have external dependencies (maybe regex or gcc-rs) which means we should probably use cargo to compile the actual tests. At which point it may be worth considering if compile-test is needed at all or if cargo is enough (Have the support library in src/lib.rs, the new run-make tests in tests/*.rs and auxiliary files in subdirectories of tests/ named after the main test file).

If anyone wants to get involved in the process, please leave a comment on this issue or ping me on IRC.

Original Issue Description

Based on a short experiment, it looks like rust on msvc only has five build dependencies: Visual Studio, Git, Python, CMake and make, where make is only used for the run-make tests as far as I can tell.

Of those five, the first four are easily installable natively on windows, whereas make wasn't as straight forward to installe when I tried and required msys2 / mingw.

The questions the are:

Is there any interest in performing such a conversion?
If so, then which language should those tests be migrated to? Python or Rust seem like the logical choices.
How should the switch happen? We'll probably want some kind of incremental strategy, since there are quite a few run-make tests.

The text was updated successfully, but these errors were encountered:

petrochenkov · 2017-03-21T20:52:09Z

cc #38565 (comment)

petrochenkov · 2017-03-21T20:53:27Z

rust on msvc only has five build dependencies: Visual Studio, Git, Python, CMake and make

IIRC, diff was a dependency as well, probably replaceable with git diff

alexcrichton · 2017-03-21T21:12:27Z

Oh dear I completely forgot about filing an issue to do this!

I would love to have this implementation, I think we should drop make as fast as we can. The test are horribly difficult to write and easy to get wrong whereas I think the equivalent Rust code would be much nicer to read.

I would discourage use of Python for basically the sole reason that "contributors to rust-lang/rust are likely Rust programmers" in the sense that it's far easier for us as a community to maintain Rust code than Python code (empircally this seems true as well).

I agree an incremental strategy is best, and I think that we've got a lot of flexibility in terms of what we move to. In general I think it should look like:

All tests are basically a main.rs script that's the makefile today.
Tests can run arbitrary code (this is a crucial part of run-make vs other test suites)
Tests are given tons of inputs about the compiler and whatnot
I think that we'll want a support library (e.g. librun_make) or something like that with utility functions (sort of like tools.mk) that all the scripts can link to as well if they'd like to.

Given all that I'd imagine this would be best implemented by extending compiletest slightly to compile a librun_make library and then compile scripts with rustc and then execute them (passing variables as appropriate). That way we could add something like src/test/run-make2 and just slowly start transitioning tests over there.

(note that this'd make a fantastic quest issue to migrate over these tests)

bstrie · 2017-03-23T20:38:07Z

@alexcrichton Define "quest issue"? Do you mean something like what we did for #35233 ?

alexcrichton · 2017-03-23T23:18:21Z

Yep! Precisely like that

TimNN · 2017-07-08T12:11:46Z

I just updated the original issue with a somewhat more detailed plan on going forward, as well as some open questions. Any feedback / thoughts are greatly appreciated.

I will personally start to work on the steps mentioned, although that will have to wait for about two weeks until I finish my bachelor's thesis.

kennytm · 2017-07-27T20:34:36Z

Some of the run-make tests like crate-data-smoke, codegen-options-parsing and allow-non-lint-warnings-cmdline could be rewritten as ui tests.

Some must still rely on make, mainly dep-info and dep-info-spaces.

Replace most call to grep in run-make by a script that cat the input. Introduced a new `src/etc/cat-and-grep.sh` script (called in run-make as `$(CGREP)`), which prints the input and do a grep simultaneously. This is mainly used to debug spurious failures in run-make, such as the spurious error in #45810, as well as real errors such as #46126. (cc #40713) Some `grep` still remains, mainly the `grep -c` calls that count the number of matches and print the result to stdout.

matklad · 2023-06-26T15:08:45Z

Rather than writing our own shell library it might be nice to use https://docs.rs/xshell/latest/xshell/.

Yeah, xshell has the right scope here. It aims to be essentially an "in-process sh + coreutils", exactly for the purpose of doing shell-ly stuff without having any extra deps besides rust-std.

At the same time, currently xshell is more of a prototype and "works for me", rather than an 1.0 grade project. If we want to move to xshell, it'd be best if someone from t-compiler/contributors (except me, that is :-) ) has commit access to the crate and is ready to play a role of an active maintainer.

Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context. ### Basic Description of `run-make` V2 `run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps: 1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib. 2. We build the recipe `rmake.rs` and link in the support library. 3. We run the recipe to build and run the tests. `rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code. ### Planned Changes - [x] Get rid of the builder style patterns in `rmake_support` and instead use something like ```rust let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy - Zunstable-options"); ``` as per Nils' suggestion. This can probably use something like `xshell`. - [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`. - [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves. ### Disclaimer The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.

Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context. ### Basic Description of `run-make` V2 `run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps: 1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool lib. 2. We build the recipe `rmake.rs` and link in the support library. 3. We run the recipe to build and run the tests. `rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code. The support library is built using cargo, and so can depend on external crates if desired. The infrastructure implemented by this PR is very barebones, and is the minimally required infrastructure needed to build, run and pass the two example `run-make` tests ported over to the new infrastructure. ### Example `run-make` V2 test ```rs // ignore-tidy-linelength extern crate run_make_support; use std::path::PathBuf; use run_make_support::{aux_build, rustc}; fn main() { aux_build() .arg("--emit=metadata") .arg("stable.rs") .run(); let mut stable_path = PathBuf::from(env!("TMPDIR")); stable_path.push("libstable.rmeta"); let output = rustc() .arg("--emit=metadata") .arg("--extern") .arg(&format!("stable={}", &stable_path.to_string_lossy())) .arg("main.rs") .run(); let stderr = String::from_utf8_lossy(&output.stderr); let version = include_str!(concat!(env!("S"), "/src/version")); let expected_string = format!("stable since {}", version.trim()); assert!(stderr.contains(&expected_string)); } ```

Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example ## Preface See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context. ## Basic Description of `run-make` V2 `run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps: 1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool lib. 2. We build the recipe `rmake.rs` and link in the support library. 3. We run the recipe to build and run the tests. `rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code. The support library is built using cargo, and so can depend on external crates if desired. The infrastructure implemented by this PR is very barebones, and is the minimally required infrastructure needed to build, run and pass the two example `run-make` tests ported over to the new infrastructure. ### Example `run-make` V2 test ```rs // ignore-tidy-linelength extern crate run_make_support; use std::path::PathBuf; use run_make_support::{aux_build, rustc}; fn main() { aux_build() .arg("--emit=metadata") .arg("stable.rs") .run(); let mut stable_path = PathBuf::from(env!("TMPDIR")); stable_path.push("libstable.rmeta"); let output = rustc() .arg("--emit=metadata") .arg("--extern") .arg(&format!("stable={}", &stable_path.to_string_lossy())) .arg("main.rs") .run(); let stderr = String::from_utf8_lossy(&output.stderr); let version = include_str!(concat!(env!("S"), "/src/version")); let expected_string = format!("stable since {}", version.trim()); assert!(stderr.contains(&expected_string)); } ``` ## Follow Up Work - [ ] Adjust rustc-dev-guide docs

jieyouxu · 2024-03-01T19:19:40Z

There's now very basic infrastructure support for writing run-make tests in Rust instead of Makefiles, now that #113026 is merged!

#121876 is the tracking issue for porting over all of the remaining run-make tests to use Rust.

Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example ## Preface See [issue #40713: Switch run-make tests from Makefiles to rust](rust-lang/rust#40713) for more context. ## Basic Description of `run-make` V2 `run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps: 1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool lib. 2. We build the recipe `rmake.rs` and link in the support library. 3. We run the recipe to build and run the tests. `rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code. The support library is built using cargo, and so can depend on external crates if desired. The infrastructure implemented by this PR is very barebones, and is the minimally required infrastructure needed to build, run and pass the two example `run-make` tests ported over to the new infrastructure. ### Example `run-make` V2 test ```rs // ignore-tidy-linelength extern crate run_make_support; use std::path::PathBuf; use run_make_support::{aux_build, rustc}; fn main() { aux_build() .arg("--emit=metadata") .arg("stable.rs") .run(); let mut stable_path = PathBuf::from(env!("TMPDIR")); stable_path.push("libstable.rmeta"); let output = rustc() .arg("--emit=metadata") .arg("--extern") .arg(&format!("stable={}", &stable_path.to_string_lossy())) .arg("main.rs") .run(); let stderr = String::from_utf8_lossy(&output.stderr); let version = include_str!(concat!(env!("S"), "/src/version")); let expected_string = format!("stable since {}", version.trim()); assert!(stderr.contains(&expected_string)); } ``` ## Follow Up Work - [ ] Adjust rustc-dev-guide docs

WaffleLapkin · 2024-03-02T19:40:07Z

@jieyouxu is there any reason why you are still using "make" terminology? (rmake.rs, run_make_support)

I think since this does not involve make, it would be much less confusing to have something else. Like test-recipe.rs + test_recipe_support, since you used "recipe" terminology already.

jieyouxu · 2024-03-02T20:04:40Z

@jieyouxu is there any reason why you are still using "make" terminology? (rmake.rs, run_make_support)

I think since this does not involve make, it would be much less confusing to have something else. Like test-recipe.rs + test_recipe_support, since you used "recipe" terminology already.

Absolutely zero reason, it's still "make" because I didn't bother coming up with new names. Feel free to change the actual thing lol. I do note however, the entire test suite is called "run-make" so shrug.

WaffleLapkin · 2024-03-02T20:09:59Z

@jieyouxu I don't think I have time for that rn 😅

we could also rename the test suite 👀

TimNN added A-build A-testsuite Area: The testsuite used to check the correctness of rustc labels Mar 21, 2017

Mark-Simulacrum mentioned this issue May 16, 2017

consider linting test/run-pass/*.rs to ensure all have assert/assert_eq #23113

Closed

Mark-Simulacrum added I-nominated T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Jun 27, 2017

Mark-Simulacrum added I-nominated and removed I-nominated labels Jul 7, 2017

Mark-Simulacrum assigned TimNN Jul 7, 2017

Mark-Simulacrum removed the I-nominated label Jul 7, 2017

TimNN changed the title ~~Interest in switching run-make to python or rust~~ Switch run-make tests from make to rust Jul 8, 2017

TimNN changed the title ~~Switch run-make tests from make to rust~~ Switch run-make tests from Makefiles to rust Jul 8, 2017

Mark-Simulacrum added the C-enhancement Category: An issue proposing an enhancement or a PR with one. label Jul 27, 2017

kennytm mentioned this issue Nov 23, 2017

Replace most call to grep in run-make by a script that cat the input. #46207

Merged

Mark-Simulacrum mentioned this issue Feb 4, 2018

Run the run-make tests last, so more tests run on Windows when make is unavailable #47996

Merged

jonas-schievink added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) and removed T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) A-build labels Apr 21, 2019

Mark-Simulacrum mentioned this issue Dec 8, 2019

rustc_llvm fails to build if source directory contains whitespace #56650

Open

Mark-Simulacrum mentioned this issue Oct 2, 2021

Trim runtest #89240

Closed

jyn514 added E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. E-help-wanted Call for participation: Help is requested to fix this issue. labels Mar 30, 2023

Dylan-DPC unassigned TimNN Mar 30, 2023

jieyouxu mentioned this issue Jun 25, 2023

Introduce run-make V2 infrastructure, a run_make_support library and port over 2 tests as example #113026

Merged

1 task

Mark-Simulacrum mentioned this issue Jan 20, 2024

dump-ice-to-disk/check.sh: convert needless bashism in a /bin/sh script. #119992

Closed

This was referenced Mar 1, 2024

Update run-make test description rust-lang/rustc-dev-guide#1920

Merged

Tracking Issue for porting run-make tests to use Rust #121876

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch `run-make` tests from Makefiles to rust #40713

Switch `run-make` tests from Makefiles to rust #40713

TimNN commented Mar 21, 2017 •

edited

petrochenkov commented Mar 21, 2017

petrochenkov commented Mar 21, 2017

alexcrichton commented Mar 21, 2017

bstrie commented Mar 23, 2017

alexcrichton commented Mar 23, 2017

TimNN commented Jul 8, 2017

kennytm commented Jul 27, 2017 •

edited

matklad commented Jun 26, 2023

jieyouxu commented Mar 1, 2024 •

edited

WaffleLapkin commented Mar 2, 2024

jieyouxu commented Mar 2, 2024 •

edited

WaffleLapkin commented Mar 2, 2024

Switch run-make tests from Makefiles to rust #40713

Switch run-make tests from Makefiles to rust #40713

Comments

TimNN commented Mar 21, 2017 • edited

petrochenkov commented Mar 21, 2017

petrochenkov commented Mar 21, 2017

alexcrichton commented Mar 21, 2017

bstrie commented Mar 23, 2017

alexcrichton commented Mar 23, 2017

TimNN commented Jul 8, 2017

kennytm commented Jul 27, 2017 • edited

matklad commented Jun 26, 2023

jieyouxu commented Mar 1, 2024 • edited

WaffleLapkin commented Mar 2, 2024

jieyouxu commented Mar 2, 2024 • edited

WaffleLapkin commented Mar 2, 2024

Switch `run-make` tests from Makefiles to rust #40713

Switch `run-make` tests from Makefiles to rust #40713

TimNN commented Mar 21, 2017 •

edited

kennytm commented Jul 27, 2017 •

edited

jieyouxu commented Mar 1, 2024 •

edited

jieyouxu commented Mar 2, 2024 •

edited