Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prototype Rust Autotesting #482

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

1whatleytay
Copy link

@1whatleytay 1whatleytay commented Jan 13, 2024

Rust has a standard testing framework called cargo test (previously libtest). This pull request pulls support for this target into MarkUs. Tests are automatically inferred and run.

Some caveats:

  • cargo-nextest, an alternative test runner is used instead of cargo-test. Machine readable test output is nightly-only for cargo-test, but cargo-nextest supports it (experimentally) in all versions. It is also built from source so it may add time to project setup.
  • The test-scripts schema property is present but ignored. Information about test files seems to be elided when built (only the names of modules are kept). I might replace this with a run-modules property instead if possible.

@1whatleytay
Copy link
Author

Add test-modules as an array, which generated UI as follows. I'm not a huge fan of this interface, and the debug string in the item names. I might be able to tinker the schema or UI gen to get a better interface, but I think instead I'm going to force one module per test group. My justification is:

  • Rust modules are composable (ex. pub use), so a new module can probably be created with just the needed tests.
  • Interface is simple and provides some room for a description.

The interface before with the test-modules: array schema is as follows. I can confirm this works with testing!

Screenshot 2024-01-14 at 6 32 15 PM

@1whatleytay 1whatleytay marked this pull request as ready for review January 15, 2024 00:35
@1whatleytay
Copy link
Author

Tested the behaviour of tests on compilation failure, and made the mentioned tweaks for running certain modules only. It's one rust module per test group now. Removing draft marker.

@1whatleytay
Copy link
Author

1whatleytay commented Jan 15, 2024

Fixed a bug that would cause the autotester to fail with an empty test_module field. Made a last minute change that broke something.

Screenshot 2024-01-15 at 1 29 45 PM

Comment on lines +119 to +120
if module is not None and "-" not in module:
command.append(module)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@1whatleytay I see on nextest's documentation that you're treating module as "[FILTERS]". I wasn't able to figure out exactly how that part worked, but at the very least it seems to match more than just the module name? With the sample files you added on your MarkUs PR, using a value of test_enqueue does a filter based on test name.

But, I also see the -E '...' test filter expression option, which seems clearly defined. I wonder if this is a better target for user configuration, if it supports filtering both by package name (e.g. package(test)?) and test function name (e.g. test(dequeue)). It would require the instructor to understand this syntax, but I think that's okay?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants