Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unit tests, integration tests and fuzzing #1

Open
asumagic opened this issue Sep 6, 2021 · 0 comments
Open

Unit tests, integration tests and fuzzing #1

asumagic opened this issue Sep 6, 2021 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@asumagic
Copy link
Collaborator

asumagic commented Sep 6, 2021

This would be a huge nice to have by now because:

  • We are reaching the point where accurate emulation matters. Automatically testing edge cases especially for some API drawing functions may be useful, e.g. to optimize code more aggressively while reducing the testing workload.
  • Test-driven development would make sense in some cases while implementing the API. String rendering, RNG, and some other things could benefit from this approach, in particular.
  • We could probably do some nice testing against PICO-8 by injecting a printh of the PICO-8 memory somewhere in update functions. This would allow detecting subtle bugs and differences in the implementation especially if performed automatically.
  • Fuzzing the API to detect differences in parameter count handling or argument type handling may unravel some inaccuracies in the interface implementation.

This raises a few issues:

  • CMake does not support cross-compilation for different architectures within the same build directory. Making two build directories is possible, but this would likely need some tinkering with VSC so that the CMake integration isn't dogcrap to use. If external development is ever opened up this has to be tackled. Ideally only one command should be run to compile for both arches and execute tests, at least as an option. Emulating ARMv6-M with qemu or something is an option but would be complicated to set up, would make instrumentation difficult, and is of limited use since recent bugs that were difficult to debug happened both on the Pico and on the desktop builds.
  • How to even execute tests? We could use CTest from CMake, but how about running the tests themselves? Integration tests might be easier to implement in pure Python, unit tests might be a lot easier to make using Catch or a similar library.
@asumagic asumagic added the enhancement New feature or request label Sep 6, 2021
@asumagic asumagic self-assigned this Sep 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant