IPEP 14: Notebook Regression Tests
| Status | Active |
| Author | @dwyde |
| Created | April 2013 |
| Updated | April 2013 |
| Discussion | |
| Implementation |
The HTML notebook currently lacks unit or integration tests. Regression tests would help with potential refactoring, and are generally good to have.
Features to test:
- User interface: toolbars, keyboard shortcuts
- JavaScript code
- Code cell execution
- Code output event handling / rendering
- Text cell (Markdown / LaTeX) rendering
CasperJS is a JavaScript test utility built on top of PhantomJS, a headless WebKit implementation.
- Can test code cell execution from JavaScript
- Multi-page navigation
- xUnit-friendly output option
- Extra software dependencies
Selenium is a web browser automation library with Python bindings. It can be installed with pip.
- Integrates with Python's standard
unittestmodule - Can test code execution in the kernel
- Tests still need to execute strings of JavaScript, for example, to get text from CodeMirror input areas.
- Relatively slow
- An extra software dependency
Customize whether to run tests in Firefox, Chrome, Internet Explorer, etc.
- Use a wrapper script, or a config file
Make the tests aware of a notebook server's URL
- Run a notebook server in a subprocess, and write the port to the test process? This is straightforward with
multiprocessing.
QUnit is a JavaScript unit test framework.
- No additional software dependencies: just HTML, JavaScript, and CSS.
- Can't easily test executing code in a kernel
- Can it directly test the menubar and other UI elements, or just JavaScript models?
Create an HTML page that includes the notebook's JavaScript, and QUnit's code.
- Inherit a tests page from the notebook.html template, and add an associated handler to the Tornado app
Jasmine is a JavaScript behavior-driven development (BDD) framework. It's a head-to-head competitor with QUnit, modeled after test tools in Ruby on Rails.
Behavior-driven development, if you're a fan.
DOM-less test runner?
Similar to QUnit.
The notebook's regression tests should check that code cell execution works, which is a vote for either CasperJS or Selenium.
CasperJS has less of an impedance mismatch for testing JavaScript - Selenium tests have to write JavaScript code as strings.