Migration from Selenium 1 to Selenium 2
This page was written during the migration from Selenium 1 API to Selenium 2 API (Webdriver). Most of the work should be done now.
The testing framework uses Selenium to control the browser. However, only a small subset of the Selenium API is needed, essentially to start/end browser, load pages and get data from the browser. The test runner is written in Python and uses the Selenium Python API. The basic component and communication are:
[test runner] <=> [Selenium server] <=> [Browser]
To test UI features, user interactions (mouse click, etc) are currently simulated by Javascript events inside the test pages. This is not perfect and Selenium 2's "native events" (fired at the OS level) may be a better way to achieve that result.
There is an important difference between the ways Selenium 1 and 2 control the browser. Selenium 1 uses a common Javascript program and injects the code inside the browser. Selenium 2 has taken Google's Webdriver idea: choose the method which is the most adapted to each browser. The first approach allows to have a wide browser support and uniform set up while the second approach allows a finer and faster control of the browser. The Selenium 2 server understands commands from both versions. However, to use Webdriver's approach, some additional browser-specific programs are sometimes needed. Hence, the changes to do in the MathJax testing framework are essentially:
- Upgrade the Selenium server (easy).
- Update the test runner to use the Selenium 2 (Webdriver) Python API.
- Install and set up additional programs on test machines.
Selenium 2 adds support for more platforms (Opera, Android...) but drivers for other browsers (Safari, Konqueror...) are not available. Hence the MathJax testing framework allows to choose between Selenium 1 or 2 API. Browser-specific details are given in other sections below. The next section describes the update of the test runner.
The Selenium API is used in two places in testRunner/:
- reftest.py, reftest::runTest. This is the function executing a single test. When a test fails, a Python exception may be raised. The function has been updated to take into account the error message in Webdriver's exception. Other data attached to Webdriver's exception, such as screen capture, may also be relevant to consider.
- seleniumMathJax.py: This implements a class to describe a Selenium object. Initially, this class inherited from the Selenium object of the Selenium 1 API. It has now been updated to be a container, with two member variables (mWebdriver for Selenium 2 API and mSelenium for Selenium 1 API). The member functions contain if statements to use one variable or the other, depending on the API requested.
The commands used in Selenium 1 API to get information from the browser have been easily converted to their Webdriver counterparts. However, the function wait_for_condition (used to wait the end of the test page execution, whose status is indicated by the value of the class attribute on the <html/> element) is no longer available in Webdriver API. Now, the javascript header page creates an element of id "webdriver_test_complete" and seleniumMathJax.py uses find_element_by_id to do an "explicit wait".
With Selenium 1 API, some "sleep time" of arbitrary duration were sometimes needed after some commands. With the new Webdriver API, they are no longer necessary and have been removed. Hence, we got rid of a lot of random timing issue.
Taking screenshot of the browser canvas is important for visual reftests. For that purpose, we used not really reliable methods to put the browser in fullscreen mode and to determine the browser canvas. With Webdriver API, this is no longer needed and we can even have several testing instances running on the same OS simultaneously, without worrying about conflicts when screenshots are taken.
To choose Internet Explorer document mode, Java keyboard events were used to control the browser menu. They were also used to close some browser windows when the Selenium commands for that purpose fails. Using Java keyboard events does not seem currently possible with Python Webdriver API, and so these issues remain. See issue 2564
The testing framework also has a configuration option "browserPath" to provide a custom browser executable. This may be useful to test other browser versions than the default install on the system. Currently, this option is used with Selenium 1 but not with Python Webdriver API.
Finally, it may also be important to clear browsing data. For example when we update MathJax, we need to clear the cache before testing again. This was not possible with Selenium 1 API. I have not looked yet whether Webdriver API may help.