This chapter includes real-world advice on how to plan, organize and version control your tests. Other handy tips tell you how to get the most out of "smoke tests" and how to communicate your test results with the team.
Now you might be thinking "huh? I haven't even started yet and I have to think about communicating my results?" The answer is a resounding YES! Unfortunately, Quality Assurance and test automation specifically, at least in the software world, tends to get a bad rap. Creating solid test automation that can stand up over time and provide valuable ongoing feedback is a difficult task which leads a lot of people (read management) to think time and effort spent working on automation is wasted.
To solve this problem, you'll want to focus on getting your test results published quickly and consistently to prove that your efforts are worth the investment. TestComplete provides some facilities for producing log output though you'll want to be sure and iron out your strategy from the start. For example, you may want to setup a web server where you can publish results. In fact, TestComplete supports exporting log results to HTML which could be a good starting place.
The preparation we're referring to has to do with your choice of testing tool, TestComplete. Although it's important to plan the application features to be tested, it's also important to be prepared with respect to the testing facilities available in TestComplete. You should gain a thorough understanding of TestComplete features before you embark on writing test automation to avoid a situation where your weeks into test development only to discover a feature that could have saved you numerous hours or days.
TestComplete is rich with features that make test automation easier although like any development tool it provides many different ways to solve the same problem. For example, TestComplete has a powerful and indeed almost alluring recorder, making it easy to quickly create automated tests. While a test recorder is a great tool, recorded tests tend to be more brittle than hand written tests because they capture a single iteration at a given point in time and don't take into account unexpected events like an error dialog popping up. The alternative is handwritten tests where you can methodically plan how the test will react in unexpected circumstances. The down side of handwritten tests is they tend to take longer to develop though over the long run they'll likely require less maintenance because of the tendency to design the test more rigorously.
TestComplete provides the ability to customize the Project Items that are included by default on a new project using Project Templates. In order to maintain consistency across your testing organization you should work towards creating a project template(s) that's pre-configured for your test environment. For example, if you're testing a Window 32-bit GUI application, create a Project Template that has the Tested Applications node with your application already added to it. Also, look closely at how you can utilize Aliases to avoid using long dotted object names like:
Sys.Process("iexplore").IEFrame(0).Window("WorkerW", "", 1).Window( "ReBarWindow32", "", 1).Window("ComboBoxEx32", "", 1).Window("ComboBox", "", 1).Window("Edit", "", 1)
TestComplete includes an automatic Name Mapping feature has been added to make this even easier.
TestComplete is a development tool and as such the projects, scripts, data, etc. used in your tests should be kept under version control. If this is something you're already doing, skip ahead. If not, you should consider version control the next priority and work to get your test suites under version control. There are lots of options including some excellent Open Source projects which are freely available like Subversion, otherwise known as SVN. While it's beyond the scope of this document to discuss the specific merits of source control it's a subject that shouldn't be ignored.
Arguably, the first test you should focus on is a "smoke test". In this section we'll discuss what a smoke test is and provide guidance as to how to construct the test in such a way as to maximize its effectiveness.
Probably the most important test you'll write is your Smoke Test, at least it should be. Typically, the goal of a Smoke Test is to verify that the latest build delivered to QA is either worth further consideration or "DOA" (dead on arrival). The smoke test is crucial because of the time savings it can provide both R&D and QA. A good Smoke Test should:
Run quickly -- a smoke test should not last for hours but minutes and test the most crucial functionality.
Fail quickly -- as soon as a failure is detected in the smoke test should end and trigger a failure notification.
Cover a broad range of functionality, focusing on breadth not depth. Require minimal setup/configuration of the application under test. Be setup to run against every build.
Adapt over time as the application under test evolves. Serve as a model for new test automation.
If you take the time to organize your Smoke Test to cover these goals you will undoubtedly save time and resources over the long run. Your Smoke Test should serve as a model that embodies your "best practices" from which your QA team can draw from for their own tests.
If you've never written a Smoke Test before, start small. In the beginning simply get the smoke test to verify even a single piece of functionality consistently. Over time, work to increase its coverage but remain focused on the quality of the test. It's unacceptable to have a Smoke Test that can't run consistently and without problems.
Once your Smoke Test is written and under version control the next step is to automate its execution. SmartBear has a product called Automated Build Studio which can automate your TestComplete tests and trigger them to execute when a new build is delivered to QA,a process called Continuous Integration or "CI". In addition to Automated Build Studio there are Open Source CI servers such as CruiseControl.NET.
The main benefit of a CI server is to reduce the amount of time it takes to execute your test automation as well as ensure that it executes against every build. By setting up a CI server you can not only free your QA engineers from having to manually execute their tests. CI helps quickly identify tests that are unable to consistently run to completion and may require closer scrutiny. In addition, many CI servers include a means of publishing test results providing for greater visibility into the automation efforts.
The following are general tips and best practices to help you get the most out of TestComplete:
Record/Playback is a quick and easy way to get automated tests up and running but tend to be brittle leading to problems when the application changes, etc.
Use the TestItems of the project as a framework for Test Cases (see the "Project Organization" chapter).
Separate Data from the Test Framework (see "Data Driven Testing").
Use the code metrics of the Code Explorer to improve the quality of the script code (see Overview of TestComplete).
Use reusable routines whenever possible (see the "Project Organization" chapter). Keep routines short, i.e. less than a page of code.
Use meaningful variable names. The default variable names of a recording are not acceptable for production use.
The following are tips for making effective use of TestComplete when testing web pages:
Make sure that "Make the Page object a child of the browser process" is selected in the Project properties under Open Applications | Web Testing.
Use the "Tree Model" for best performance. You can set this under the Project properties in the General options.
In this chapter you learned how to plan, organize and version control your tests. You also reviewed tips on how to get the most out of "smoke tests" and how to communicate your test results with the team.