[continued from Standard treatment]
There are a number of other testing techniques that are used during development, and I want to touch briefly on a few.
One essential technique is known as “compatibility testing”. As the name implies, this is testing the software for compatibility on a variety of different system configurations. There are companies that will perform extensive compatibility testing, but this is not inexpensive. Alpha and beta testing should cover a range of systems, but it will be far from comprehensive.
For a Windows product, one must test on some flavors of Win9x and NT, at an absolute minimum, and preferably on every supported operating system. Game and multimedia products need to be tested with different video cards and sound cards. Products with printing features need to be checked on different types of printers, including at least a color inkjet and a laser printer, from different manufacturers. In short, you must cover as much of your target audience as absolutely possible.
Another external testing technique, related to compatibility testing, is product certification. This involves submitting your software for certification according to the rules of some program. Instead of checking different system configurations, product certification programs check other criteria, depending on the goals of the particular certification. These range in cost from free to very expensive.
For a slightly less formal review of the usability and general quality of the software, one can conduct “focus group” testing. Focus groups are essentially a collection of people in the target audience who are brought together in one location specifically to give their opinions and feedback. Professional firms can conduct such groups with quasi-scientific questionnaires, hidden cameras, and written analysis, for a tidy sum.
The easier and, in my experience, no less effective method to perform focus group testing is to find a location, such as the computer lab in a local school, and advertise free pizza and drinks for computer users who will show up and try your new product. I cannot comment on how this would work for business products, but it works well for games.
Finally, throughout the entire testing process, you need to conduct “regression testing”. Regression testing is a method of making sure that bugs that were fixed are not reintroduced into the program. This concept is really as simple as trying to reproduce each of the fixed bugs and making certain that they have not reappeared.
My first exposure to regression testing was a spiral notebook into which every bug was written as it was reported and checked as it was solved. Before we would send a game build to the publisher, we simply tested each item in the notebook as part of the test plan. It hardly needs to be more complicated than that.
[continued in Gamma testing?]