There are a lot of metrics that help us evaluate the quality of a piece of software and how much value it brings to our customers, but one metric stands out from the rest—working software. As software developers, our aim is perfection. As human beings, mistakes are inevitable. However, it’s possible to mitigate damage by creating and running tests.
Why do tests matter?
Software is everywhere, from everyday items like phones and TVs to hospitals and airplanes. Software failures not only cost money, but they can also cost lives. It’s estimated that 100 to 900 deaths are caused by software problems and computer failures in the UK's National Health Service.
In August 2016, a slot machine at Resorts World Casino printed a prize ticket of $42,949,672.76 as the result of an overflow bug. The casino refused to pay this amount and called it a malfunction, showing in their defense that the machine clearly stated that the maximum payout was $10,000, so any prize exceeding that amount had to be the result of a programming bug. The Iowa Supreme Court ruled in favor of the casino.
Useful tests could have prevented all the incurred legal costs and customer dissatisfaction in the case stated above. In the same way, problems in our software can be mitigated when we have reliable tests in place.
What are tests anyway?
We test things every day. For example, when we buy new electronics, we start asserting all functionalities are working correctly. Tests are a way to determine that something is behaving as expected. The problem with manual testing, although it's important, is that it's not possible to assert large, complex systems will work in any given situation.
The same happens with software—sometimes it's so complicated that it could take an entire human lifetime to make sure it's flawless. To avoid this scenario, we rely on computers to run the assertions we find useful to be sure software is working correctly.
There are different kinds of tests; the most common are integration, functional, acceptance, performance, and unit tests. All these tests have different granularities, and a good rule of thumb is to use the test pyramid to help you establish your own test suite. Since the base of a healthy pyramid is unit tests, we’ll focus on unit tests here.
What is a unit test?
Gerard Meszaros, the author of the xUnit Test Patterns book and one of the pioneers on collecting testing patterns, defines a unit test as:
A test that verifies the behavior of some small part of the overall system. What makes a test a unit test is that the system under test (SUT) is a very small subset of the overall system and may be unrecognizable to someone who is not involved in building the software.
He also shares wise words about the size of a unit test:
The actual SUT may be as small as a single object or method.
There are different variants of the definition given by different authors, but this one is more than enough to gain a good understanding of how we’re going to use the term here.
How do I know a unit test is useful?
Kent Beck, who is credited with having developed or "rediscovered" the TDD technique, compiled a list of properties that useful tests can present. For the sake of simplicity—and brevity—the code example won’t cover all of them, but it will help us identify some. As an engineer, I find it easier to explain testing using code, so let’s see how we can identify these properties using an example.
Recall the slot machine error previously mentioned. This is an oversimplified version of its code (the GameEngine class was omitted as it’s not very important).
As stated before, we use tests to assert something is behaving correctly. There are only two behaviors exposed in the SlotMachine class: add_credits and play. The other two methods are implementation details that are hidden from the software user. Changing them shouldn’t change the test output. With that in mind, let’s first see how we could write a test for the add_credits method.
Let’s start with the test name. One of the properties Kent Beck enumerated was Readability. The test name should communicate clearly to the reader the motivation (use case) it was written for. Naming tests using just the method/function name or with suffixes like test_method_raises_error or test_method_is_ok will force the reader to go read the implementation or, even worse, create the same test again.
While we’re still talking about readability, note that the test is split into three different sections separated by new lines. The first section arranges everything needed. The second section calls the method acting in the subject under test, and the third section asserts if the expected outcome is achieved. The idea behind the pattern is to help tell the story using the Given, Then, When pattern from behavior-driven development (BDD).
Although the test above is readable if it fails, the assertion error will be AssertionError: False is not true, which is true, but not Specific enough. When we’re running a large test suite, we want all errors to be as obvious as possible. The xUnit implementations provide a way to make the assertion error/failure better, which is shown below.
By setting the msg, we can make the assertion error say something more specific, like AssertionError: False is not true: Balance was increased by 1, expected was 10.
If you notice test names are starting to get repetitive, maybe it’s time to use the unittest subTest feature. This feature allows you group related tests into a single test method as shown below.
When using subTest, make sure you keep the tests Isolated as all subtests will share the same state of the parent test method. So, if we change the state of the slots object in one subtest, it will be reflected in the next ones, and that can lead to flaky tests.
Since this is an oversimplified example, we’re not talking about external dependencies that would slow down tests or mocking and its common problems when test code quality isn’t taken seriously. There is much more to be said about testing, but a good start is xUnit Patterns and, for Python-specific tests, our recent blog, Top 5 reasons why you should migrate to Pytest.
Unit tests can save you time and a lot of money if your test suite embraces the property of readability as described by Kent Beck and explained in the examples above. Useful tests should work as an oracle when deciding when to deploy code to production and are a powerful tool to increase a team's morale and confidence.
Writing tests is a difficult art to master, but the more you write, the better you’ll get.