Making Your Tests Better or How to Break Your Tests Into Steps
Writing tests is not an easy task as it can seem at first glance. In many cases, writing a good test can be as hard as writing a good application code. While in the application code the readability and maintainability are both...
Writing tests is not an easy task as it can seem at first glance. In many cases, writing a good test can be as hard as writing a good application code. While in the application code the readability and maintainability are both desirable features, it is not an absolute requirement in all cases. For tests, on the other hand, readability and maintainability is a must. This article will explore a simple approach to make your tests better and show how you can improve your tests by breaking them into steps. Because there are different types, we will specifically focus on the writing of a functional test.
Something to Test
To explore a manner in which we can make a test better, we need to write one. For this
we need something to test. What could it be? After much thought and for no
obvious reason a simple ls
utility found on all Unix-like systems comes to mind.
1 | ls foo |
1 | ls: cannot access 'foo': No such file or directory |
What Will We Test?
Any good test that is worth the memory in which it is stored, needs to verify one or more requirements. Having picked
the ls
utility, we need to come up with a simple requirement
that we can verify. Let’s pick the following requirement:
When a file or a directory does not exist then an error message should be printed in
stderr
and the exit code should be2
.
Having the requirement ready we can proceed to create a test. We want to get things done as we will use the ad-hoc method.
First Step
There are plenty of test frameworks which we can use to write a test but
of course, we will use for this purpose as it
has a convenient Connect module that provides Shell class
which will allow us to test
ls
utility in an intuitive way.
Let’s throw a few lines together to get us going into test.py
.
1 | from testflows.core import Test |
And, run it.
1 | python3 ./test.py |
1 | Nov 25,2019 18:42:19 ⟥ Test Check 'ls' behaviour when file or directory does not exist |
The output is very much the same as you would see in a terminal window.
The result of the test is OK
as no exception has been raised. However,
we have not asserted anything inside the test. Now is the time to add two
assertions to verify the requirement.
One to check the error message and another to check the exit code.
1 | from testflows.core import Test |
The code is short and sweet. We have added two assertions and used Python’s default assert statement and did not need to use any special assertion libraries. However, we did use the Asserts module to provide us a useful error() function. This function prints detailed debugging information when the assertion fails. The assertion did not fail for me and hopefully, it did not fail for you too. If it did then the error() function should have provided you with details about what went wrong. Hey, make it fail and see what happens!
Taking the Next Step
Are we done with the test? We could be, but we also could think about what would happen if our
test was much more complex. Have we covered all the important cases? No, only one.
What else can be tested? In truth, many more cases exist that we could and should check.
For example, we could also check what happens if a path that we pass to the ls
differs only
by one last character from a valid one. Maybe there is a bug and then it will fail?
We will leave the test as this to keep it simple. But will start thinking about a test as nothing but an implementation of a test procedure. For a simple test, the procedure is simple and for a complex test, the procedure helps understand what the test is doing. The test procedure should be formally defined in a test specification but who has time to write one? Unless you work on a project where it is required, you probably don’t have the time.
Nevertheless, whether or not we identify the test procedure beforehand in the form of a formal test specification, or just throwing a test together on the go, the procedure is always there. Therefore, we can identify it. So let’s revisit the test and add some comments to identify the procedure.
1 | with Test("Check 'ls' behaviour when file or directory does not exist"): |
The test has a simple procedure which is made out of at least two steps.
Step one, execute ls
with an invalid path. Step two, check the error message and exit code.
Right, now we have test procedure added to the test as comments but could we do better?
Yes, we can. We can add explicit steps to the test by using a Step.
1 | from testflows.core import Test, Step |
When executed with , the test above produces the output that includes
the test steps.
1 | Nov 26,2019 17:57:25 ⟥ Test Check 'ls' behaviour when file or directory does not exist |
The test steps are even more apparent if we use the short output format.
1 | python3 ./test.py -o short |
1 | Test Check 'ls' behaviour when file or directory does not exist |
Read through the output and the code. By adding steps we have added documentation to our test code and the test output at the same time. Did we just kill two birds with one stone? Figuratively speaking, yes, we did. This is no small feat. For nontrivial tests, having test procedure explicitly documented in the code and be present in the output is a big advantage. It not only helps to analyze test results by looking at the output but also helps to understand the test code itself.
Taking a Step in The Right Direction
Take a look at your tests and see how a simple idea of breaking your tests into steps can help make your tests more readable and easier to debug. The cost of test maintenance far outweighs a few additional lines that can be added to make your tests better. Unfortunately, there a plethora of test frameworks out there that do not provide any support for defining steps inside a test. They simply ignore the importance of explicitly identifying test procedure. Well, that’s a mistake, breaking tests into steps is literally a step in the right direction.