Writing tests is not an easy task as it can seem at first glance. In many cases, writing a good test can be as hard as writing a good application code. While in the application code the readability and maintainability are both desirable features, it is not an absolute requirement in all cases. For tests, on the other hand, readability and maintainability is a must. This article will explore a simple approach to make your tests better and show how you can improve your tests by breaking them into steps. Because there are different types, we will specifically focus on the writing of a functional test.

Something to Test

To explore a manner in which we can make a test better, we need to write one. For this we need something to test. What could it be? After much thought and for no obvious reason a simple ls utility found on all Unix-like systems comes to mind.

ls foo

ls: cannot access 'foo': No such file or directory

What Will We Test?

Any good test that is worth the memory in which it is stored, needs to verify one or more requirements. Having picked the ls utility, we need to come up with a simple requirement that we can verify. Let’s pick the following requirement:

When a file or a directory does not exist then an error message should be printed in stderr and the exit code should be 2.

Having the requirement ready we can proceed to create a test. We want to get things done as we will use the ad-hoc method.

First Step

There are plenty of test frameworks which we can use to write a test but of course, we will use for this purpose as it has a convenient Connect module that provides Shell class which will allow us to test ls utility in an intuitive way.

Let’s throw a few lines together to get us going into test.py.

from testflows.core import Test
from testflows.connect import Shell

with Test("Check 'ls' behaviour when file or directory does not exist"):
     with Shell() as bash:
        bash('ls foo')

And, run it.

python3 ./test.py

Nov 25,2019 18:42:19   ⟥  Test Check 'ls' behaviour when file or directory does not exist
               215ms        [bash] bash# ls foo
               216ms        [bash] ls: cannot access 'foo': No such file or directory
               217ms        [bash] bash# echo $?
               217ms        [bash] 2
               217ms        [bash] bash#
               232ms        [bash] 
               233ms   ⟥⟤ OK Check 'ls' behaviour when file or directory does not exist, /Check 'ls' behaviour when file or directory does not exist

Passing

✔ [ OK ] /Check 'ls' behaviour when file or directory does not exist

1 test (1 ok)

Total time 233ms

The output is very much the same as you would see in a terminal window. The result of the test is OK as no exception has been raised. However, we have not asserted anything inside the test. Now is the time to add two assertions to verify the requirement. One to check the error message and another to check the exit code.

from testflows.core import Test
from testflows.connect import Shell
from testflows.asserts import error

with Test("Check 'ls' behaviour when file or directory does not exist"):
     with Shell() as bash:
        cmd = bash('ls foo')
        assert "ls: cannot access 'foo': No such file or directory" in cmd.output, error()
        assert cmd.exitcode == 2, error()

The code is short and sweet. We have added two assertions and used Python’s default assert statement and did not need to use any special assertion libraries. However, we did use the Asserts module to provide us a useful error() function. This function prints detailed debugging information when the assertion fails. The assertion did not fail for me and hopefully, it did not fail for you too. If it did then the error() function should have provided you with details about what went wrong. Hey, make it fail and see what happens!

Taking the Next Step

Are we done with the test? We could be, but we also could think about what would happen if our test was much more complex. Have we covered all the important cases? No, only one. What else can be tested? In truth, many more cases exist that we could and should check. For example, we could also check what happens if a path that we pass to the ls differs only by one last character from a valid one. Maybe there is a bug and then it will fail?

We will leave the test as this to keep it simple. But will start thinking about a test as nothing but an implementation of a test procedure. For a simple test, the procedure is simple and for a complex test, the procedure helps understand what the test is doing. The test procedure should be formally defined in a test specification but who has time to write one? Unless you work on a project where it is required, you probably don’t have the time.

Nevertheless, whether or not we identify the test procedure beforehand in the form of a formal test specification, or just throwing a test together on the go, the procedure is always there. Therefore, we can identify it. So let’s revisit the test and add some comments to identify the procedure.

with Test("Check 'ls' behaviour when file or directory does not exist"):
     with Shell() as bash:
        #  execute 'ls' with an invalid path"
        cmd = bash('ls foo')
        #  check error message and exit code"):
        assert "ls: cannot access 'foo': No such file or directory" in cmd.output, error()
        assert cmd.exitcode == 2, error()

The test has a simple procedure which is made out of at least two steps. Step one, execute ls with an invalid path. Step two, check the error message and exit code. Right, now we have test procedure added to the test as comments but could we do better? Yes, we can. We can add explicit steps to the test by using a Step.

from testflows.core import Test, Step
from testflows.connect import Shell
from testflows.asserts import error

with Test("Check 'ls' behaviour when file or directory does not exist"):
     with Shell() as bash:
        with Step("execute 'ls' with an invalid path"):
            cmd = bash('ls foo')
        with Step("check error message and exit code"):
            assert "ls: cannot access 'foo': No such file or directory" in cmd.output, error()
            assert cmd.exitcode == 2, error()

When executed with , the test above produces the output that includes the test steps.

Nov 26,2019 17:57:25   ⟥  Test Check 'ls' behaviour when file or directory does not exist
Nov 26,2019 17:57:25     ⟥  Step execute 'ls' with an invalid path
               174ms          [bash] bash# ls foo
               178ms          [bash] ls: cannot access 'foo': No such file or directory
               179ms          [bash] bash# echo $?
               179ms          [bash] 2
               179ms          [bash] bash#
               179ms     ⟥⟤ OK execute 'ls' with an invalid path, /Check 'ls' behaviour when file or directory does not exist/execute 'ls' with an invalid path
Nov 26,2019 17:57:25     ⟥  Step check error message and exit code
                 2ms     ⟥⟤ OK check error message and exit code, /Check 'ls' behaviour when file or directory does not exist/check error message and exit code
               193ms          [bash] 
               196ms   ⟥⟤ OK Check 'ls' behaviour when file or directory does not exist, /Check 'ls' behaviour when file or directory does not exist

The test steps are even more apparent if we use the short output format.

python3 ./test.py -o short

Test Check 'ls' behaviour when file or directory does not exist
  Step execute 'ls' with an invalid path
  OK
  Step check error message and exit code
  OK
OK

Passing

✔ [ OK ] /Check 'ls' behaviour when file or directory does not exist

1 test (1 ok)
2 steps (2 ok)

Total time 206ms

Read through the output and the code. By adding steps we have added documentation to our test code and the test output at the same time. Did we just kill two birds with one stone? Figuratively speaking, yes, we did. This is no small feat. For nontrivial tests, having test procedure explicitly documented in the code and be present in the output is a big advantage. It not only helps to analyze test results by looking at the output but also helps to understand the test code itself.

Taking a Step in The Right Direction

Take a look at your tests and see how a simple idea of breaking your tests into steps can help make your tests more readable and easier to debug. The cost of test maintenance far outweighs a few additional lines that can be added to make your tests better. Unfortunately, there a plethora of test frameworks out there that do not provide any support for defining steps inside a test. They simply ignore the importance of explicitly identifying test procedure. Well, that’s a mistake, breaking tests into steps is literally a step in the right direction.

Share It