<!-- agents: TestFlows, a blog post. Index: https://testflows.com/llms.txt -->

# Get Your Software Covered Using Covering Arrays

Published 2023-10-02 by Vitaliy Zakaznikov.

For software systems with any non-trivial number of parameters, exhaustively testing all possible combinations of parameter values is not feasible.
For example, if we have `10` parameters (`k = 10`) that each have `10` possible values (`v = 10`), the
number of all possibilities is {% katex %}v^k=10^{10} = {10}_{billion}{% endkatex %}, thus requiring 10 billion tests for complete coverage. <!-- more -->

Given that exhaustive testing might not be practical, a covering array (CA) could give us a much smaller
number of tests if we choose to check all possible interactions only between some fixed number
of parameters at least once, where an interaction is some specific combination, where order does not matter,
of some `t` number of parameters, covering all possible values that each selected parameter could have.

The US National Institute of Standards and Technology (NIST) has an excellent [Introduction to Covering Arrays](https://math.nist.gov/coveringarrays/coveringarray.html) page, if you are not familiar with them. Note that in the literature,
a covering array is usually defined as {% katex %} CA(t,k,v) {% endkatex %},
where `t` is the strength of the array, `k` is the number of parameters, and `v`
is the number of values. In most cases, it is assumed that each parameter
has the same number of values, `v`, and such arrays are referred to
as uniform covering arrays.

The NIST, upon request, provides a tool called [Advanced Combinatorial Testing System (ACTS)](https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software), which runs on Java platforms and can be used to generate covering arrays to ensure t-way coverage of input parameter values using advanced algorithms such as [IPOG-F](https://math.nist.gov/coveringarrays/ipof/ipof-abstract.html). However, for tests developed in Python, a pure Python implementation would be preferred if the number of parameters is not too large. If the number of parameters is large, then you can use ACTS or check
if one of the [NIST-provided precomputed covering arrays generated by IPOG-F](https://math.nist.gov/coveringarrays/ipof/ipof-results.html) can be applied.

**{% testflows %}**, starting with version [2.1.5](https://pypi.org/project/testflows/2.1.5/), provides a `CoveringArray` class to allow you to calculate a covering array
for some `k` parameters having the same or different number of possible values. Therefore, it allows you to generate both uniform and non-uniform covering arrays. The class uses [IPOG], a pure Python implementation of an in-parameter-order algorithm as described in [IPOG: A General Strategy for T-Way Software Testing] by Yu Lei et al.

The `CoveringArray(parameters, strength=2)` class is provided by the `testflows.combinatorics` module and takes the following arguments:

* `parameters` specifies parameter names and their possible values as a `dict[str, list[value]]`, where *key* is the parameter name and
   *value* is a list of possible values for a given parameter.
* `strength` specifies the strength `t` of the covering array that indicates the number of parameters
   in each combination, for which all possible interactions will be checked.
   If `strength` equals the number of parameters, then you get the exhaustive case.

The return value of the `CoveringArray(parameters, strength=2)` is a `CoveringArray` object that is an iterable
of tests, where each test is a dictionary, with each `key` being the parameter name and its `value` being the parameter value.

For example,

```python
from testflows.combinatorics import CoveringArray

parameters = {"a": [0, 1], "b": ["a", "b"], "c": [0, 1, 2], "d": ["d0", "d1"]}

print(CoveringArray(parameters, strength=2))
```

Gives the following output:

```bash
CoveringArray({'a': [0, 1], 'b': ['a', 'b'], 'c': [0, 1, 2], 'd': ['d0', 'd1']},2)[
6
a b c d
-------
0 b 2 d1
0 a 1 d0
1 b 1 d1
1 a 2 d0
0 b 0 d0
1 a 0 d1
]
```

Given that in the example above, the `strength = 2`, all possible 2-way (pairwise)
combinations of parameters `a`, `b`, `c`, and `d` are the following:

```python
[('a', 'b'), ('a', 'c'), ('a', 'd'), ('b', 'c'), ('b', 'd'), ('c', 'd')]
```

The six tests that make up the covering array cover all the possible interactions
between the values of each of these parameter combinations. For example, the `('a', 'b')`
parameter combination covers all possible combinations of the values that
parameters `a` and `b` can take.

Given that parameter `a` can have values `[0, 1]` and parameter `b` can have values `['a', 'b']`, all possible interactions are the following:

```python
[(0, 'a'), (0, 'b'), (1, 'a'), (1, 'b')]
```

where the first element of each tuple corresponds to the value of the parameter `a`, and the second
element corresponds to the value of the parameter `b`.

Examining the covering array above, we can see that all possible combinations, also referred to as interactions, of parameters `a` and `b` are indeed covered at least once. The same check can be done for other parameter combinations.

# Checking the Covering Array

The `CoveringArray.check()` method can be used to verify that the tests
inside the covering array cover all possible `t-way` interactions at least once and thus
meet the definition of a covering array.

For example,

```python
from testflows.combinatorics import CoveringArray

parameters = {"a": [0, 1], "b": ["a", "b"], "c": [0, 1, 2], "d": ["d0", "d1"]}
tests = CoveringArray(parameters, strength=2)

print(tests.check())
```

This is handy to confirm that the covering array that was calculated is valid.

# Dumping the Covering Array

The `CoveringArray` object implements a custom `__str__` method, and therefore it can be easily converted into
a string representation similar to the format used in the [NIST covering array tables](https://math.nist.gov/coveringarrays/ipof/ipof-results.html).

For example,

```python
   print(CoveringArray(parameters, strength=2))
```

```bash
CoveringArray({'a': [0, 1], 'b': ['a', 'b'], 'c': [0, 1, 2], 'd': ['d0', 'd1']},2)[
6
a b c d
-------
0 b 2 d1
0 a 1 d0
1 b 1 d1
1 a 2 d0
0 b 0 d0
1 a 0 d1
]
```

Dumping covering arrays is useful for debugging and to get a general understanding, and the NIST-like format allows you to share it easily with your colleagues.


# Wrapping Up

Combinatorial testing is an exciting topic, and in this article, I
have introduced you to covering arrays and shown you
how the new [CoveringArray class](/docs/framework/combinatorial-tests.md#Covering-Arrays-Pairwise-N-wise-Testing) allows you to
calculate covering arrays for your tests. We have looked at how you can
generate, check, and dump covering arrays. Covering arrays
are critical for combinatorial testing, and {% testflows %} now brings
them to your test programs without using any third-party libraries.

If you are interested in seeing how you can implement your combinatorial tests
with the TestFlows.com open-source testing framework, check out the [Combinatorial Tests](/docs/framework/combinatorial-tests.md#Combinatorial-Tests)
section in the [Handbook](/docs/framework.md). Until next time,
happy combinatorial testing using covering arrays!

[IPOG]: https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=1362e14b8210a766099a9516491693c0c08bc04a
[IPOG: A General Strategy for T-Way Software Testing]: https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=1362e14b8210a766099a9516491693c0c08bc04a