Our Users And Supporters

who implement their QA process with us

Ultimate ClickHouse experience
Altinity empowers developers to add high-performance analytics to any application regardless of where it runs, how much data it needs to answer questions, or how quickly the data arrives.

Write

Write a simple test.
from testflows.core import Scenario

with Scenario("Hello TestFlows"):
    pass

Execute

Then execute it.
$ python3 ./test.py
Nov 20,2019 16:42:00   ⟥  Scenario Hello TestFlows
                 2ms   ⟥⟤ OK Hello TestFlows, /Hello TestFlows

Passing

✔ [ OK ] /Hello TestFlows

1 scenario (1 ok)

Total time 2ms

Executed on Nov 20,2019 16:42
TestFlows.com Open-Source Software Testing Framework v1.3.191112.1234833



Asynchronous Tests

take advantage of asynchronous programming




Using asyncio with async def and async with


1
2
3
4
5
6
7
8
9
10
11
import asyncio
from testflows.core import *

@TestModule
async def module(self):
    async with Test("my async test"):
        async with Step("my async test step"):
            note("Hello from asyncio!")

        
asyncio.run(module())



Parallel Tests

easily run tests in parallel




Specify parallel and sync with join


1
2
3
4
5
with Feature("my feature"):
    Scenario(run=my_test1, parallel=True)
    Scenario(run=my_test2, parallel=True)
    join()
    Scenario(run=my_test3, parallel=True)



Semi-Automated and Manual Tests

seamlessly work with automated, semi-automated, and manual tests




Mix and match different test types in the same test flow


1
2
3
4
5
6
7
8
9
10
11
with Scenario("semi-automated scenario"):
    with Given("automated setup"):
        pass
    with When("manual action", flags=MANUAL):
        pass

with Scenario("manual scenario", flags=MANUAL):
    with Given("manual setup"):
        pass
    with When("manual action"):
        pass



Testing Documentation

you can easily automate your documentation

and keep it up to date




Write executable documentation by mixing Markdown and Python code using testflows.texts


# Welcome to TestFlows Texts

```python:testflows
# any Python code
text("Add this string to my Markdown document"
     "from this Python code block")
```



Professional Reports

right at your fingertips



Requirements Coverage Report

$cat test.log | tfs report coverage requirements.py
Requirements Coverage Report


Requirements Traceability Report

$cat test.log | tfs report traceability
ScreenShot




Results Report

$cat test.log | tfs report results
ScreenShot


Specification Report

$cat test.log | tfs report specification
ScreenShot



Results Comparison Report

$tfs report compare results --log *.log
ScreenShot


Metrics Comparison Report

$tfs report compare metrics --log *.log
ScreenShot



Write and Work with Requirements

in a straightforward and convenient way




Write requirements in Markdown document


1
2
3
4
5
6
7
8
9
10
# SRS001 Software Requirements Specification

## Requirements

### Generic

### RQ.SRS001-CU.LS
version: 1.0

The [ls](#ls) utility SHALL list the contents of a directory.


Convert & use in tests


$cat requirements.md | tfs requirements generate > requirements.py 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
from requirements import *

@TestScenario
@Requirements(RQ_SRS001_CU_LS_Default_Directory("1.0"))
def list_current_working_directory(self, shell):
    """Check that the `ls` utility, when run without
    any arguments, lists the contents of the
    current working directory.
    """
    with When("I execute `ls` command without arguments"):
        r = shell("ls")

    with Then("exitcode should be 0"):
        assert r.exitcode == 0, error()


Track your test coverage


$cat test.log | tfs report coverage requirements.py
Requirements


Impress and share requirements with your team and customers


$cat requirements.md | tfs document convert > requirements.html
Requirements



Results Analytics

designed to provide analytics

using ClickHouse and Grafana




On-the-fly loading of logs

into ClickHouse


$python3 test.py --database host=localhost 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ clickhouse-client
ClickHouse client version 20.7.1.4130.
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 20.7.1 revision 54437.

user-node :) select test_name from messages WHERE message_keyword = 'RESULT' LIMIT 5

SELECT test_name
FROM messages
WHERE message_keyword = 'RESULT'
LIMIT 5

┌─test_name───────────────────────────────────────────────────────────────────────────────────────┐
│ /ls                                                                                             │
│ /ls/list current working directory                                                              │
│ /ls/list current working directory/I execute `ls` command without arguments                     │
│ /ls/list current working directory/exitcode should be 0                                         │
│ /ls/list current working directory/it should list the contents of the current working directory │
└─────────────────────────────────────────────────────────────────────────────────────────────────┘

5 rows in set. Elapsed: 0.010 sec.

user-node :)


Dashboards

available in Grafana


Dashboards



More Than Just a Framework

build on top of a messaging protocol

for robustness and advanced applications




Messages

use JSON objects


$cat test.log | tfs transform raw 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{"message_keyword":"PROTOCOL","message_hash":"6e6fca73","message_object":0,"message_num":0,"message_stream":null,"message_level":1,"message_time":1595181231.201574,"message_rtime":0.000875,"test_type":"Module","test_subtype":null,"test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec","test_name":"/ls","test_flags":0,"test_cflags":0,"test_level":1,"protocol_version":"TFSPv2.1"}
{"message_keyword":"VERSION","message_hash":"2fdb9078","message_object":0,"message_num":1,"message_stream":null,"message_level":1,"message_time":1595181231.201672,"message_rtime":0.000973,"test_type":"Module","test_subtype":null,"test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec","test_name":"/ls","test_flags":0,"test_cflags":0,"test_level":1,"framework_version":"1.6.200716.1214830"}
{"message_keyword":"TEST","message_hash":"b60b2b60","message_object":1,"message_num":2,"message_stream":null,"message_level":1,"message_time":1595181231.201716,"message_rtime":0.001018,"test_type":"Module","test_subtype":null,"test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec","test_name":"/ls","test_flags":0,"test_cflags":0,"test_level":1,"test_uid":null,"test_description":"The `ls` utility regression module.\n    "}
{"message_keyword":"REQUIREMENT","message_hash":"cb7abae2","message_object":1,"message_num":3,"message_stream":null,"message_level":2,"message_time":1595181231.201765,"message_rtime":0.001066,"test_type":"Module","test_subtype":null,"test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec","test_name":"/ls","test_flags":0,"test_cflags":0,"test_level":1,"requirement_name":"RQ.SRS001-CU.LS","requirement_version":"1.0","requirement_description":"The [ls] utility SHALL list the contents of a directory.\n","requirement_link":null,"requirement_priority":null,"requirement_type":null,"requirement_group":null,"requirement_uid":null}
{"message_keyword":"TEST","message_hash":"fa644d52","message_object":1,"message_num":0,"message_stream":null,"message_level":2,"message_time":1595181231.204069,"message_rtime":0.00099,"test_type":"Test","test_subtype":"Scenario","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0","test_name":"/ls/list current working directory","test_flags":0,"test_cflags":0,"test_level":2,"test_uid":null,"test_description":"Check that `ls` utility when run without\n    any arguments lists the contents of the \n    current working directory.\n    "}
{"message_keyword":"REQUIREMENT","message_hash":"4d7b28df","message_object":1,"message_num":1,"message_stream":null,"message_level":3,"message_time":1595181231.204191,"message_rtime":0.001112,"test_type":"Test","test_subtype":"Scenario","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0","test_name":"/ls/list current working directory","test_flags":0,"test_cflags":0,"test_level":2,"requirement_name":"RQ.SRS001-CU.LS.Default.Directory","requirement_version":"1.0","requirement_description":"The [ls] utility SHALL by default list information about the contents of the current directory.\n","requirement_link":null,"requirement_priority":null,"requirement_type":null,"requirement_group":null,"requirement_uid":null}
{"message_keyword":"TEST","message_hash":"50a6e014","message_object":1,"message_num":0,"message_stream":null,"message_level":3,"message_time":1595181231.218689,"message_rtime":0.014245,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"test_uid":null,"test_description":null}
{"message_keyword":"NONE","message_hash":"41598c19","message_object":0,"message_num":1,"message_stream":"bash","message_level":4,"message_time":1595181231.259461,"message_rtime":0.055018,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"message":"bash# ls"}
{"message_keyword":"NONE","message_hash":"40f3a41f","message_object":0,"message_num":2,"message_stream":"bash","message_level":4,"message_time":1595181231.260848,"message_rtime":0.056404,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"message":"coverage.html  regression.py  \u001b[0m\u001b[01;34mrequirements\u001b[0m  test1.log  test.log"}
{"message_keyword":"NONE","message_hash":"2e45138a","message_object":0,"message_num":3,"message_stream":"bash","message_level":4,"message_time":1595181231.271679,"message_rtime":0.067236,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"message":"bash# echo $?"}
{"message_keyword":"NONE","message_hash":"9ad51777","message_object":0,"message_num":4,"message_stream":"bash","message_level":4,"message_time":1595181231.27186,"message_rtime":0.067416,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"message":"0"}
{"message_keyword":"NONE","message_hash":"b060314c","message_object":0,"message_num":5,"message_stream":"bash","message_level":4,"message_time":1595181231.272425,"message_rtime":0.067981,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"message":"bash#"}
{"message_keyword":"RESULT","message_hash":"b89bbe47","message_object":1,"message_num":6,"message_stream":null,"message_level":3,"message_time":1595181231.272822,"message_rtime":0.068378,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"result_message":null,"result_reason":null,"result_type":"OK","result_test":"/ls/list current working directory/I execute `ls` command without arguments"}
{"message_keyword":"TEST","message_hash":"4b9b249c","message_object":1,"message_num":0,"message_stream":null,"message_level":3,"message_time":1595181231.274689,"message_rtime":0.001416,"test_type":"Step","test_subtype":"Then","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/1","test_name":"/ls/list current working directory/exitcode should be 0","test_flags":0,"test_cflags":0,"test_level":3,"test_uid":null,"test_description":null}
{"message_keyword":"RESULT","message_hash":"a9298fca","message_object":1,"message_num":1,"message_stream":null,"message_level":3,"message_time":1595181231.274933,"message_rtime":0.00166,"test_type":"Step","test_subtype":"Then","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/1","test_name":"/ls/list current working directory/exitcode should be 0","test_flags":0,"test_cflags":0,"test_level":3,"result_message":null,"result_reason":null,"result_type":"OK","result_test":"/ls/list current working directory/exitcode should be 0"}
{"message_keyword":"TEST","message_hash":"b86c0af8","message_object":1,"message_num":0,"message_stream":null,"message_level":3,"message_time":1595181231.276228,"message_rtime":0.000985,"test_type":"Step","test_subtype":"Then","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/2","test_name":"/ls/list current working directory/it should list the contents of the current working directory","test_flags":0,"test_cflags":0,"test_level":3,"test_uid":null,"test_description":null}
{"message_keyword":"RESULT","message_hash":"f890ca0e","message_object":1,"message_num":1,"message_stream":null,"message_level":3,"message_time":1595181231.276464,"message_rtime":0.001221,"test_type":"Step","test_subtype":"Then","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/2","test_name":"/ls/list current working directory/it should list the contents of the current working directory","test_flags":0,"test_cflags":0,"test_level":3,"result_message":null,"result_reason":null,"result_type":"OK","result_test":"/ls/list current working directory/it should list the contents of the current working directory"}
{"message_keyword":"METRIC","message_hash":"18f22bec","message_object":1,"message_num":2,"message_stream":null,"message_level":3,"message_time":1595181231.276705,"message_rtime":0.073627,"test_type":"Test","test_subtype":"Scenario","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0","test_name":"/ls/list current working directory","test_flags":0,"test_cflags":0,"test_level":2,"metric_name":"metric","metric_value":0.4723491269298419,"metric_units":"ms","metric_type":null,"metric_group":null,"metric_uid":null}


Compressed

using LZMA compression


$cat test.log 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
00000000  fd 37 7a 58 5a 00 00 04  e6 d6 b4 46 02 00 21 01  |.7zXZ......F..!.|
00000010  16 00 00 00 74 2f e5 a3  e0 29 89 05 03 5d 00 3d  |....t/...)...].=|
00000020  88 89 a6 94 60 d9 bd bf  72 0c 7c b5 f6 cd 99 04  |....`...r.|.....|
00000030  c8 be 8f 34 f6 48 3f f5  48 cd aa 7d 4b e4 05 14  |...4.H?.H..}K...|
00000040  fb 14 0e 7f ef 43 d8 77  43 bc d1 61 1c 7c a7 e9  |.....C.wC..a.|..|
00000050  12 15 52 86 28 10 be 4b  64 18 ed 7d 39 6d 66 80  |..R.(..Kd..}9mf.|
00000060  ce 5d 8d a3 5b f8 e6 15  78 33 eb d8 39 d4 a7 c7  |.]..[...x3..9...|
00000070  cf 66 70 29 c8 32 15 46  ab 5d 5d 79 01 fb a9 a3  |.fp).2.F.]]y....|
00000080  8d 44 a4 1a 7f fc 75 4b  b5 55 0c 37 e1 e2 3e d2  |.D....uK.U.7..>.|
00000090  2d 04 3c 37 e6 7c 48 a7  70 89 8e 32 c7 6c f3 5e  |-.<7.|H.p..2.l.^|
000000a0  3b e6 e3 1f d7 f2 15 06  f0 63 c0 bf 0e e5 04 3c  |;........c.....<|
000000b0  38 3d b6 f2 f4 5a 2a 5d  af e5 a5 b0 04 45 00 cc  |8=...Z*].....E..|
000000c0  0b de 48 56 3d 01 91 61  b8 23 3b 68 a5 72 ce bf  |..HV=..a.#;h.r..|
000000d0  3b 7a ef 83 dc 7a 58 ae  6d d3 3b 3b f9 e6 6d 7c  |;z...zX.m.;;..m||
000000e0  c7 60 d8 17 b1 6a 0e a5  68 a4 3a a1 4f de df 1c  |.`...j..h.:.O...|
000000f0  55 4b d2 ba 41 e7 b3 0b  cf f9 c8 f2 45 ee 62 88  |UK..A.......E.b.|
00000100  f6 48 e4 0b 65 9b 6c bd  91 5b 12 13 bf 02 23 1c  |.H..e.l..[....#.|
00000110  ef 98 d4 18 c1 c2 54 a4  9a e9 f8 b4 3a 37 bf fe  |......T.....:7..|
00000120  7d 4a 93 7a d1 27 df a1  1b 26 b4 71 d0 14 37 95  |}J.z.'...&.q..7.|



Actively Developed

become part of the team

and join us on GitHub

or follow us to get the latest updates