Our Users And Supporters

who implement their QA process with us

Ultimate ClickHouse experience
Altinity empowers developers to add high-performance analytics to any application regardless of where it runs, how much data it needs to answer questions, or how quickly the data arrives.

Write

Write a simple test.
from testflows.core import Scenario

with Scenario("Hello TestFlows"):
pass

Execute

Then execute it.
$ python3 ./test.py
Nov 20,2019 16:42:00 ⟥ Scenario Hello TestFlows
2ms ⟥⟤ OK Hello TestFlows, /Hello TestFlows

Passing

✔ [ OK ] /Hello TestFlows

1 scenario (1 ok)

Total time 2ms

Executed on Nov 20,2019 16:42
TestFlows.com Open-Source Software Testing Framework v1.3.191112.1234833




Asynchronous Tests

take advantage of asynchronous programming





Using asyncio with async def and async with


1
2
3
4
5
6
7
8
9
10
11
import asyncio
from testflows.core import *

@TestModule
async def module(self):
async with Test("my async test"):
async with Step("my async test step"):
note("Hello from asyncio!")


asyncio.run(module())




Parallel Tests

easily run tests in parallel





Specify parallel and sync with join


1
2
3
4
5
with Feature("my feature"):
Scenario(run=my_test1, parallel=True)
Scenario(run=my_test2, parallel=True)
join()
Scenario(run=my_test3, parallel=True)




Semi-Automated and Manual Tests

seamlessly work with automated, semi-automated, and manual tests





Mix and match different test types in the same test flow


1
2
3
4
5
6
7
8
9
10
11
with Scenario("semi-automated scenario"):
with Given("automated setup"):
pass
with When("manual action", flags=MANUAL):
pass

with Scenario("manual scenario", flags=MANUAL):
with Given("manual setup"):
pass
with When("manual action"):
pass




Testing Documentation

you can easily automate your documentation

and keep it up to date





Write executable documentation by mixing Markdown and Python code using testflows.texts


# Welcome to TestFlows Texts
```python:testflows # any Python code text("Add this string to my Markdown document" "from this Python code block") ```




Professional Reports

right at your fingertips




Requirements Coverage Report

$cat test.log | tfs report coverage requirements.py
Requirements Coverage Report


Requirements Traceability Report

$cat test.log | tfs report traceability
ScreenShot




Results Report

$cat test.log | tfs report results
ScreenShot


Specification Report

$cat test.log | tfs report specification
ScreenShot



Results Comparison Report

$tfs report compare results --log *.log
ScreenShot


Metrics Comparison Report

$tfs report compare metrics --log *.log
ScreenShot




Write and Work with Requirements

in a straightforward and convenient way





Write requirements in Markdown document


1
2
3
4
5
6
7
8
9
10
# SRS001 Software Requirements Specification

## Requirements

### Generic

### RQ.SRS001-CU.LS
version: 1.0

The [ls](#ls) utility SHALL list the contents of a directory.


Convert & use in tests


$cat requirements.md | tfs requirements generate > requirements.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
from requirements import *

@TestScenario
@Requirements(RQ_SRS001_CU_LS_Default_Directory("1.0"))
def list_current_working_directory(self, shell):
"""Check that the `ls` utility, when run without
any arguments, lists the contents of the
current working directory.
"""
with When("I execute `ls` command without arguments"):
r = shell("ls")

with Then("exitcode should be 0"):
assert r.exitcode == 0, error()


Track your test coverage


$cat test.log | tfs report coverage requirements.py
Requirements


Impress and share requirements with your team and customers


$cat requirements.md | tfs document convert > requirements.html
Requirements




Results Analytics

designed to provide analytics

using ClickHouse and Grafana





On-the-fly loading of logs

into ClickHouse


$python3 test.py --database host=localhost
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ clickhouse-client
ClickHouse client version 20.7.1.4130.
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 20.7.1 revision 54437.

user-node :) select test_name from messages WHERE message_keyword = 'RESULT' LIMIT 5

SELECT test_name
FROM messages
WHERE message_keyword = 'RESULT'
LIMIT 5

┌─test_name───────────────────────────────────────────────────────────────────────────────────────┐
│ /ls │
│ /ls/list current working directory │
│ /ls/list current working directory/I execute `ls` command without arguments │
│ /ls/list current working directory/exitcode should be 0 │
│ /ls/list current working directory/it should list the contents of the current working directory │
└─────────────────────────────────────────────────────────────────────────────────────────────────┘

5 rows in set. Elapsed: 0.010 sec.

user-node :)


Dashboards

available in Grafana


Dashboards




More Than Just a Framework

build on top of a messaging protocol

for robustness and advanced applications





Messages

use JSON objects


$cat test.log | tfs transform raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{"message_keyword":"PROTOCOL","message_hash":"6e6fca73","message_object":0,"message_num":0,"message_stream":null,"message_level":1,"message_time":1595181231.201574,"message_rtime":0.000875,"test_type":"Module","test_subtype":null,"test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec","test_name":"/ls","test_flags":0,"test_cflags":0,"test_level":1,"protocol_version":"TFSPv2.1"}
{"message_keyword":"VERSION","message_hash":"2fdb9078","message_object":0,"message_num":1,"message_stream":null,"message_level":1,"message_time":1595181231.201672,"message_rtime":0.000973,"test_type":"Module","test_subtype":null,"test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec","test_name":"/ls","test_flags":0,"test_cflags":0,"test_level":1,"framework_version":"1.6.200716.1214830"}
{"message_keyword":"TEST","message_hash":"b60b2b60","message_object":1,"message_num":2,"message_stream":null,"message_level":1,"message_time":1595181231.201716,"message_rtime":0.001018,"test_type":"Module","test_subtype":null,"test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec","test_name":"/ls","test_flags":0,"test_cflags":0,"test_level":1,"test_uid":null,"test_description":"The `ls` utility regression module.\n "}
{"message_keyword":"REQUIREMENT","message_hash":"cb7abae2","message_object":1,"message_num":3,"message_stream":null,"message_level":2,"message_time":1595181231.201765,"message_rtime":0.001066,"test_type":"Module","test_subtype":null,"test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec","test_name":"/ls","test_flags":0,"test_cflags":0,"test_level":1,"requirement_name":"RQ.SRS001-CU.LS","requirement_version":"1.0","requirement_description":"The [ls] utility SHALL list the contents of a directory.\n","requirement_link":null,"requirement_priority":null,"requirement_type":null,"requirement_group":null,"requirement_uid":null}
{"message_keyword":"TEST","message_hash":"fa644d52","message_object":1,"message_num":0,"message_stream":null,"message_level":2,"message_time":1595181231.204069,"message_rtime":0.00099,"test_type":"Test","test_subtype":"Scenario","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0","test_name":"/ls/list current working directory","test_flags":0,"test_cflags":0,"test_level":2,"test_uid":null,"test_description":"Check that `ls` utility when run without\n any arguments lists the contents of the \n current working directory.\n "}
{"message_keyword":"REQUIREMENT","message_hash":"4d7b28df","message_object":1,"message_num":1,"message_stream":null,"message_level":3,"message_time":1595181231.204191,"message_rtime":0.001112,"test_type":"Test","test_subtype":"Scenario","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0","test_name":"/ls/list current working directory","test_flags":0,"test_cflags":0,"test_level":2,"requirement_name":"RQ.SRS001-CU.LS.Default.Directory","requirement_version":"1.0","requirement_description":"The [ls] utility SHALL by default list information about the contents of the current directory.\n","requirement_link":null,"requirement_priority":null,"requirement_type":null,"requirement_group":null,"requirement_uid":null}
{"message_keyword":"TEST","message_hash":"50a6e014","message_object":1,"message_num":0,"message_stream":null,"message_level":3,"message_time":1595181231.218689,"message_rtime":0.014245,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"test_uid":null,"test_description":null}
{"message_keyword":"NONE","message_hash":"41598c19","message_object":0,"message_num":1,"message_stream":"bash","message_level":4,"message_time":1595181231.259461,"message_rtime":0.055018,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"message":"bash# ls"}
{"message_keyword":"NONE","message_hash":"40f3a41f","message_object":0,"message_num":2,"message_stream":"bash","message_level":4,"message_time":1595181231.260848,"message_rtime":0.056404,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"message":"coverage.html regression.py \u001b[0m\u001b[01;34mrequirements\u001b[0m test1.log test.log"}
{"message_keyword":"NONE","message_hash":"2e45138a","message_object":0,"message_num":3,"message_stream":"bash","message_level":4,"message_time":1595181231.271679,"message_rtime":0.067236,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"message":"bash# echo $?"}
{"message_keyword":"NONE","message_hash":"9ad51777","message_object":0,"message_num":4,"message_stream":"bash","message_level":4,"message_time":1595181231.27186,"message_rtime":0.067416,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"message":"0"}
{"message_keyword":"NONE","message_hash":"b060314c","message_object":0,"message_num":5,"message_stream":"bash","message_level":4,"message_time":1595181231.272425,"message_rtime":0.067981,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"message":"bash#"}
{"message_keyword":"RESULT","message_hash":"b89bbe47","message_object":1,"message_num":6,"message_stream":null,"message_level":3,"message_time":1595181231.272822,"message_rtime":0.068378,"test_type":"Step","test_subtype":"When","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/0","test_name":"/ls/list current working directory/I execute `ls` command without arguments","test_flags":0,"test_cflags":0,"test_level":3,"result_message":null,"result_reason":null,"result_type":"OK","result_test":"/ls/list current working directory/I execute `ls` command without arguments"}
{"message_keyword":"TEST","message_hash":"4b9b249c","message_object":1,"message_num":0,"message_stream":null,"message_level":3,"message_time":1595181231.274689,"message_rtime":0.001416,"test_type":"Step","test_subtype":"Then","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/1","test_name":"/ls/list current working directory/exitcode should be 0","test_flags":0,"test_cflags":0,"test_level":3,"test_uid":null,"test_description":null}
{"message_keyword":"RESULT","message_hash":"a9298fca","message_object":1,"message_num":1,"message_stream":null,"message_level":3,"message_time":1595181231.274933,"message_rtime":0.00166,"test_type":"Step","test_subtype":"Then","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/1","test_name":"/ls/list current working directory/exitcode should be 0","test_flags":0,"test_cflags":0,"test_level":3,"result_message":null,"result_reason":null,"result_type":"OK","result_test":"/ls/list current working directory/exitcode should be 0"}
{"message_keyword":"TEST","message_hash":"b86c0af8","message_object":1,"message_num":0,"message_stream":null,"message_level":3,"message_time":1595181231.276228,"message_rtime":0.000985,"test_type":"Step","test_subtype":"Then","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/2","test_name":"/ls/list current working directory/it should list the contents of the current working directory","test_flags":0,"test_cflags":0,"test_level":3,"test_uid":null,"test_description":null}
{"message_keyword":"RESULT","message_hash":"f890ca0e","message_object":1,"message_num":1,"message_stream":null,"message_level":3,"message_time":1595181231.276464,"message_rtime":0.001221,"test_type":"Step","test_subtype":"Then","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0/2","test_name":"/ls/list current working directory/it should list the contents of the current working directory","test_flags":0,"test_cflags":0,"test_level":3,"result_message":null,"result_reason":null,"result_type":"OK","result_test":"/ls/list current working directory/it should list the contents of the current working directory"}
{"message_keyword":"METRIC","message_hash":"18f22bec","message_object":1,"message_num":2,"message_stream":null,"message_level":3,"message_time":1595181231.276705,"message_rtime":0.073627,"test_type":"Test","test_subtype":"Scenario","test_id":"/cdc771be-c9e8-11ea-be5a-2477034de0ec/0","test_name":"/ls/list current working directory","test_flags":0,"test_cflags":0,"test_level":2,"metric_name":"metric","metric_value":0.4723491269298419,"metric_units":"ms","metric_type":null,"metric_group":null,"metric_uid":null}


Compressed

using LZMA compression


$cat test.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
00000000  fd 37 7a 58 5a 00 00 04  e6 d6 b4 46 02 00 21 01  |.7zXZ......F..!.|
00000010 16 00 00 00 74 2f e5 a3 e0 29 89 05 03 5d 00 3d |....t/...)...].=|
00000020 88 89 a6 94 60 d9 bd bf 72 0c 7c b5 f6 cd 99 04 |....`...r.|.....|
00000030 c8 be 8f 34 f6 48 3f f5 48 cd aa 7d 4b e4 05 14 |...4.H?.H..}K...|
00000040 fb 14 0e 7f ef 43 d8 77 43 bc d1 61 1c 7c a7 e9 |.....C.wC..a.|..|
00000050 12 15 52 86 28 10 be 4b 64 18 ed 7d 39 6d 66 80 |..R.(..Kd..}9mf.|
00000060 ce 5d 8d a3 5b f8 e6 15 78 33 eb d8 39 d4 a7 c7 |.]..[...x3..9...|
00000070 cf 66 70 29 c8 32 15 46 ab 5d 5d 79 01 fb a9 a3 |.fp).2.F.]]y....|
00000080 8d 44 a4 1a 7f fc 75 4b b5 55 0c 37 e1 e2 3e d2 |.D....uK.U.7..>.|
00000090 2d 04 3c 37 e6 7c 48 a7 70 89 8e 32 c7 6c f3 5e |-.<7.|H.p..2.l.^|
000000a0 3b e6 e3 1f d7 f2 15 06 f0 63 c0 bf 0e e5 04 3c |;........c.....<|
000000b0 38 3d b6 f2 f4 5a 2a 5d af e5 a5 b0 04 45 00 cc |8=...Z*].....E..|
000000c0 0b de 48 56 3d 01 91 61 b8 23 3b 68 a5 72 ce bf |..HV=..a.#;h.r..|
000000d0 3b 7a ef 83 dc 7a 58 ae 6d d3 3b 3b f9 e6 6d 7c |;z...zX.m.;;..m||
000000e0 c7 60 d8 17 b1 6a 0e a5 68 a4 3a a1 4f de df 1c |.`...j..h.:.O...|
000000f0 55 4b d2 ba 41 e7 b3 0b cf f9 c8 f2 45 ee 62 88 |UK..A.......E.b.|
00000100 f6 48 e4 0b 65 9b 6c bd 91 5b 12 13 bf 02 23 1c |.H..e.l..[....#.|
00000110 ef 98 d4 18 c1 c2 54 a4 9a e9 f8 b4 3a 37 bf fe |......T.....:7..|
00000120 7d 4a 93 7a d1 27 df a1 1b 26 b4 71 d0 14 37 95 |}J.z.'...&.q..7.|




Actively Developed

become part of the team

and join us on GitHub

or follow us to get the latest updates