What is it?

TestFlows Machine is a deterministic execution environment where you run, explore, debug and test your software. It is built on our deterministic hypervisor, which runs your programs, Docker images and Docker Compose environments unchanged.

Nothing in the machine is left to chance. It controls time, interrupts, random numbers and input from devices, and it records all of them while your program runs. Replay the recording and the program does exactly what it did the first time, instruction for instruction.

That gives you three things to work with:

  • Record. Every run keeps a log of everything that could have turned out differently.
  • Replay. Play the log back and the run repeats itself. If it failed once, it fails every time.
  • Branch. Stop at any point, branch from it, and try something else. The original run stays as it was.

If you have ever chased a test that fails one run in fifty, you know why this is handy. Record the failing run once, then replay it as often as you need.

You work with Machine through a client called machine, or from Python using the SDK. To begin, install the client. It is one command.

Questions, feature requests and bug reports are welcome in the TestFlows Machine repository on GitHub. See Getting help.

Concepts

There are seven of them, and the rest builds on them.

Disk. What a machine boots. It is your program (or a whole application) packaged with a small Linux system. You build one with machine disks build.

Session. The vCPUs and RAM you allocate, which one or more runs execute in. It is what you are billed for, from the moment you create it until you delete it, so delete it when you are done.

Run. A deterministic machine booting a disk inside a session. Every run has a name. Nearly every action is something you do to a run.

Entry. A run writes its history as a log. Each item in the log is an entry. Entry #42 is always the same point in that run.

Checkpoint. A saved copy of the whole machine at one point in the log. Every run has checkpoint 1, taken at its first entry, and you can add your own and name them.

Branch. A run that starts from another run at some point. It shares history with its parent up to that point and has its own after that. You create one with machine fork or machine go. Runs that grow out of one another form a tree.

Mark. A second name for a branch, given after the fact. The generated name says where a branch started. A mark says what it turned out to be, like crashed.

Points

A lot of commands take a point, which is a place in a run’s history. You can name one in several ways.

Point Means
^booted the checkpoint named booted
#42 entry 42
1.5s a time on the machine’s clock
-10ms ten milliseconds of machine time back from where the run is now
now where the run is now
parent the point this run branched from
root the start of the tree
%crashed the branch with the crashed mark
console~login where the console first printed a match of login

The same points work everywhere a point is taken: --at on fork, detach and go, rewind ... to, and replay --to. --from takes them too, except now and a mark. A bare word is read as a checkpoint’s name first, then as a time, then as an entry, so write ^name or #42 when you mean one of them for sure.

A branch can only start from a checkpoint. machine go and machine fork take any of these points. If the one you name isn’t a checkpoint, they start the branch at the nearest checkpoint before it and replay the rest of the way for you. fork --no-start refuses such a point, because it leaves no machine to replay on, and so does detach. A machine time such as 1.5s or -10ms names the first entry at the last machine time at or before the one you asked for.

Getting started

The client

You work with Machine through a client called machine. Install it with one command:

1
curl https://testflows.com/machine/install -fsS | bash

The client is given to a TestFlows account, so the installer first signs you in, or creates your account, at the terminal: you give your email and the code sent to it. An account that has not accepted the current Terms of Service and Client EULA is shown both and accepts them before anything is downloaded. The installer signs out when it ends, so sign in with machine login afterwards. Without a terminal, as in CI, the installer takes an API key from TESTFLOWS_MACHINE_API_KEY instead, from an account that has accepted them.

The installer then downloads the client over HTTPS, checks it against the SHA-256 checksum published next to it, and puts machine and machine-env in ~/.local/bin. If the checksum doesn’t match, nothing is installed. Where another program named machine is already on your PATH, as /usr/bin/machine is on a Mac, machine goes to ~/.local/share/machine/bin instead, which is not on your PATH, and ~/.local/bin holds machine-env alone. If ~/.local/bin isn’t on your PATH yet, the installer says so and shows the line to add for your shell. The installer also puts the SDK Core, which the Python SDK runs on, in ~/.testflows/machine/sdk, checked the same way. Run the same command again whenever you want to update.

Supported systems. The client runs on Linux, x86_64 and arm64, and on Macs with Apple Silicon. It needs only the OpenSSH client, which macOS and most Linux systems already have, and the installer never asks for root. If ssh is missing, install it with your package manager, for example sudo apt install openssh-client on Debian and Ubuntu. It doesn’t run on Intel Macs or natively on Windows; on Windows you can use it inside WSL. On an ARM machine, a Mac with Apple Silicon or arm64 Linux, the disks you build must still hold x86_64 programs; see building disks on an ARM machine. If you need another platform, contact us and tell us which one. The download page has the full list, and a way to check the download yourself.

Behind a proxy that inspects TLS. The client trusts only Let’s Encrypt, which issues the certificate of the Machine API, so a certificate from any other CA is refused, including a proxy’s. The error names TESTFLOWS_MACHINE_API_CA_FILE, an environment variable the client reads: set it to a PEM file of the CAs to trust instead, such as your proxy’s.

1
export TESTFLOWS_MACHINE_API_CA_FILE=/etc/ssl/certs/corporate-proxy-ca.pem

The file replaces the Let’s Encrypt roots rather than adding to them. If the client must also connect without the proxy, add the Let’s Encrypt roots to the same file. The variable applies to the connection to the API alone: sessions are reached over SSH with certificates TestFlows issues, and image pulls use your system’s CAs. A file that is missing or holds no certificate is refused, and the error names the file. The Python SDK reads the same variable.

On a Mac, use machine-env. macOS ships its own /usr/bin/machine, which prints the processor type, so typing machine may not run the client. Putting the client first on your PATH would hide the system command for every other program. machine-env, which the installer writes on every system, starts a shell in which machine is the client and changes nothing else. Its prompt starts with (machine), your usual shell setup is loaded, and exit leaves. It supports zsh and bash.

1
2
3
4
5
$ machine-env
(machine) $ machine login
(machine) $ machine --version
(machine) $ exit
$

The (machine) at the start of the prompt says you are in it: there machine is this client. exit leaves it, back to the shell you were in. To run one command there without starting a shell:

1
machine-env -- machine --version

That form runs one command in that environment and starts no shell, which suits scripts and CI. To see which machine you get, run which machine: /usr/bin/machine is the system’s, and the client is ~/.local/share/machine/bin/machine on a Mac or ~/.local/bin/machine on Linux. If machine --version prints a processor type instead of a version, you are running the system command. On Linux, if another command on your system is also named machine, use machine-env the same way.

To read the installer first, install a specific version or into another folder, check the download by hand, or uninstall, see the download page.

Run machine with no arguments, or with --help, to see every command. Each command has its own --help too, and that is the place to look when a flag here is not enough.

1
2
machine --help
machine fork --help

machine --version prints the client’s version and its license. Include the version when you report a problem.

1
machine --version
1
2
3
4
5
6
7
8
9
10
11
12
13
  ---- o o o ----
| o o |
| 1 o 10010 o 0 |
| o o | TestFlows Machine Client YYYYMMDD-HHMM
--- o o oxx -- https://testflows.com
/ xx \
/ ^^^ xx \
------------------

Copyright (C) 2026 Katteli Inc. All rights reserved.

Licensed under the Client EULA:
https://testflows.com/machine/legal/client-eula/

If something does not work, machine ping checks that you can reach the service and your sessions, and how long the round trip takes.

In machine-env, Tab already completes commands, options and their values, and the names of your runs. Where machine is the client in every shell, as on most Linux systems, add the line for your shell to its startup file to have the same there. On a Mac, use machine-env instead: outside it, machine is the system’s command, and the line would run that one.

1
2
3
eval "$(machine completion bash)"     # ~/.bashrc
eval "$(machine completion zsh)" # ~/.zshrc
machine completion fish | source # ~/.config/fish/config.fish

The client answers each Tab itself, from its own commands, so a newer client needs no newer line.

To try it in one terminal first, run the same line there instead. Tab completes in that shell until it exits, and no other shell changes.

Signing up

The installer creates your account if you have none. You can also create it on the sign-up page, or from the terminal with machine account signup. Then sign in from the terminal.

1
machine login

Machine emails you a one-time code and asks for it. You stay signed in until you sign out with machine logout. machine account show tells you who you are signed in as.

For scripts and CI there is no need to sign in at all. Create an API key and put it in the TESTFLOWS_MACHINE_API_KEY environment variable.

1
machine account api-keys create ci --expiry 90

When you create your first session, Machine sets up your account’s storage first, so that session takes longer to create. To do that step on its own beforehand, run machine account provision.

Your first run

If you have not yet signed up and signed in, do that first: see Signing up.

This example creates a session, builds a disk, runs a program, creates a checkpoint and starts a branch from it. The program is one line of Python, and you need nothing installed but the client: no Docker and no compiler.

First, create a session. Machine makes it the one your commands use.

1
machine sessions create first --cpus 2
1
2
✔ Session first is running.
✔ Now using session first in this terminal.

Next, save this as hello.py.

1
print("hello, world")

Build a disk named hello that runs it. --from names the public Python image, --add puts your file in it, and what follows -- is the command the machine runs.

1
machine disks build --from python:3.12-slim --add hello.py hello -- python hello.py
1
2
3
4
5
6
7
➤ Packing hello (1 service)
➤ Hashing hello.pack
➤ Waiting for the upload to start…
➤ Uploading 20.5kB → hello
➤ Building the disk…
✔ Built hello
Boot machine create --disk hello

Only your file is uploaded. The build pulls the image itself and writes it onto the disk, which takes about two minutes for an image of this size. You build a disk once and create as many runs from it as you like.

A program you compile, an image of your own, or several services together build the same way from other sources. See Disks.

Create a run from the disk. The machine waits for your commands. With --no-daemon it would boot and run to the end by itself.

1
machine create hello-run --disk hello
1
✔ Started hello-run (whUxdR33Zad9Qxc0ASfig)

Now the run is waiting. It only moves when you drive it, and machine run is how you drive it. The --until tasks part means “keep going until Linux is up.”

1
machine run hello-run --until tasks
1
2
3
4
iterations: 15646  run ic: 271676660  vtime: +0.692741050s

VCPU EXIT ID RUN IC RCB TOTAL IC REGS HASH RIP RCX ITER
0 HYPERCALL 15791 271676660 192153 271676660 0x785930775efa707 0xffffffff81f9eeda 43778048 15645

Read the last five lines the machine printed. --plain prints the lines alone. Without it, a terminal also shows the entries, the source and the number of each line.

1
machine output hello-run -n -5 --plain
1
2
3
4
5
futex hash table entries: 256 (order: 2, 16384 bytes, linear)
NET: Registered PF_NETLINK/PF_ROUTE protocol family
thermal_sys: Registered thermal governor 'step_wise'
thermal_sys: Registered thermal governor 'user_space'
cpuidle: using governor ladder

To see the full console log, use -n :. Ranges use the same colon as a Python slice. Lines are numbered from 1 and both ends count, so -n 1: is the first line through the last.

A negative number counts from the end, and -1 is the last line. -n -5 and -n -5: are the last five lines, and -n -10:-2 is the tenth line from the end through the second. Both ends of a range count from the same end.

Create a checkpoint of the machine’s state so you can come back to it, and start a branch from there.

1
machine checkpoint hello-run booted
1
2
✔ Created checkpoint booted at entry 15792
Make it durable machine sync hello-run

The checkpoint is in your session. machine sync publishes the run, so the checkpoint outlasts the session and another session can start from it. Stopping the machine publishes it too.

1
machine fork hello-run --at ^booted --name try-1
1
✔ Forked hello-run/try-1 (whUxfLYSFJMDHzKaiBOvH)

The branch starts exactly where booted was taken. Notice its full name, hello-run/try-1. A branch is named after the run its tree starts from, a slash, and the name you gave it. Look at the tree.

1
machine branches hello-run
1
2
● hello-run (whUxdR33Zad9Qxc0ASfig) [0, ∞)  ← this run
╰─ @15792 0.692741050s+1 → ● hello-run/try-1 (whUxfLYSFJMDHzKaiBOvH) [15793, ∞)

Linux is up on the branch and your program has not started yet. Drive the branch until the machine shuts down, which it does when the program exits.

1
machine run hello-run/try-1 --until halted
1
2
3
4
iterations: 17148  run ic: 2095789032  checkpoints: 1

VCPU EXIT ID RUN IC RCB TOTAL IC REGS HASH RIP RCX ITER
0 IO (EXIT_SHUTDOWN) 40050 2095789032 39 2367459400 0x62064a27645eaf14 0xffffffff8103f4fc 0 17147

The program’s own lines start with app-1. Find them in the branch’s console.

1
machine output hello-run/try-1 -n : | grep app-1
1
2
app-1  | hello, world
app-1 exited with code 0

The parent hello-run is still at booted, with the program yet to run. You can fork it again and get the same start every time.

When you are done, delete the runs and then the session. A run keeps its log and checkpoints until you delete it, and a session costs money until you do.

1
2
machine delete hello-run --recursive --stop --yes
machine sessions delete first --yes

That’s the whole loop. You built a disk, created a checkpoint in a run, branched from it and ran a program to its end. Everything else in this page is a variation on it.

Getting help

TestFlows Machine is in public beta, and reports from users directly influence what gets fixed and built next. Report bugs, request features and ask questions in the TestFlows Machine repository on GitHub.

To report a bug, search the issues first, then open an issue with the commands you ran, what you expected, what happened, and the output of machine --version. To request a feature, open a feature request that describes what you’re trying to do. To ask a question or share an idea, start a discussion in Discussions. You can also send feedback through the contact form.

✋ Issues and discussions are public. Never include an API key, a token, a password or other secrets in them.

For questions about your account or billing, email support@testflows.com. For privacy-related questions, email privacy@testflows.com. Report abuse to abuse@testflows.com. Report security vulnerabilities to security@testflows.com, or through GitHub’s private vulnerability reporting.

Disks

A disk is what a machine boots, and machine disks build makes one. You give it exactly one source.

Source What it takes
--binary path a static x86_64 executable, wrapped in a scratch image
--from ref a public image, by name, with the files you add to it
--image ref a Docker image of your own, as a save tar or a name in your local Docker
--compose dir a Compose project directory and the images its services name

Everything that runs in a machine runs under Docker Compose, inside the machine. A Compose project is one container for each of its services. An image or a binary is a project of one service, and --entrypoint and the arguments after -- are that service’s entrypoint and command. A binary is no exception: --binary wraps the executable in a scratch image, one layer holding that file and nothing else, with no base image, no loader and no libraries, and the machine runs that image as its one service. machine writes this image itself, so --binary needs no Docker on your computer.

Docker and Compose

An image of your own needs Docker. A Compose project also needs Docker Compose. Two sources need neither. machine disks build --binary wraps a static executable and nothing else, so a program you compile needs no Docker. The executable has to be self-contained. The image has no loader and no libraries, so a dynamically linked program cannot run. --from names a public image, and the build pulls it for you.

1
2
machine disks build --image myapp:latest app
machine disks build --compose ./deploy stack

The name defaults to whatever the source is called. Use --entrypoint to choose the executable the service runs. The service’s own arguments go after a --. This one runs data-race with the argument 3:

1
machine disks build --binary ./data-race race -- 3

From Python, client.disks.build takes the same sources as keywords: binary=, from_=, image= or compose=, with command= for the arguments after --. The SDK’s Disks has the rest.

1
disk = client.disks.build("race", binary="./data-race", command=["3"])

Programs that need a runtime

A program that is not a static executable needs an image that already contains what it runs: an interpreter, a runtime, its libraries. Public images have those, so name one with --from and put your files in it with --add. It reads like a Dockerfile, FROM then ADD then the command, and you need no Docker for it. The build pulls the image itself, so only your files are uploaded.

An image you built yourself, or one from a private registry, goes in with --image from your local Docker.

A complex application, several services together, is a Compose project.

Python program

Save this as hello.py.

1
print("hello, world")
1
machine disks build --from python:3.12-slim --add hello.py hello-py -- python hello.py

It runs in a machine with the default 256MB of memory, and takes about ten seconds from power-on to the program’s exit.

Node.js program

Save this as hello.js.

1
console.log("hello, world")
1
machine disks build --from node:22 --add hello.js hello-node -- node hello.js

ClickHouse queries

The ClickHouse® image has clickhouse local, which runs SQL with no server. Save this as queries.sql.

1
2
3
CREATE TABLE orders (id UInt32, customer String, amount UInt32) ENGINE = Memory;
INSERT INTO orders VALUES (1, 'ada', 30), (2, 'grace', 45), (3, 'ada', 25);
SELECT customer, sum(amount) AS total FROM orders GROUP BY customer ORDER BY customer;
1
2
machine disks build --from clickhouse/clickhouse-server:24.8 --add queries.sql \
--entrypoint clickhouse ch-sql -- local --queries-file queries.sql

The machine’s console shows the result, and the machine ends when the queries do.

1
2
3
app-1  | ada	55
app-1 | grace 45
app-1 exited with code 0

PostgreSQL queries

The PostgreSQL image starts a server, and it needs a password before it will. --env sets a variable in the service’s environment. The image runs every .sql file in /docker-entrypoint-initdb.d when the server first starts, so add the queries there. Save them as queries.sql.

1
2
3
CREATE TABLE orders (id int, customer text, amount numeric);
INSERT INTO orders VALUES (1, 'ada', 30), (2, 'grace', 45), (3, 'ada', 25);
SELECT customer, sum(amount) AS total FROM orders GROUP BY customer ORDER BY customer;
1
2
machine disks build --from postgres:17 --env POSTGRES_PASSWORD=secret \
--add queries.sql:/docker-entrypoint-initdb.d/ pg
1
2
3
4
5
6
7
8
9
app-1  | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/queries.sql
app-1 | CREATE TABLE
app-1 | INSERT 0 3
app-1 | customer | total
app-1 | ----------+-------
app-1 | ada | 55
app-1 | grace | 45
app-1 | (2 rows)
app-1 | 2026-03-01 00:00:50.435 UTC [1] LOG: database system is ready to accept connections

The server keeps running after the queries, so this machine does not end by itself. To run queries and have the machine end, use the PostgreSQL project. The disk holds each --env value as written, and a value is not changed on the way: a $ in it arrives as a $.

Both ran in a machine with 1024MB of memory. A server that other services connect to is a Compose project.

Where added files go

--add puts a file or a directory in the image, as one more layer on top of the public one. You can repeat it.

You write It lands at
--add hello.py the image’s working directory, as hello.py
--add conf the working directory, as the directory conf
--add hello.py:/app/ /app/hello.py
--add hello.py:/app/main.py /app/main.py
--add conf:/etc/app/ /etc/app/conf, the directory inside /etc/app
--add conf:/etc/app /etc/app, the directory’s contents

The rule is the one cp -r follows: a destination that ends in / is a directory the path goes into, and any other destination is what the path becomes. A Dockerfile ADD of a directory copies its contents; the last row is how you write that here.

A script that was executable still is. --add works with --binary too, for a program that reads a file beside it.

--add takes files, directories and links. A device, a FIFO or a socket in the directory is refused. Every added file belongs to root in the image, so the set-user-id and set-group-id bits are not kept: a bit that meant “run as this file’s owner” on your computer would mean “run as root” there. An image that needs one is built with Docker and named with --image.

The image is the one the name pointed at when you ran the build. The build records its digest and pulls exactly that, so a tag that moves later does not change your disk. You can name a digest yourself, as python@sha256:… or the python:3.12@sha256:… that docker pull prints.

The disk’s size defaults to what the image needs plus a gigabyte. Use --dry-run to see the numbers, and --size for more room.

--from takes public images of up to 5GB as they download. For a private image, a larger one, or one you build yourself, use --image:

1
2
docker build --platform linux/amd64 -t myapp:latest .
machine disks build --image myapp:latest app

An image that needs more than one service, such as a database server with a client, is a Compose project.

Compose projects

The examples have Compose environments under compose/. Clone them, pull each image for linux/amd64 into the local Docker, then build the disk from the project directory. machine disks build --compose packs those images and the project. The machine runs the project and ends it when a service exits on its own, stopping the others, or when the project is taken down with docker compose down. A service that Machine stops, kills or crashes with machine containers ends nothing.

1
2
git clone https://github.com/testflows/Machine.git
cd Machine/examples

The same project can be run on this computer first, from its own directory, which is how to fix it without waiting for a boot. The build commands below are from the examples directory.

A two-service project

compose/two-services is a busybox server and a client that fetches a file from it by the service name server. Services in a project reach each other by name, on the network Compose gives the project.

depends_on only orders the start. The client retries, because nothing waits for the server to accept connections. A service that takes time to become ready should use a healthcheck, and depends_on with condition: service_healthy.

1
2
docker pull --platform linux/amd64 busybox:1.36
machine disks build --compose compose/two-services two

To run it on this computer first:

1
2
cd compose/two-services
docker compose up --abort-on-container-exit

ClickHouse project

compose/clickhouse is a ClickHouse server and a client that queries it. Both services use one image, so the disk carries that image once. The client waits until the server is healthy, prints its result, and exits, which ends the project.

The server takes time to start before it accepts a query, so the project uses a healthcheck rather than a sleep. config.xml in the project replaces the image’s config. It listens on IPv4 only, because the machine’s kernel has no IPv6, and it caps the background pools, because the image’s defaults hang startup in a small machine. CLICKHOUSE_SKIP_USER_SETUP stops the entrypoint from restricting the default user to loopback, which would refuse the client.

The config is mounted from ./config.xml, a path inside the project. A bind mount from outside the project directory is refused. A named volume or a tmpfs is created empty when the machine boots.

1
2
docker pull --platform linux/amd64 clickhouse/clickhouse-server:24.8.14.39-alpine
machine disks build --compose compose/clickhouse ch

PostgreSQL project

compose/postgres is a PostgreSQL server and a client that runs queries.sql against it. Both services use one image. The client prints the results and exits, which ends the project, so this is how to run queries against PostgreSQL and have the machine end.

The client waits until the server is healthy. The healthcheck asks over TCP, because the image first starts a temporary server on its Unix socket alone to set the database up, and a check on the socket would pass before the real server is listening. ON_ERROR_STOP makes a failed query the client’s exit code. queries.sql is mounted from the project directory.

1
2
docker pull --platform linux/amd64 postgres:17
machine disks build --compose compose/postgres pg
1
2
3
4
5
6
7
8
client-1    | CREATE TABLE
client-1 | INSERT 0 3
client-1 | customer | total
client-1 | ----------+-------
client-1 | ada | 55
client-1 | grace | 45
client-1 | (2 rows)
client-1 exited with code 0

Carrying images

A service can start containers of its own, through the machine’s Docker. The images it starts are named by no service this project runs, and a machine reaches no registry, so an image that is not named here is not on the disk.

scale: 0 names an image and starts nothing. The build packs every image the project names, including these, and Compose starts none of them. compose/clickhouse-regression carries the images its suite starts this way. Another image, such as ZooKeeper or MinIO, is another service.

1
2
3
4
5
6
7
8
9
10
services:
regression:
image: regression-runner:local
working_dir: /opt/compose
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./:/opt/compose
carried_server:
image: clickhouse/clickhouse-server:24.8.14.39-alpine
scale: 0

The socket is /var/run/docker.sock. /run/docker.sock is not the machine’s socket, and the build refuses it.

./ is mounted at /opt/compose, the path the machine runs the project from. Compose resolves a bind on the machine’s filesystem, not inside the container that asked for it, so the two paths have to be the same. The images still have to be in the local Docker, under those names, before the build.

Project rules

Every service names an image with a tag or a digest. An image with no tag is a different image tomorrow, and a service that only has build has nothing to pack. Build the image, then set image.

The images have to be linux/amd64 and already in the local Docker. A logging driver, if the project sets one, is json-file, local, or none.

Volumes a crash cuts

machine containers crash cuts a service’s data the way a power cut would. It cuts the volumes the project names in an x-machine block, each with the most it may hold:

1
2
3
4
5
6
7
8
9
services:
db:
image: postgres:16
volumes: [db-data:/var/lib/postgresql/data]
volumes:
db-data: {}
x-machine:
volumes:
db-data: 4G

Each becomes an ext4 filesystem of its own on the disk, which costs only what is written to it. The disk is made big enough for every volume to fill, and --dry-run shows the room they take:

1
2
3
4
size
used 811MB
volumes 4096MB
capacity 6144MB

The capacity is what the disk holds, plus the volumes full, plus a gigabyte, rounded up to a whole GB.

A volume in x-machine is a plain Compose volume: no driver_opts, not external. A project names 8 at most. A volume two services mount can’t be crashed while both run. Such a volume has a journal, which is what recovers it after a crash, and an fsync on it takes about four times as long as on a volume the block doesn’t name. Name only the volumes whose crashes you want to test.

Building disks on an ARM machine

A machine is an x86_64 computer, so a disk holds linux/amd64 programs, whatever machine you build it on. On a Mac with Apple Silicon or on arm64 Linux, Docker builds and pulls arm64 images by default, and those cannot run in a machine. machine disks build checks every image and refuses one that is not linux/amd64 before uploading anything. Ask Docker for linux/amd64:

1
2
3
docker pull --platform linux/amd64 myapp:latest
docker build --platform linux/amd64 -t myapp:latest .
DOCKER_DEFAULT_PLATFORM=linux/amd64 docker compose -f ./deploy/compose.yaml build

The last one builds a Compose project’s images for linux/amd64; setting platform: linux/amd64 on each service does the same. Docker builds them under emulation, which is slower than a native build but works. For --binary, compile for x86_64 Linux, for example GOOS=linux GOARCH=amd64 go build.

Images at boot

A disk boots with its images already loaded, so the project starts as soon as Docker does.

A disk built with --load-at-boot keeps the images as docker save tars instead, and Docker loads them each time the disk boots.

1
machine disks build --load-at-boot --image myapp:latest app

That boot is slower by the whole load, every time, and the disk holds each image twice: the tar, and what it unpacks to. Use it when a service runs docker save or docker push on one of the disk’s own images from inside the machine. Starting containers does not need it.

Layers compressed with zstd are not read; save the image with gzip or uncompressed layers.

Disk size

By default Machine measures what the disk holds and adds a gigabyte, rounded up to a whole GB. Use --size to set the size you want. Not sure how big it will be? --dry-run reports what the disk would hold without building it.

1
machine disks build --dry-run --from python:3.12-slim --add hello.py hello -- python hello.py
1
2
3
4
5
6
7
8
9
disk
name hello
content
rootfs 285MB
images 142MB
project 1MB
size
used 428MB
capacity 2048MB

rootfs is the platform every disk carries. images is the images unpacked, with a layer two images share counted once. used is what a smaller --size is refused against, and the rest of capacity is what the machine can write. With --load-at-boot there is one more row, images (tar).

To manage your disks:

1
2
3
4
machine disks list
machine disks show app
machine disks rename app app-v2
machine disks delete app-v2

machine disks list and machine disks show give two sizes. SIZE is the disk as the machine sees it. DATA is what the disk takes in your account’s storage, after deduplication and compression.

1
2
NAME   ID            SIZE   DATA   STATE  CREATED  LABELS
hello 4d13d6d911ee 2.1GB 148MB ready 1h ago —

Disks share what they have in common: every disk carries the same platform, and two disks built from one image share its layers. Storage keeps one copy, and each disk’s DATA counts it, so the column adds up to more than your disks take together. machine storage has that figure, and it is the one your quota counts. A disk that shows — for DATA gets a figure from machine disks verify.

If you ever doubt the copy of a disk in your account’s storage, check it:

1
machine disks verify app
1
✔ Verified app (4d13d6d911ee)

machine disks verify reads the disk back and computes its ID again. It takes about as long as reading the disk, and a disk that does not read as it was stored is reported as damaged, in which case build it again.

Deleting a disk is permanent. Machine refuses to delete one that a run still boots. --force deletes it anyway. Those runs stay, and can no longer start. machine disks transfers shows the transfers in progress, and machine disks cancel stops one.

Sessions

Runs live in a session, and you create one with machine sessions create. Pick its size when you create it.

1
machine sessions create ci-1 --cpus 2 --mem 4G

--cpus sets the number of vCPUs, --mem the memory and --storage the disk space. Your plan caps each session: 2 vCPUs, 4G of memory and 32G of disk on Free, 4 vCPUs, 8G and 128G on Starter and on Free with usage credits, 6 vCPUs, 12G and 192G on Plus, and 8 vCPUs, 16G and 256G on Pro. A hardware class can give a session less than the plan allows, and the smaller of the two applies. Your plan also limits the vCPUs your sessions hold together: 4 on Free, 8 on Starter and on Free with usage credits, 12 on Plus and 24 on Pro. A session counts the vCPUs it is charged for: 2 vCPUs with 8G of memory count 4. machine account show shows how many are in use. --class picks the hardware class, and the default comes from your plan. machine sessions classes lists the classes, with the region and the CPU architecture of each. A session runs in its class’s region, and its runs are stored there. standard-ip, on intel-1, is the class to prefer. standard-ie, on intel-2, costs less. A run starts only in a session of the architecture it was recorded on.

1
2
machine sessions classes
machine sessions create arm-1 --class standard-ie

The name is optional. Leave it off and Machine makes one up. Machine waits until the session is running and tells you when it is. Add --no-wait to return once the request is accepted.

Then Machine uses the new session in this terminal, so your next command goes to it. Add --no-use to keep using the session you are on. A session created with --no-wait is not running yet, so it is not used.

Your first session also sets up your account’s storage, so it takes longer to create, and Machine shows this as a step of its own. To set the storage up before that, run machine account provision.

A machine takes its memory plus 128MB of the session’s memory, and 256MB of a session is not available to machines. A machine with the default 256MB needs 384MB, and one created with --mem 2048 needs 2176MB, so a 4G session holds one and an 8G session holds three. A machine holds its memory until you stop it, whether or not its program has finished.

A branch you switch onto with machine go uses the machine that is already running, so it needs no more memory. A branch you start with machine fork is another machine and needs its own.

machine runs --size adds two columns for what a run takes of the session’s resources. RAM is the session memory its machine holds right now. It can be above the machine’s own MEMORY. DATA is what the run holds on the session’s disk. A run with no machine shows — for RAM.

The MEM column of machine sessions list shows the memory taken over the session’s memory, for example 4.5G/8G with two machines of 2048MB. A new machine starts when what it needs fits in the rest, and is refused when it does not.

To switch to another session, tell Machine which one to use.

1
machine sessions use ci-1

machine sessions use applies to the terminal you ran it in. If you only have one session, it needs no name at all. After that, every command that needs a machine goes to ci-1. You can send one command somewhere else with -s, for example machine -s ci-2 runs. -s goes before the command.

In a script, name the session once with TESTFLOWS_MACHINE_SESSION, and every command after it goes to that session.

1
2
3
export TESTFLOWS_MACHINE_SESSION=ci-1
machine sessions use ci-1
machine create --disk hello

The variable names the session. machine sessions use connects this computer to it, and is needed once. A tool that starts a new shell for each command, as an AI agent does, loses the export between commands. There, put -s ci-1 on each command. Commands that need no session, such as machine disks list, ignore it.

You manage the rest of your sessions like this:

1
2
3
4
machine sessions list
machine sessions show ci-1
machine sessions rename ci-1 ci-2
machine sessions delete ci-2

✋ A session costs from the moment you create it, whether or not anything is running in it. Delete sessions you are not using.

Idle sessions stop

A session is idle while it serves no command. Your plan sets its idle time: 10 minutes on Free, 30 on Starter and on Free with usage credits, 45 on Plus, and 60 on Pro. machine account show shows yours. When a session reaches its idle time, it stops, and so do its runs. The runs are kept. Start one again in any session with machine start <run>.

Every command sent to the session resets the idle time, from any terminal, the SDK or the control prompt. To step away and keep the session, keep it:

1
machine sessions keep ci-1 30m

A keep holds the session idle for up to 30 minutes on Free, and an hour on Starter, Free with usage credits, Plus and Pro. machine sessions list shows under IDLE how long each session has gone without a command, and machine sessions show when it stops. machine sessions list --all also lists the sessions that stopped. One that stopped idle is listed as stopped idle, with the idle time it reached under IDLE. In the control prompt, /keep 30m keeps the session you are driving, and the line under the clock shows the time left.

From Python, client.sessions does the same: create, use, show, keep and delete. A session is a context manager, so a with block deletes it at the end. The SDK’s Sessions has the calls.

When the service does not answer

A command that cannot reach the service, or that the service refuses as unavailable, ends with a hint to the status page, status.testflows.com. The page shows whether the website, the API, sign-in and sessions on each class work now. It also lists open incidents, the incidents of the last 14 days and any planned maintenance. If every component is operational, check your network. If it works, contact support.

Running a machine

Creating a run

machine create creates a run from a disk and starts it. The run’s name goes first and is optional. Without one, Machine generates a name.

1
machine create app --disk app --cpus 2 --mem 512

The defaults are 1 CPU and 256 MB of memory. Other options of create:

Option What it does
--no-daemon boot and run to the end by itself, instead of waiting for your commands
--pin-cpu n run on host CPU n, for this start only (the default picks the least-loaded)
--like run start from another run’s options, then apply the ones you give
--label name=value attach a label to the run, as many times as you like
--rng, --rng-seed n an RNG device, on by default, seeded with n; the default seed is 0, and --no-rng leaves the device out
--fs-sync write to the disk as the program writes, instead of later
--mem-hash hash the machine’s memory at every entry, so a replay compares it too (off by default)
--disk-hash hash the machine’s disk at every entry, so a replay compares it too (off by default)
--timer-slack d how late a timer may be in code that takes no exit, 20ms by default; 0 keeps every timer on time
--limit-margin n how many conditional branches short of an exact stop the CPU’s counter interrupts; the default is the host’s
--repeatable record so that recording the run again gives the same entries; the default is the session’s, which machine describe shows, and recording is slower
--vtime-base n the date the machine’s clock starts at, as a Unix timestamp; the default is 2026-03-01, see Machine time
--no-wait return as soon as the request is accepted

Random numbers

A machine has an RNG device, on unless the run was created with --no-rng. The device gives the guest its random bytes from a seed. The default seed is 0, so a program sees the same random numbers every time. --rng-seed chooses another, from 0 to 2^64-1.

1
2
machine create app --disk app --rng-seed 42
machine rng app

machine rng prints the seed, and whether the guest reseeds its own generator at once.

To seed the device again, where the run is now:

1
machine rng app seed 7

The run records that seed, so a replay seeds the device at the same point. A guest kernel that reads the seed reseeds at once. A kernel that does not read the seed picks it up the next time that kernel reseeds. A run created with --no-rng has no device, and the command refuses it.

A branch can draw other random numbers than its parent. machine fork, machine detach and machine go take --rng-seed, and the new run’s device is seeded from it once the run is up:

1
machine fork app --at ^booted --name other --rng-seed 7

At the control prompt the same commands are /rng and /rng seed 7, and a branch takes rng-seed 7.

The seed on machine run --seed is a different one. It draws how long each stretch of execution lasts. Schedules covers it.

From Python, client.create("app", disk="app", rng_seed=42) seeds a new run, run.rng() reads the seed, run.rng.seed(7) seeds the device again, and run.fork(at="^booted", rng_seed=7) makes a branch with its own.

Multiple vCPUs

A machine can have more than one vCPU. Ask for them with --cpus and the guest sees every one, so its kernel spreads threads across them.

Build a disk from Ubuntu 26.04 whose container runs sleep infinity, so it stays up and waits for your commands. Then create a run with four vCPUs and ask the guest what it has:

1
2
3
4
machine disks build --from ubuntu:26.04 ubuntu-26.04 --entrypoint /bin/sleep -- infinity
machine create app4 --disk ubuntu-26.04 --cpus 4
machine exec app4 run --container app -- nproc
machine exec app4 run --container app -- grep -E '^processor|^model name' /proc/cpuinfo
1
2
3
4
5
6
7
8
9
4
processor : 0
model name : TestFlows Machine v1
processor : 1
model name : TestFlows Machine v1
processor : 2
model name : TestFlows Machine v1
processor : 3
model name : TestFlows Machine v1

Nothing about recording changes with more vCPUs. The machine records each of them, along with every interrupt one vCPU sends to another, so a run on four vCPUs replays exactly like a run on one. A race between two threads on different vCPUs plays out the same way every time you replay it.

Each scheduling cycle gives every vCPU a turn. To drive just one of them, use machine run --vcpu N. And to explore how threads on several vCPUs could interleave, see Schedules.

Driving it

A run executes only while you drive it. machine run is the way, and --until says when to stop. (A run created with --no-daemon answers ✗ Machine not in control mode, because the machine isn’t waiting for you.)

1
machine run app --until tasks

A plain machine run can stop before the point you are waiting for. With --until, Machine keeps going until what you asked for is true. If it never gets there, the command exits with status 3.

These are the conditions you can wait for. machine run --until and machine wait --for take the same ones.

Condition True when
running, paused, stopped, halted the machine is in that state
ready the machine accepts commands
tasks Linux is up and its task table can be read, which is before it starts anything in userland
console~REGEX the console output matches the pattern
ttyS1~REGEX the output of another source matches it, here the serial port ttyS1
entry>=N the log has reached entry N
entry=+N N more entries have been recorded
vtime>=T the machine’s clock has reached T, a duration such as 500ms
vtime=+T the machine’s clock has moved T from where it was
/app:exit the program at /app has exited

The last row is one example of a task condition. A condition on a program needs the program focused first, with machine focus app add /app, and the command refuses it otherwise. There are more task conditions under Steering programs.

The two commands read a text condition differently. run --until waits for text printed during the run, so a line already on the console does not end it, and the same --until can stop at the next of several lines alike. wait --for asks whether the text has been printed at all, before the run or during it:

1
2
machine run app --until 'console~ready'    # runs until the next ready line
machine wait --for 'console~ready' app # true if one was printed already

A source name with a ~ in it is quoted inside the pattern, as in '"/output/keep/app/app.log~"~ERROR'.

--iters limits how many iterations a drive may spend, and the global --timeout limits how long you are willing to wait. Whichever hits first wins.

An iteration has no fixed length. It is one dispatch of a vCPU, and it ends where the guest next exits to the machine: after five instructions or after a million. So no count is right in advance, and nobody can know how long a program will run. Stop on a condition, and use --iters or a vtime condition as a limit. Every run ends by saying how far it went:

1
iterations: 2000  run ic: 238893097  vtime: +0.383620410s

vtime is how far the machine’s clock moved. A vtime condition is checked between iterations, so the run stops at the end of the iteration that reaches the time, a little late. A run that went too far is not lost: machine rewind and machine go take it back to an earlier point.

A time bound

--within ends a run when the machine’s clock has moved that far:

1
2
machine run app --within 10s                           # run for 10 s of machine time
machine run app --until 'console~done' --within 30s # the first of the two

Alone, it is the same run as --until vtime=+10s. With another --until it is a bound on it: if the clock gets there first, the run ends with exit status 3 and says Machine time ran out. The machine checks the bound itself, so a slow host does not end the run sooner, which the global --timeout in seconds would.

The clock moves as the programs in the machine work and while the machine is idle.

Now and then the host interrupts an entry while it is being recorded. Machine records that entry again, and you see nothing except a short pause. --retry sets how many more times one entry is tried, 2 by default. If they are all interrupted the run stops with Host interrupted the run at entry N. It is not recorded. and the machine rewind command that puts the run back at the entry before.

Schedules

A program that uses threads can behave differently depending on how they interleave. Machine can draw how long each stretch of execution lasts, so every run explores a different interleaving, and --seed makes those draws repeatable.

1
2
machine focus app add /usr/bin/app
machine run app --until /usr/bin/app:exit --mode step,fast --step 1:200 --seed 1

--mode is how the machine runs: fast, step, step-into, free or fence, or a comma list to draw from. --step and --fast give the length of a stretch in each mode, as a number or a range like 1:200. --strategy sets how the draws are made, random (the default), golden or sweep, and --seed is the seed for the random and golden ones.

A fence stretch ends in the same place every time the same state is recorded, where a fast stretch ends close to its length but not always at the same instruction. That exactness makes a fence stretch somewhat slower than a fast one. Give its length with --fence or --limit. Recording that repeats shows how to make a whole run record the same way every time.

What a task does

Drawing the length of a stretch varies where threads are interrupted. To vary what the guest’s scheduler does with them, draw a mark for the task on the vCPU at each dispatch:

1
2
3
machine run app --task-sched keep,hold,yield --task-slice 10us:600us --seed 3 --iters 500
machine focus app add /usr/bin/app
machine run app --task-int none,preempt --when /usr/bin/app:on --seed 4 --iters 500

--task-sched draws among keep, hold, yield, drain and drain-all: hold takes the task off the run queue, yield puts it behind its peers, drain undoes the oldest of those marks, drain-all undoes every one, and keep changes nothing. --task-slice draws how long the task runs before the scheduler decides again, from a span such as 10us:600us, and default puts the kernel’s own back. --task-int draws whether to preempt it. Each takes a comma list to draw from, and --seed makes the draws repeatable.

--when applies the draws only while a predicate holds, such as /usr/bin/app:on for the program running its own code. A predicate on a program needs the program focused first, as the second example does with machine focus, and the run refuses it otherwise. To draw different things at different moments, write a plan, where each clause has its own when.

Combine that with branches. Fork several from one checkpoint, give each its own seed, and keep the ones that crash. The explore_seeds.py example in the SDK does exactly that.

To hold an interrupt a drawn time before it is delivered, see Holding interrupts.

A run takes checkpoints for you too. --auto-checkpoints N takes one every N entries, and 0 turns them off.

Cancelling a drive

If a drive or a checkpoint is taking too long, machine cancel app cancels whichever one is in progress. A run stops at once, where the host’s clock put the stop, so a repeatable run no longer repeats from that entry. A replay stops before its next entry.

--soft stops the run at the end of the dispatch in progress instead. That stop is an ordinary entry, so the run stays repeatable. It takes at most the run’s dispatch bound, 1s by default.

1
2
machine cancel app           # stop now
machine cancel app --soft # stop at the end of the dispatch in progress

Ctrl-C on a drive cancels softly, and so does the global --timeout on run and exec run. Ctrl-C prints Stopping at the end of the current dispatch. Press Ctrl-C again to stop now, and a second Ctrl-C stops the run at once. A run still busy 10 seconds after a Ctrl-C or a timeout is stopped at once as well. At the control prompt, /cancel stops the run at once and /cancel soft at the end of its dispatch.

Recording that repeats

A replay always repeats a recording exactly. Recording the same run a second time is different: by default two recordings of one run, from the same disk and the same seeds, start out the same and then drift apart a little.

When you need the second recording to match the first, create the run with --repeatable. Every recording of it then records the same entries, whether the run has one vCPU or several, and a branch you fork from one of its checkpoints records exactly what its parent recorded after that checkpoint.

1
2
3
4
5
6
machine create app --disk app --repeatable --cpus 2
machine run app --until tasks
machine checkpoint app booted
machine run app --until 'console~done'
machine fork app --at ^booted --name again # records the same entries again
machine fork app --at ^booted --no-repeatable # a branch that records as usual

Repeatable recording is slower, and how much slower depends on the program. For example, booting PostgreSQL 17 on one vCPU takes 1 minute 10 seconds to record with --repeatable and 14 seconds without it. Loading its test data with pgbench -i, which computes with hardly a pause, takes 1 minute 20 seconds against 4 seconds. A program that spends its time waiting on the disk, the network or a timer pays much less. Replaying a repeatable run is as fast as replaying any other.

--timer-slack is how late a timer may be while a program runs code that takes no exit. The default is 20ms, and it applies to every fast run. A timer that comes due in code that exits to the machine is delivered at that exit. 0 keeps every timer on time, and recording is slowest that way. The slack takes ns, us, ms or s, as in --timer-slack 1s. A drive can draw the slack for that drive:

1
machine run app --timer-slack 0:20ms

--limit-margin is how many conditional branches short of an exact stop the CPU’s counter interrupts. The default is the host’s, and machine describe shows that default as host's. A smaller margin records faster. One too small misses stops, and a later recording may not repeat. machine replay --limit-margin sets it for that replay.

Four more options decide where a recording stops. Each takes 0 for its default, and machine describe shows them as Instr Margin, Dispatch Bound, Fast Backstop and IRQ Wait:

Option What it does
--instr-margin n how many instructions short of a stop placed by an instruction count the CPU’s counter interrupts; the default is the host’s, 50,000 on amd-1, the standard class, and 3,630 on Intel, which describe shows as host's
--dispatch-bound d how much host time a fast dispatch runs before it stops, 1s by default; it takes a duration such as 500ms or 2s
--fast-backstop n how many instructions a fast dispatch runs with no other stop, 100,000,000 by default
--irq-wait n how many conditional branches a dispatch runs while an interrupt waits for the guest to take it, 200 by default

A stop placed by --dispatch-bound falls where host time put it, so another recording of the same state does not stop there. A margin the run gives must be smaller than its backstop, and the run is refused otherwise.

1
machine create app --disk app --repeatable --instr-margin 100000 --dispatch-bound 2s

A branch keeps its parent’s --repeatable, --timer-slack, --limit-margin, --instr-margin, --dispatch-bound, --fast-backstop and --irq-wait unless you give it others. machine fork, machine detach and machine go take all seven. machine start keeps what the run was created with.

Where a run stops repeating

Some stops are placed by host time, and no other recording of the same state stops at the same instruction: a stop at the dispatch bound, an immediate machine cancel and a signal the host sends the machine. A repeatable run repeats up to the first such entry. For a repeatable run, machine describe shows that entry as Repeats To, and tip when the whole run repeats:

1
2
Repeatable:             true
Repeats To: entry 804, a host-time stop

machine checkpoints has a REPEATS TO column for a repeatable run: for each of its own checkpoints, the first entry after it where recording again from that checkpoint stops repeating:

1
2
3
NAME  ID     VTIME              DURABLE  REPEATS TO
1 1 0.000013850s no entry 804, a host-time stop
pre 19502 1.380127790s no entry 19580, a host-time stop

A machine cancel --soft and a Ctrl-C end the run at the end of the dispatch in progress, so they keep the run repeatable. See Cancelling a drive.

What you send a run is part of what a recording repeats. Commands given with machine exec, put and get in the same order at the same points name their processes alike in every recording, such as 40787p1: the entry the run stood at, and the command’s place among those sent there.

Time compression

Idle time takes almost no time.

When every vCPU is halted and waiting for a timer, the machine doesn’t sit through the wait. It moves its clock straight to the earliest timer the guest armed, and the guest wakes up at exactly the time it asked for, as if it had waited. No instruction runs in between, so there is nothing for the host to do. Time the guest spends working is never skipped.

So a program that sleeps for ten minutes sleeps ten minutes of machine time, and only a few seconds of yours. Try it on a one-vCPU run of the Ubuntu disk from Multiple vCPUs:

1
2
machine create app --disk ubuntu-26.04
time machine exec app run --container app -- sh -c 'date; sleep 600; date'
1
2
3
4
Sun Mar  1 00:00:05 UTC 2026
Sun Mar 1 00:10:05 UTC 2026

real 0m2.802s

Ten minutes went by in the guest, and 2.8 seconds went by for you.

Every vCPU idle

Time compression works the same way with several vCPUs. The machine only moves the clock once all of them are idle, so a vCPU that still has work finishes it first. Here is the same sleep on the four-vCPU run:

1
time machine exec app4 run --container app -- sh -c 'date; sleep 600; date'
1
2
3
4
Sun Mar  1 00:00:07 UTC 2026
Sun Mar 1 00:10:07 UTC 2026

real 0m7.166s

On replay

Every move of the clock is recorded in the run’s log, so a replay makes the same moves at the same points and ends up in the same state. Fork a branch from the start of the run and replay it, ten minutes of sleep included:

1
2
3
machine stop app4
machine fork app4 --at ^1 --name check
machine replay app4/check

Then machine diff compares the branch with the original:

1
machine diff app4/check
1
✔ No differences

How much a run compresses depends on how idle the guest really is. A database that keeps running background tasks while your test sleeps uses machine time for them, and that time is never skipped.

Turning time compression off

--no-idle-skip keeps the clock moving at its own rate while every vCPU is idle, instead of jumping to the next timer. Use it to see what a program does while it waits in real time, at the cost of the speed:

1
machine run app --no-idle-skip --within 10s

Waiting

machine wait doesn’t drive anything. It watches a run, or several, and returns when a condition holds.

1
2
machine wait --for stopped app
machine wait --for 'console~ready' app
1
✔ Reached stopped: app (whUxfLYSFJMDHzKaiBOvH) 2.1s

It exits with 0 when the condition is true, 2 on an error and 3 on a timeout, so scripts can tell them apart. Remember that it only watches. Waiting for a condition on a run nobody is driving never finishes unless you set --timeout. With --timeout 0 it checks once, and the message tells you why it failed: The machine was waiting for commands.

At the control prompt the two are /run until and /wait for, and /run takes within too:

1
2
/run until 'console~ready' within 30s
/wait for 'console~ready'

/wait for answers at once, because nothing drives the machine while the prompt waits. timeout 5 makes it wait 5 seconds, for a run another terminal drives.

From Python, run.run(until="console~ready", within="30s") drives until a condition, and run.wait(for_="stopped", timeout=60) waits for one without driving. Bounding a drive shows both.

Inspecting a run

Command Shows
machine runs your runs, newest first (-w keeps the list live)
machine describe app the full configuration of a run
machine state app its state and where it stands; a run with no machine reads stopped
machine now app the last entry it recorded, and the time on the machine’s clock
machine output app what the machine printed (-f follows, -n -20 shows the last 20 lines, --source reads another source than the console)
machine entries app the entries in its log
machine ops operations on runs and how each ended, newest first: checkpoint, branch, switch, rewind, start and delete; one you interrupted with Ctrl-C is still listed
machine debug log app the debug output of a run (machine debug err shows its error and crash log)
machine dump app the machine’s registers, code and other state (--show picks the sections, --entry reads an earlier one); its hash section is what the newest entry recorded
machine hash app the hashes of the machine as it stands, to compare with another stopped at the same point; see Comparing two machines
machine vcpu app where one vCPU stopped, and how its last dispatch ended (--vcpu picks another)
machine irqs app the interrupts the machine owes or holds; see Holding interrupts

machine output also takes --entries, if you only want what a few entries printed.

The run’s log is a list of entries, one for each time the machine stopped, with why it stopped and where. machine entries reads it, and -f follows new entries as the run records them:

1
2
3
machine entries app -n -20      # the last 20 entries
machine entries app -n 100:120 # entries 100 to 120
machine entries app -f

With many runs, machine runs --roots shows one row for each tree, with how many of its runs are running, and --not-synced shows the runs nothing has been published from yet.

Finding a line

Often you want the opposite: you know what the machine printed, and you want the entry that printed it, so you can go there. A line is printed over a range of entries, but a byte is printed by exactly one, and Machine can tell you which.

machine output app search finds a pattern and gives the entry of each match:

1
machine output app search 'Linux version'

Each match is a row with the columns ENTRY, TIME, SOURCE, LINE:COL and TEXT. ENTRY is the entry that printed the first byte of the match, and TIME is that entry’s time, a point such as 1.194000000s+2. TEXT is the whole line, with the match in bold.

A match has to cover at least one character, because it is the entry that printed it that you get back. A pattern that can match nothing at all, such as 6*, is refused, in a search and in a console~REGEX condition or point alike; 6+ is the one to write.

The search runs where the console is kept, so a large console is not downloaded. -f keeps going with what the machine prints next, and --max 1 stops at the first match. The entry is a point, so you can go straight to it:

1
machine go app '#1276'

console~REGEX is the same thing written as a point, where the console first printed a match. machine go, machine fork --at and machine rewind all take it:

1
machine go app 'console~Linux version'

console:N is a line by its number, as machine output -n numbers it: the point where the console finished printing line N. Any source takes it, such as ttyS1:20. A line counts from 1, so the point names the same place each time it is read:

1
machine go app console:120

If you would rather search with your own tools, pipe the console to them. grep -b prints the byte offset of a match, and machine output app entry turns an offset into its entry:

1
2
machine output app -n : | grep -bo 'Linux version'    # 438:Linux version
machine output app entry 438 # 1276

On a terminal, every line shows the range of entries that printed it, the source it came from and its number. Into a pipe or a file the lines are printed alone, as the machine printed them. --wide and --plain choose either way:

1
2
machine output app -n -5                      # with their entries
machine output app -n : --plain > boot.log # the text alone

On a terminal, a control character the guest printed shows as text, such as ␛[?25l, and your terminal does not act on it. Colors stay colors. A pipe or a file gets every byte as the guest printed it.

From Python, run.output.find returns the first match, with the entry that printed it and a place that go takes, and run.output.search yields the matches one at a time:

1
2
3
4
5
hit = run.output.find("Linux version", from_="^booted")
branch = run.go(hit.place)

for match in run.output.search("ERROR", from_="^booted", to="^done"):
print(match.entry_id, match.text)

Sources

The console is not the only place a machine prints. It has four serial ports, ttyS0 to ttyS3, and the console is the first. A program in the guest that writes to /dev/ttyS1 prints on ttyS1, apart from the boot log and from everything else on the console:

1
2
# inside the guest
my-app > /dev/ttyS1 2> /dev/ttyS2

Everything a run prints is kept in one log, in the order it was printed, and each part of it has a source. machine output app sources lists the sources that printed:

1
machine output app sources
1
2
3
4
SOURCE      BYTES      LINES  ENTRIES
ttyS0 10767 198 29:21127
ttyS1 116 10 16916:16949
ttyS2 99 10 16917:16948

--source reads one of them. Without it you get the console, as before:

1
2
3
machine output app --source ttyS1
machine output app search ready --source ttyS1
machine output app entry 0 --source ttyS1

--source all reads every source together, in the order printed, with the source in front of each line:

1
machine output app --source all -n -4
1
2
3
4
ttyS0     starting the app
ttyS1 listening on 8080
ttyS2 warning: no config file
ttyS1 ready

A line belongs to one source. If one source prints while another has a line unfinished, that line ends there, so what you read is always what one source printed, in order.

The name of a source also takes the place of console in a condition and in a point. A run that waits for text on ttyS1 stops at the entry that printed it, and what the other sources print in the meantime does not interrupt it:

1
2
machine run app --until 'ttyS1~ready'
machine go app 'ttyS1~ready'
Source What it is
console the console of the run, which is the default
ttyS0 to ttyS3 the four serial ports; ttyS0 is the console
hvc0 the virtio console
debugcon the firmware’s debug port
a path under /output an output file, or every file under a directory
all every source, in the order printed

Output files

A service on a disk has no serial port to write to, and three ports are not many. Every service is given a directory instead, and each file it writes there is a source of its own. The directory is named by a variable:

1
2
# inside the service
my-app --events "$MACHINE_OUTPUT_DIR/events.jsonl"

The file is stored with the run as it is written, each write with its entry. Its name as a source is its path: /output/keep, the name of the service, and the path the service wrote. A disk built from a binary or one image has one service, app.

1
machine output app sources
1
2
3
SOURCE                            BYTES      LINES  ENTRIES
ttyS0 10767 198 29:21127
/output/keep/app/events.jsonl 2210 31 16916:20410

Everything that takes a source takes the path, and the path of a directory is every file under it:

1
2
3
machine output app --source /output/keep/app/events.jsonl
machine output app search '"setup_complete"' --source /output/keep/app
machine run app --until '/output/keep/app/events.jsonl~setup_complete'

A file does not have to be text. machine output app bytes writes what a source printed exactly as it was written:

1
machine output app bytes --source /output/keep/app/core.bin > core.bin

Dropped output

Debug logs can be gigabytes, and a run that stores them is slow to copy and to read. A service writes those under a second directory, and what is written there is not stored:

1
2
# inside the service
my-app --debug-log "$MACHINE_DROP_OUTPUT_DIR/debug.log"

The file is a source under /output/drop. It still has its name, and machine output app sources lists it with 0 bytes, so you can see what a run dropped. A run can still wait for a line of it. The wait stops the run at the entry that wrote the line and stores nothing:

1
machine run app --until '/output/drop/app/debug.log~connection refused'

Because a run replays exactly, the file is not lost. Fork a branch that keeps the file and starts from a point before the part you care about:

1
2
machine fork app --from ^before --name look --keep-output /output/drop/app/debug.log
machine output look --source /output/drop/app/debug.log

--from replays the branch from that point to where the run is now, and the branch stores the file on the way. Use --at to stop the branch at an earlier point. machine go takes --from too, and moves the machine onto the branch. machine detach takes it for a new root that holds the file and shares nothing with the run it came from.

--keep-output and --drop-output say what a run stores besides what the two directories say. They are options of create, fork, detach and go, the commands that make a run, and a run keeps them for its whole life. machine describe shows the two directories and the names the run gives.

  • A name is a file or a directory under /output, a port such as ttyS1, or a pattern such as '/output/drop/*/*.log'. A pattern is in the syntax of Python’s glob module: *, ?, [abc], [a-z] and [!abc] stay within one directory name, and ** matches any number of directories, as in '/output/**/*.log'. A name that begins with a dot is matched like any other.
  • When a kept and a dropped name cover the same file, a name spelled out decides over a pattern, and of two names spelled out the longer one decides: --keep-output /output/drop/app --drop-output /output/drop/app/huge.log keeps the directory except that file. Otherwise the file is stored.
  • A branch has its parent’s two lists. An option on the fork replaces the list of its own kind and leaves the other as it was. The name none, given alone, is an empty list: --keep-output none keeps nothing beyond what the directories say.

What is stored never changes what the machine executes, so a branch that keeps more still reproduces the recording.

Sending input

A program that reads what a person types can be typed to. machine send writes input to a device of the run:

1
2
3
4
5
machine send app ls                      # type ls and Enter on the console
machine send app ls --device ttyS1 # the same on another serial port
machine send -n app y # no Enter after it
machine send -n -e app '\x03' # Ctrl-C
machine send app - < script.sh # a file, exactly as it is

It works like echo: the text is sent as written and a newline follows, which is Enter. -n leaves the newline out, and -e reads escapes such as \t and \x03.

send does not run the machine. It queues the input and returns, and the device takes it the next time the run executes:

1
2
machine send app 'echo hello'
machine run app --until 'console~hello'

The run records the input where it arrived, so a replay reproduces it without being sent anything. To try other input, fork a branch and send to the branch.

At the control prompt the same command is /send. The options come first and the text is the rest of the line, as you typed it, quotes included:

1
2
3
4
/send ls -la
/send --device ttyS1 echo hello
/send echo "a;b"
/send -n -e \x03

The prompt reads quotes as a shell does. A ; separates commands, except inside quotes, so /send echo "a;b" sends echo "a;b". A quote that is not closed refuses the whole line, so write /send "don't". Put the text in quotes to keep spaces at its ends, as in /send " y ", and put -- before a text that begins with a dash.

Device Is
console the run’s console, which is the default
ttyS0 to ttyS3 the serial ports
hvc0 the virtio console, on a run created with --console virtio
keyboard the PS/2 keyboard

A serial port and the virtio console take bytes. The keyboard takes text and types it on a US keyboard, a key pressed and released for each character, with Shift or Ctrl held where the character needs one. Use it for a program that reads the screen’s terminal and not a serial line.

Commands in the guest

machine exec executes a command in the machine, beside the programs already running there:

1
2
3
machine exec app run -- ls -la /             # execute it and print its output
machine exec app run -- sh -c 'ls | wc -l' # a shell line, on a disk with sh
machine exec app start -- /app/load --rows 9 # send it and return its id

run runs the machine until the command exits, prints what it printed, and exits with its status. A command gets 30 minutes of machine time to end, and --within gives it another: machine exec app run --within 10s -- make check. start sends the command and returns an id. The process starts the next time the run executes, so follow it with machine run.

A command is a program and its arguments, written after --. No shell reads it. Use -c to give a line to the shell of the disk instead, with its pipes and redirections: machine exec app run -c 'ls / | wc -l'.

Processes

Each command is a process. The run records every line the process prints and how it ends, each at its entry, so a replay reproduces them and a branch has the processes of its parent.

machine exec app with no verb lists every command executed in the run, with the entries it started and ended at:

1
2
3
ID       PID  STATE     ENTRIES      COMMAND
23035p1 138 exited 0 23084:23090 ls
23090p1 139 exited 0 23220:23226 echo foo > /tmp/foo.txt

A process’s id is the entry it was sent at and its number. proc shows one process:

1
2
3
4
5
machine exec app proc 23090p1          # pid, state, exit status
machine exec app proc 23090p1 events # every line and state change, by entry
machine exec app proc 23090p1 stdout # its standard output
machine exec app proc 23090p1 stderr # its standard error
machine exec app proc 23090p1 exits # how it ended, and at which entry

An event’s entry is a point. machine go app '#23224' forks a branch at the entry that printed a line.

The output of a process is a source like any other, under /output/exec/<id>. machine output reads it, and machine run app --until '/output/exec/23090p1/stdout~done' stops the run at a line the process prints. In the machine, its standard output and standard error are the files /.machine/output/exec/23090p1/stdout and /.machine/output/exec/23090p1/stderr, and its events are in /.machine/output/exec/23090p1/status.

Inside a container

Use --container to run a command in a container of your Compose project. It takes the container’s id, its name or the name of its service:

1
2
3
4
machine exec app run --container db -- psql -c 'select 1'
machine exec app run --container db -c 'ls / | wc -l' # the container's shell
machine exec app run --container db --user root -- id # as another user
machine exec app start --container app -- ./worker

--net runs a program of the machine on a container’s network instead: the program comes from the machine’s disk, so it needs nothing in the container’s image, and it sees the container’s interfaces and addresses. A capture of db’s packets with the machine’s tcpdump, in Network faults:

1
machine exec app start --net db -- tcpdump --immediate-mode -U -i eth0 -w /.machine/output/keep/db.pcap

machine containers lists the containers with the columns SERVICE, CONTAINER, ID, STATE, IMAGE, ADDRESS and LINK: each container’s address on each of its networks, and the machine’s interface at the other end of its link, which a capture names in Network faults. The service, the container name and the id are what --container takes:

1
machine containers app

Each service’s output, every line it prints across every start of its container, is a source of its own. The console shows all of them together, each line after its service’s name:

1
machine output app --source /output/keep/db/console

The command has the container’s environment and working directory. --user picks the user it runs as, by name or as uid:gid; without it the command runs as the container’s own user. A service that runs several containers is refused, and the message names them, so you pick one. machine put and machine get take the same --container.

A command without --container runs in the machine, beside Docker, and can call Docker itself. The project is named compose:

1
2
3
4
machine exec app run -- docker ps --format '{{.Names}}'                 # the containers
machine exec app run -- docker compose -p compose exec -T db hostname # by service
machine exec app run -- docker exec compose-db-1 hostname # by container
machine exec app run -- docker compose -p compose exec -T db sh -c 'ls / | wc -l'

docker compose exec needs -T, since the command has no terminal. A shell line after the service is run by the image’s own shell, with the image’s tools.

/exec at the control prompt

At the control prompt the command is /exec, with the same verbs and flags. It needs no --: the arguments are the words after the verb, and a line for the shell is the rest of the line after -c, its quotes kept for the shell.

1
2
3
4
5
6
7
/exec run ls -la /
/exec run -c ls / | wc -l
/exec run -c 'X=1;echo $X'
/exec run --container db -- psql -c 'select 1'
/exec start -c gdb -p 36 < /dev/gdb_pipe
/exec proc 23090p1 stdout
/exec

A ; separates commands at the prompt, except inside quotes, so a line with one is written in quotes, as in a terminal. A pair of quotes around the whole -c line is taken off. A $ line, below, needs none of this.

The machine’s shell: $

$ puts you in the machine’s shell. Type a line after it as you would in a terminal. The prompt runs the machine until the line finishes and prints what it wrote. A line that exits other than 0 says so after its output.

Nothing in the line is the prompt’s. Everything after the $ goes to the shell as you typed it, so pipes, redirections, ;, &&, variables and quotes all work without escaping:

1
2
3
4
5
6
7
$ uname -a
$ ls -la /srv
$ ps aux | grep worker
$ cd /srv; make test
$ X=1; echo $X
$ grep -c ERROR /var/log/app.log || echo none
$ echo 'retries=5' >> /etc/app.conf

A $ line is /exec run -c with that line, so the run records it like any other command. /exec lists it, a checkpoint keeps what it changed, and a fork or a replay repeats it. Look around between two steps of a run, change a file, and rewind to a checkpoint to try something else.

A container’s shell: db$

Put a container before the $ to run the line in that container’s shell. Name the container by its Compose service or by its container name, as --container takes it. Put a user and @ before the container to run the line as that user, by name or by uid. Without a user the line runs as the container’s own user.

1
2
3
4
5
6
7
db$ psql -c 'select 1'                          # in the db service
compose-db-1$ hostname # by container name
root@db$ id # as root
70@db$ ls -l /var/lib/postgresql/data # as uid 70
app$ ps aux | grep clickhouse # pipes are the shell's
app$ cd /var/log/clickhouse-server; tail -5 clickhouse-server.log
clickhouse@app$ clickhouse-client -q 'SELECT version()'

db$ LINE is /exec run --container db -c with that line, and root@db$ LINE adds --user root. Everything after the $ goes to the container’s shell as you typed it, and the run records it as it records a $ line.

Commands over several lines

A line goes on in the next one while a quote is open, after a trailing backslash, and through a here-document, as in a shell. The prompt shows > for each line after the first, and runs the whole command once it is complete. A pasted query of several lines is one command:

1
2
3
4
5
6
7
8
9
clickhouse@app$ clickhouse-client -q '
> SELECT name, value
> FROM system.build_options
> LIMIT 3'
$ cat <<EOF > /etc/app.conf
> retries=5
> EOF
$ tar czf /tmp/logs.tgz \
> /var/log/app

The lines of a command are edited together: Backspace at the start of a line goes on into the line above, and the arrows move between the lines. Ctrl+C drops the command.

Input for a program you started

A program that reads commands from its standard input, such as a debugger, is driven through a named pipe. Make the pipe, keep it open, start the program with its input on the pipe, and send to the pipe:

1
2
3
4
5
6
7
machine exec app run   -c 'mkfifo /dev/gdb_pipe'
machine exec app start -c 'sleep 1000000 > /dev/gdb_pipe' # keeps the pipe open
machine exec app start -c 'gdb -p 36 < /dev/gdb_pipe'
machine send app 'b main' --device /dev/gdb_pipe
machine send app c --device /dev/gdb_pipe
machine run app --until vtime=+3s
machine exec app proc 40787p1 stdout # what gdb printed

A --device that begins with / is the path of a file in the machine, and machine send puts the bytes at its end. A process in the machine writes them, and send prints its id: machine exec app proc with that id says whether they were written. One send to a path is one request of at most 4096 bytes. A send too large for it is refused, and the message gives the most bytes that fit.

machine send writes the machine’s own files. For a program in a container, make the pipe in the container and write it from there:

1
2
3
4
5
machine exec app run   --container db -- mkfifo /tmp/p
machine exec app start --container db -c 'sleep infinity > /tmp/p'
machine exec app start --container db -c 'gdb -p $(pidof postgres) < /tmp/p'
machine exec app start --container db -c 'echo c > /tmp/p'
machine run app --until '/output/exec/<id>/stdout~Continuing' --within 10s

<id> is the id exec start printed for GDB.

GDB inside the machine walks through a whole session this way, on ClickHouse.

machine exec uses the serial port ttyS3 for its requests. A program in the machine that reads ttyS3 takes them.

From Python, run.exec(["cat", "/etc/hostname"]) runs a program and returns a Process with exit_code, stdout and stderr. A string is a shell line, container="db" and user="root" pick where it runs, and wait=False returns at once with the process’s id:

1
2
proc = run.exec("ls /data", container="app")
print(proc.exit_code, proc.stdout)

Copying files

machine put copies a file of your machine into the running machine, or into one of its containers:

1
2
3
4
5
machine put app ./app.conf /etc/app.conf                    # into the machine
machine put app ./tool.sh /usr/local/bin/tool --mode 755 # with a mode
machine put app ./seed.sql /srv/seed.sql --container compose-db-1
machine put app ./d.bin /data/d.bin --owner 1000:1000 # with an owner
machine put app - /tmp/in < data.bin # standard input
  • A file of any size is put.
  • The directories on the way to the path are made, and a file already there is replaced.
  • The file has its name only once all of it is written, so a program never reads part of it.
  • The machine runs while the file goes in, and the other programs in it run too.
  • The bytes are a recorded input, so a replay puts the same file and a branch has it.
  • The file is on the disk when put returns. A checkpoint taken after it has the file, and machine fs reads it there.

--container takes a container’s id or name, as machine containers app lists them, or the name of its Compose service: --container db is the container of the service db. A service that runs several containers is refused, and the message names them, so you pick one. The container must be running. A path is resolved inside the container, as a program there would resolve it.

machine send --device <path> is for a line into a pipe, one request of at most 4096 bytes. put is for a file.

machine get copies a file the other way, from the machine or from a container to your machine:

1
2
3
machine get app /var/log/app.log ./app.log
machine get app /srv/dump.sql ./dump.sql --container compose-db-1
machine get app /etc/hostname - # standard output

get reads the file as it is now, with what programs wrote and the disk does not have yet. machine fs reads the disk at a checkpoint instead. The machine runs while the file comes out, so a file a program is writing can change on the way.

At the control prompt the commands are /put and /get, with the same operands and options: /put ./app.conf /etc/app.conf.

A machine whose disk was built before put used this path still takes a file, far slower. Rebuild the disk to get the speed.

From Python, run.put("./app.conf", "/etc/app.conf") puts a file in, and source can be bytes. run.get("/var/log/app.log", "./app.log") gets one out, and without a destination the Got it returns carries the bytes in data. Both take container=.

Core dumps

machine core tells a process to generate a core dump and copies it to your computer, where a debugger reads it with the program:

1
2
3
machine core app 412 ./app.core                    # pid 412 of the machine
machine core app 7 ./db.core --container db # pid 7 of a container
gdb ./app ./app.core
  • Taking a core ends the process: it dies of a signal, ABRT unless you name another with --signal.
  • To keep your run as it is, fork first and take the core on the branch.
  • With --container, the pid is the one the process has inside the container, as machine exec app run --container db -- ps shows it.
  • A process that handles the signal dumps no core. Name another, such as --signal SEGV.
  • The first process of a container dumps no core. Run your program under an init, init: true in the Compose file, so it is not the first.

From Python, run.core(412, "./app.core") gets the core of process 412, and container="db" names the process by its pid in that container.

Profiling

machine profile shows where a run spent its time, by function and by call stack:

1
2
3
machine profile app                            # a second of everything
machine profile app --for 5s --container db # one container
machine profile app --pid 412 # one process

Here is one second of memstress, one of the example programs, which maps, touches and unmaps memory from four threads:

1
2
3
4
5
6
7
8
463 samples at 250 a second
SELF TOTAL FUNCTION
34.1% 34.1% kernel_[k]
34.1% 34.1% sys_munmap_[k]
19.2% 94.4% worker
5.4% 5.4% sys_madvise_[k]
5.2% 5.2% sys_mmap_[k]
1.1% 1.1% sys_clone3_[k]

SELF is the share of samples a function was running in, and TOTAL the share it was on the stack of. So worker itself ran in 19% of the samples and was on the stack of 94% of them: the rest of its time went to the calls under it.

A name that ends in _[k] is time in the kernel. sys_munmap_[k] is the munmap system call. kernel_[k] is kernel time that is no system call, here the page faults of touching fresh memory.

  • --flame draws a flame graph in the terminal.

  • --folded prints every stack with its count, one a line. That text is what flame graph tools read, so you can save it and draw it with the tool you like, or hand it to an agent:

    1
    2
    3
    4
    5
    memstress;clone3;start_thread;worker;kernel_[k] 56
    memstress;clone3;start_thread;worker;munmap;sys_munmap_[k] 32
    memstress;clone3;start_thread;worker 31
    memstress;clone3;start_thread;worker;mmap64;sys_mmap_[k] 12
    memstress;clone3;start_thread;worker;madvise;sys_madvise_[k] 10
  • -o json gives the same stacks as data.

Machine takes the samples from outside the guest, at every timer interrupt of each CPU. Nothing is installed in your image and nothing runs in the guest for it, so a run is the same with a profile and without. Your programs do not need frame pointers: a stack is walked by the unwind information every program carries for exceptions.

A recorded stretch

Because a run is recorded, you can profile a stretch of it after the fact, even if no profiler was on at the time:

1
2
machine profile app --from ^request --to ^reply
machine profile app --from 12s --to 14s --flame

The stretch is replayed on a branch, which is deleted afterwards. Your run is not touched, and it does not need to be running. Profile the same stretch twice and you get the same samples.

Where names come from

  • Names come from the program files on the machine’s disk as it is when the stretch ends. You do not need a checkpoint, and a profile does not take one.
  • A file the guest has written but not synced to disk yet shows as [file]. Run sync in the guest first.
  • A stripped program with no unwind information, such as Alpine’s busybox, is one frame named after its file.
  • Time in the kernel is one frame. The functions of the kernel itself are not shown.
  • --in-guest samples with a small process inside the guest instead. Use it when you need the kernel’s own functions by name. It is part of the run, and it cannot profile a recorded stretch.
  • Samples are taken 250 times a second on each CPU.
  • C++ names are printed as the program has them. Pipe --folded through c++filt to read them.

From Python, run.profile("5s", container="db") returns a Profile with samples, functions, stacks and the folded text, and run.profile(from_="^slow", to="^done") profiles a stretch already recorded.

Interactive mode

When you are working on one run for a while, you can open a prompt for it, so you no longer type machine and the run’s name before every command.

1
machine control app

You issue commands to the run as slash commands, like /checkpoint, /branches or /rewind. Most of the commands on this page have one, written the same way without the run’s name. Commands that run the machine, take checkpoints or make and switch branches take their options as plain words, so --at ^booted becomes at ^booted. Commands that execute something in the guest or read its state, such as /exec, /send, /profile or /regs, take the same flags as on the command line: /exec run --container app nproc. Type /help to see them all, and /q or Ctrl+D to leave.

/h and a command shows that command with examples, such as /h exec. A word in capitals in the help is a type: /h RANGE says how to write a range, and /h POINT a point.

Above the prompt, a line shows where the run is: its time, its entry, the vCPU, the RIP, what the entry is, such as IO_OUT, and its state hash. Back at an entry, the same hash means the same state.

1
⌖ vtime=4.648247180s  entry=32103  vcpu=0  rip=0xffffffff8104880d  IO_OUT  hash=0x91c170d2d6da8e05

The prompt is the easiest way to try something and come back. The session below starts from a new run made from the ubuntu-26.04 disk, the one built in Multiple vCPUs from Ubuntu 26.04 with a container that waits for your commands. Disks has more on building one.

1
2
machine disks build --from ubuntu:26.04 ubuntu-26.04 --entrypoint /bin/sleep -- infinity
machine create app --disk ubuntu-26.04
1
✔ Started app (whyvcWEKxc7c5b2UwO7r6)
1
machine control app

At the prompt, it boots the machine until Docker attaches to its container, keeps that point as a checkpoint, and removes a file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
/f until console~Attaching within 60s
iterations: 24957 run ic: 1525715866 vtime: +3.730915228s checkpoints: 1

VCPU EXIT ID RUN IC RCB TOTAL IC REGS HASH RIP RCX ITER
0 IO 27777 1905719 148 1525715866 0x26180340852cf271 0xffffffff8104880d 1 4

/checkpoint name booted
✔ Created checkpoint booted at entry 27777
Make it durable /sync

/exec run cat /etc/hostname
docker-test

/exec run rm /etc/hostname

/exec run cat /etc/hostname
cat: can't open '/etc/hostname': No such file or directory
✗ Exited 1.

/f runs the machine until Attaching appears on the console, or for 60 seconds of the machine’s own time at most. Here it takes 3.7 seconds of the machine’s time, and the table shows where each vCPU stopped. To see the machine as it was before the file was removed, fork a branch at booted and switch onto it:

1
2
3
4
5
6
7
8
/go to ^booted
✔ Branched app/27777 (whyvg0aXMSf6g7H2pU5wa) at entry 27777 and switched onto it.

/exec run cat /etc/hostname
docker-test

/switch app
✔ Switched to app (whyvcWEKxc7c5b2UwO7r6)

The branch is named after the entry it was forked at, and the file is there again, while app keeps its own history without it. /switch app moves the machine back. The prompt also remembers the branches you have visited: /go back returns to where the last switch started, /go up and /go down walk the visits in order, and /history lists them. /mark good gives the branch you are on a name, and /switch %good returns to it.

A branch made at the prompt takes the same options as one made with machine fork, written as words. This one records a branch that repeats, with a smaller timer slack than the default, as Recording that repeats describes:

1
2
3
/fork at ^booted name exact repeatable timer-slack 5ms
✔ Forked app/exact (whyvhjPaLbY3hsz3KyytQ)
From checkpoint 27777 of app, nothing replayed

/fork starts the branch beside the one you are on and leaves you where you are. /switch app/exact moves onto it, and /describe:app/exact shows Repeatable: true and Timer Slack: 5ms.

-c runs a single slash command and exits, as in machine control app -c '/state', and machine control log app lists the commands that changed the run.

Keys

The prompt edits a line the way your shell does.

Key Does
Tab completes a command, and after /h a command or a type
Up, Down walk the commands typed at this prompt, the last 500; they are not saved when it closes
Ctrl+C cancels the run, checkpoint or hash in progress, and keeps the prompt; a run stops at the end of its dispatch, and a second Ctrl+C stops it at once
Ctrl+D, /q leaves the prompt; the run carries on as it was
; runs the next command after this one, and stops at the first that fails: /s;/d
;+ runs the next command whatever this one did: /replay;+/diff

Quotes work as in a shell, so a ; inside quotes is part of the word: /send "a;b" types a;b. A line with a quote left open continues on the next one, at a > prompt.

Running at the prompt

/run drives the machine as machine run does, and every option of machine run is a word: --until is until, --within is within, --irq-delay is irq-delay. Two shorter forms pick the mode: /f runs fast and /s steps. A suffix picks the vCPU and the number of dispatches: =N drives vCPU N alone, and @N runs N dispatches.

1
2
3
4
5
/f until console~ready within 30s
/s@20
/f=1@10
/run@50 mode fast,step seed 3
/run@200 task-sched keep,hold,yield task-slice 10us:600us seed 7

The first runs fast until ready appears on the console, or for 30 seconds of the machine’s time. /s@20 steps 20 dispatches, and /f=1@10 runs vCPU 1 fast for 10 dispatches. The last two draw something per dispatch from a seed: the mode, then what the focused tasks do and how long each runs, as Schedules describes. /h run lists every word.

Reading another run

A command that reads a run takes :RUN to read another one without switching to it: /describe:app/exact, /checkpoints:app/try-1, /entries:app -20, /lineage:parent, /d:app/try-1 regs. Where you stand stays the same.

Words that differ from the command line

Most slash commands are machine commands with the run’s name left out and the options of commands that act written as words. These differ:

At the prompt On the command line
/go to ^42 machine go app ^42
/go up, /go down, /go back, /history none: the prompt’s own record of the branches you visited
/delete try-1 recursive force machine delete app/try-1 -r -f; the prompt refuses the branch you are on
/mark good, /mark -good, /marks machine marks add, remove, list
/vcpu=1 machine vcpu app --vcpu 1
/d regs machine dump app --show regs
/checkpoint booted machine checkpoint app booted; /checkpoint name booted also works
/auto-checkpoints 10000 machine run app --auto-checkpoints 10000, at the prompt a setting that stays
/now machine now app

create, start, stop, pause, resume, kill, runs, sessions and disks have no slash command. Use them from the command line beside the prompt.

Scripts and history

-c sends one line and exits, so a script can use the prompt’s syntax. A line can chain commands:

1
2
3
4
machine control app -c '/checkpoint name before'
machine control app -c '/f until console~done within 60s;/d regs'
machine control log app
machine control log app --wide

machine control log lists every command that changed the run, from the prompt or the command line, with the entry where it was issued. Commands that only read are not listed. --wide adds columns.

Stopping and starting

1
2
3
4
5
6
machine pause app
machine resume app
machine stop app
machine start app
machine kill app
machine delete app

pause freezes a running machine and resume lets it carry on. stop shuts a run down and keeps everything it recorded. kill asks the machine to stop and save as stop does, and ends it if it does not stop. The next start replays a tip that was not saved. In a session, both wait until what the run recorded is saved to your storage, and show how far the save has got. start brings it back at the end of its log. If the run had gone past its last checkpoint when it stopped, Machine replays the recorded tail from that checkpoint and tells you how many entries it replayed. Add --no-recover to make it refuse instead.

start can also take a run back to an earlier point. --at replays from an earlier checkpoint, and --read-only opens a checkpoint without running it at all.

1
2
3
machine start app --at ^booted              # replay from booted to the end of the log
machine start app --read-only --at ^booted # the machine at booted, to read
machine start app --no-recover # refuse rather than replay the tail

A run recorded by another version of Machine does not start. Its recording replays only on the version that made it, so start refuses it with it was recorded by another version, followed by the version of its run log and the version the session reads.

--at brings the run up at the checkpoint and replays every entry after it, so the run ends at its tip as it would have without the option, and you see that the recording still replays. A read-only run accepts commands that read, such as machine dump, machine hash and machine output, and refuses run and anything that would record. Stop it, and start it again without --read-only to go on.

A run remembers the options you created it with: --no-daemon, --single-step, --limit and --backstop-limit. Every start and fork repeats them, so you type them once. start takes the same flags to change them for that start only, so machine start app --daemon gives you a machine to drive on a run created with --no-daemon. fork and detach take them too, and the new run keeps what you gave. --pin-cpu is the exception: no run remembers a CPU, so each start and fork picks the least-loaded one unless you name it.

What a run records is fixed when you create it: --side-dump, --side-trace, --mem-hash, --disk-hash and --repeatable. A start keeps it. A branch is where it changes: fork, detach and go take each of them, and --no-mem-hash, --no-disk-hash or --no-repeatable to turn one off, and a branch given none records what its parent records. Replay says what the two hashes add.

delete only removes stopped runs, unless you add --stop. It also refuses to remove a run that has branches, unless you add --recursive. And since machine delete --all removes every run, it asks you first.

delete waits until the run is gone, which takes a while for a run with many checkpoints. Add --no-wait and it returns as soon as the delete is accepted, after stopping the machine if you asked it to. The delete carries on either way, and machine ops shows how it went.

A delete that was interrupted, by a lost connection or a stopped service, finishes when you run the same delete again. --cleanup -a finishes every interrupted delete at once, and --dry-run lists them first:

1
2
machine delete --cleanup -a --dry-run
machine delete --cleanup -a

To continue a run in a different session, publish it first from the session that has it.

1
machine sync app

To have a run or a disk waiting in a session before you need it, preload it. Going the other way, machine offload lists the copies in this session that can be dropped to free space, and -a drops every one that isn’t in use. The durable copy stays, and starting the run brings it back.

1
2
3
machine preload run app
machine preload disk app
machine offload

machine cleanup removes stale runs and other leftovers, and --force also kills machines that have been orphaned.

Checkpoints and branches

Branching is cheap, because a branch shares its parent’s history instead of copying it. That is what makes it practical to try many things from one point.

Checkpoints

1
2
machine checkpoint app booted
machine checkpoints app

The name is optional. Leave it off and the checkpoint is named after the entry it was taken at. Checkpoints are how you mark points you plan to come back to, like “just after boot” or “right before the request”.

A name uses letters, digits, ., - and _, and starts with a letter or digit. It can’t be a number, because numbers name the checkpoints taken without one, and it can’t read as a time, like 10ms, because a time names a point too.

Machine also takes its own, at the interval create sets, named after the entry they were taken at (like 10000), so machine checkpoints app lists yours next to those.

The DURABLE column says whether a checkpoint is in your account’s storage or only in the session. A new checkpoint shows no until the run is published, with machine sync app or by stopping the machine. On a branch, the listing includes its ancestors’ checkpoints, named in the ANCESTOR column. The point a branch was forked at is listed once, as its ancestor’s checkpoint, because that is the one a fork there starts from.

--durable lists only the checkpoints in your account’s storage, the ones another session can start the run from:

1
machine checkpoints app --durable

From Python, run.checkpoint("booted") saves one and returns it, and run.checkpoints() lists them.

Starting a branch

machine fork creates a new branch at a point. The branch runs in its own machine, alongside its parent, and the parent keeps running.

1
2
3
4
machine fork app                            # at the current point
machine fork app --at ^booted # at a checkpoint
machine fork app --at console:120 # where line 120 was printed
machine fork app --at ^booted --name try-1 # and give it a name

Give a branch a name with --name. If you don’t, it is named after the entry it landed on. Add --no-start to create the branch without starting it.

A branch’s full name is the name of the run its tree starts from, a slash, and the name you gave it. So machine fork app --name try-1 makes app/try-1, and a branch of app/try-1 named try-2 is app/try-2. Every other command takes the name you gave it, machine lineage try-1, as long as no other run has a branch of that name. When two trees both have a try-1, Machine refuses the short name and lists both, so use the full name: machine lineage app/try-1.

From Python, run.fork(at="^booted", name="try-1") returns the branch as a Run, so you drive it like any other: branch.run(...), branch.replay().

Switching the machine

machine fork gives the new branch a machine of its own and leaves you where you were. go, switch and rewind work in place instead: they change the machine you are on. detach is the odd one out, and is explained below.

Command Does
machine go app ^booted creates a branch at a point and switches this machine onto it, in place
machine switch app app/try-1 switches the machine onto a branch that already exists, and never creates one
machine detach app --at ^booted copies a point into a new root, a self-contained run in a tree of its own
machine rewind app to ^booted puts the run back at a point and throws away what came after

go is the one you will use most. It creates a branch the way fork does, but it doesn’t start another machine. It switches the one you are on, in place.

1
machine go app ^booted --name g1
1
✔ Branched app/g1 (whUxiqseEZ7Si8R4osT8K) at entry 16239 and switched onto it.

After a go or a switch, the run the machine left is listed as switched and no longer running. machine start app brings it back at its tip, and tells you how many entries it had to replay to get there. The machine you switched is now the new branch, so talk to it by that name: app/g1.

A switch needs a branch that isn’t running somewhere else. Pointing it at a branch that is gets refused:

1
2
3
4
✗ Already running: app/try-1.
Stop it first machine stop app/try-1
Then switch machine switch app app/try-1
Or drive it where it is machine control app/try-1

Creating the branch with --no-start avoids this.

rewind changes the run itself, not a copy of it. Use it when you want to record again from an earlier point. The run keeps its name. Machine refuses to rewind past a branch that starts after that point, since that branch depends on entries that would no longer exist.

switch takes a branch name, a mark (%crashed), parent or root.

detach is for leaving a tree behind. A branch normally shares its ancestors’ data, which is what keeps it cheap. detach copies everything into a new root that owns all of its data and belongs to no tree but its own. That is much slower than a fork, so only reach for it when you want to get rid of the original.

If you only want a branch to have its own copy of the data, run machine fork --rebase, which is quicker. Add --lean and the branch keeps no history shared with its ancestors, so they can be deleted.

The tree

1
2
machine branches app          # the whole tree this run belongs to
machine lineage app/try-1 # the chain of ancestors back to the root

branches lists the newest 25 by default. --running shows only the ones still running, and --depth keeps one level of the tree.

Marks

When a branch turns out to be interesting, mark it.

1
2
machine marks add app/try-1 crashed
machine marks list app

A mark is unique within its tree, so you can use it anywhere a run is named, written with a percent sign.

1
machine output %crashed

From Python, run.marks.add("crashed") marks the branch, run.marks.remove("crashed") takes the mark off, and client.marks.list() lists every mark.

Replay

Replay is what makes a recording useful. A branch can play back what its parent recorded and Machine checks, step by step, that it gets the same result.

1
machine replay app/try-1

Replay is strict. An entry that does not reproduce what was recorded means the run has diverged. Before giving up on it, Machine replays that entry again from the nearest checkpoint before it, twice by default. Use --retry to change that, or --retry 0 to try each entry once.

--to stops at a point instead of the end. It takes a point as every command does: --to ^booted for a checkpoint, --to '#300' for an entry, --to 10ms for a time. The point is one of the line being replayed, so a checkpoint the parent took after the fork is a valid one. --trace replays one iteration at a time and prints each entry as it goes, which is slower.

Each entry holds hashes of the machine’s state: its registers, its devices and its timers. A replay compares them all. A run created with --mem-hash also hashes the guest’s memory at every entry, and one created with --disk-hash its disk, and a replay of it compares those as well. Hashing costs time at every entry, so both are off by default. A divergence of memory or the disk alone then shows up later, once it reaches a register or a device. A branch can turn either on or off:

1
2
3
machine create --disk app.img --mem-hash --disk-hash   # hash both from the start
machine fork app --at ^booted --mem-hash # a branch that hashes memory
machine fork app --at ^booted --no-disk-hash # a branch that does not hash the disk

A replay hashes memory and the disk exactly where the recording did, whatever the replaying branch records, so a replay compares either kind of recording. diff of an entry that hashed one against an entry that did not leaves it out, and says Memory not compared or Disk not compared.

The host can interrupt an entry too. That is not a divergence. Replay prints Interrupted: entry N, crossing 1: host SMI and replays the entry again, counted against the same --retry. A recording does the same on its own, see Driving it.

Replay margins

A replay reaches each recorded stop by letting the CPU’s counter interrupt a margin short of it, then stepping to the stop. --limit-margin is that margin in conditional branches, and --instr-margin in instructions, for a stop placed by an instruction count. A replay that names either uses the values it names. Without them, it uses the larger of the run’s margin and the host CPU’s. A start, a start --at, a rewind and a fork --from name none, so they use the larger of the two as well.

1
2
machine replay app/try-1 --limit-margin 20000
machine replay app/try-1 --instr-margin 100000

A replay that passes a recorded stop diverges and says by how much. On amd-1, the architecture of the standard class, a replay given --instr-margin 5, far below the CPU’s skid, answers:

1
2
3
✗ Divergence at entry 804. The replay passed the recorded stop by 3,370 instructions, with its margin 5.
Rewind to the entry before it machine rewind trq/ex to #803
Then replay with a margin past the skid machine replay trq/ex --instr-margin 50000

The suggested margin is twice the overshoot and the margin together, and never less than the host CPU’s margin. Here twice the overshoot and the margin is 6,750, so the hint is the AMD CPU’s margin, 50,000. A stop counted in branches names --limit-margin instead. At the control prompt the hint is /replay instr-margin 50000. A replay that reaches the stop’s address on a later pass, with other registers, gets no margin hint, because no margin explains it.

If a run does diverge, machine diff shows how.

1
2
machine diff app/try-1
machine diff app/try-1 --entry 5000 --show regs,stack

With no second argument, diff compares the entry against the run it was replayed from, and says ✔ No differences when they match. Otherwise it prints a hash by default, and --show takes any of the sections in its help: registers, stack, page tables, timers, devices, memory and more. Name another run to compare two recordings. A difference there only means the recordings differ. It says nothing about determinism.

There is also machine play, which plays a log back without checking for divergence. It is for when you want to see what happened, not whether it repeated. By default it plays the parent’s log, so it needs a branch. To play another run’s log, name it: machine play app/try-1 app.

1
2
machine play app/try-1 -n 16240:16243
machine play app/try-1 --include irq random

--include plays only the kinds of entry you name, and --exclude plays every kind but those.

Comparing entries

machine diff compares one entry. A difference often starts a few entries before the one that shows it, so --context also prints the entries leading up to it, on both sides:

1
2
machine diff app/try-1 --entry 5000 --context 5      # a branch against the run it replayed
machine diff app/try-1 app/try-2:4980 --entry 5000 # entry 5000 of try-1 against entry 4980 of try-2

Between two recordings, such as two branches with different seeds, a difference is expected: it shows where their paths parted.

A replay that diverges is different. Every recording must replay, so a divergence is a bug in Machine, not in your program. Keep the run and the branch that diverged: the recording is what reproduces it. Open an issue with the session, the run, the branch, the entry where it diverged and the commands you ran.

At the control prompt

Replay and diff are one loop at the control prompt. Open it on the branch, and chain the two with ;+, so /diff runs whether the replay converged or not:

1
machine control app/try-1
1
2
3
4
/replay to ^booted;+/diff
/diff 120 regs
/diff 120 context 5
/play 295:300 +irq

/replay takes to, retry, trace, limit-margin and instr-margin as words. /diff 120 regs compares entry 120 with its recording and shows only the registers, and context 5 adds the five entries before it. /play feeds the parent’s entries 295 to 300 without checking them, here only the interrupts.

From Python

run.replay() returns the result instead of raising. Check converged, and diverged holds what differed:

1
2
3
4
5
6
7
kid = run.fork(at="^booted", name="check", start=False)
run.run(iters=200)
kid.start()
result = kid.replay()
if not result.converged:
print(result.entry_id, result.diverged)
print(kid.diff(entry=result.entry_id).diverged)

run.replay(to="^booted", retry=5, trace=True) takes the options of the command. The SDK has the rest.

Comparing two machines

machine diff compares an entry with the one it was replayed from. machine hash compares two machines stopped at the same point: a recording, and a branch you drove there yourself.

1
2
3
machine hash app
machine hash app mem,disk
machine hash app/try-1 vcpu --vcpu 1

The command computes the hashes of the machine as it stands: memory, the disk, each device, each part of each vCPU, and irq_pending, the hash of the interrupts --irq-delay is holding. It changes nothing, and the run records nothing. Memory is read from all of RAM, so it takes as long as that reading takes. The machine refuses commands that would change it until the command returns. machine cancel stops it.

machine dump --show hash prints what the newest entry recorded, before its exit was handled. Memory and the disk are 0 there unless the run hashes them at each entry. machine hash is after that entry, with the exit handled, and it hashes memory and the disk whether or not the run does at each entry.

A replay compares the hashes the recording stored at each entry. hash is for two machines you have in front of you.

Steering programs

By default the machine schedules itself. Sometimes that isn’t what you want. A bug that needs one thread to run past another will not show up if the scheduler never picks that order. Machine lets you pick it.

Everything here is recorded, so a replay repeats it exactly.

Looking at tasks

1
machine tasks app

That lists the tasks in the machine, one row per task by its tid, with the tgid each one belongs to. You will want a tid, or a program’s path, in the commands below. machine syscalls app and machine ints app list the kernel’s system calls and interrupt vectors.

Focus

Machine matches a program by its absolute path in the machine. Tell it which programs to watch and steer.

1
2
machine focus app add /app
machine run app --until /app:exit

Focus also takes a cgroup (cgroup=/path), a single process (tgid=12), or the processes with a task of a given name (comm=cat). A name picks out one tool of a multi-call binary such as busybox, or a script, which a path cannot. machine tasks app exist comm=cat lists the tasks with that name; focus takes every thread of their processes.

A program that renames itself after it starts, with prctl(PR_SET_NAME) or /proc/self/comm, is focused by its new name at its next fork, exec or exit, not at the rename. Until then --until comm=worker:on waits. Focus the program by its path to steer it from the start. focus remove and focus clear take them off again.

1
2
machine focus app add comm=cat
machine run app --until comm=cat:exit

Whatever you focus by, the conditions below and a plan’s when name it the same way: /app:exit, cgroup=/path:on, comm=cat:kernel.

Once a program is focused you can wait for things that happen to it. The task conditions take the form program:what, where what is one of:

What Means
exec the program started
exit the program exited
sched one of its threads was given a CPU
desched one of its threads gave its CPU up
on a thread is running its own code
kernel a thread is running in the kernel
off the thread is not on a CPU at all

A run can also stop where a thread reaches a location, or after it writes or touches bytes. These need no focus. They name a tid or a tgid:

1
2
3
machine run app --until 'tid=56:at=parse'          # until tid 56 reaches parse
machine run app --until 'tid=56:write=counter' # until tid 56 writes counter
machine run app --until 'tgid=12:access=buf[0]' # until any task of tgid 12 reads or writes buf[0]

Each takes one of the run’s 3 hardware breakpoints and watchpoints, for that run alone. Debugging says what they take and what the run says when it stops.

Holding threads

1
2
3
4
machine tasks app hold 56         # take tid 56 off the run queue
machine tasks app release 56 # put it back
machine tasks app yield 56 # let its peers go first
machine tasks app slice 56 10us # how long it runs between decisions

Each of these takes effect at the next scheduling decision. If you hold the last task that could run, the machine has nothing to run. Release it, then force a decision with machine irq app preempt --vcpu all.

There is also pin, which places a thread on the vCPUs you choose the next time it wakes:

1
2
3
4
machine tasks app pin 56 2      # on vCPU 2
machine tasks app pin 56 0-1 # on vCPU 0 or 1
machine tasks app pin 56 none # on no vCPU, so it does not run
machine tasks app pin 56 free # on any vCPU again

A set is one vCPU or a range, both ends included. A pin decides where the thread goes when it wakes, so a thread already queued on another vCPU stays there until then. none keeps a thread from running at all, until you pin it somewhere again.

Two more things you can ask of tasks. machine tasks app exits lists the tasks that have left and how each one ended, and machine tasks app cgroups groups them by cgroup. --focused shows only the tasks you are steering.

From Python, the same verbs are run.tasks.hold(56), run.tasks.release(56), run.tasks.yield_(56), run.tasks.slice(56, "10us") and run.tasks.pin(56, "2"). run.tasks.exits() and run.tasks.cgroups() read the other two lists.

Plans

When one rule isn’t enough, write a plan file. A plan is a list of clauses, and each one says when it applies and what to do.

1
2
3
when /test:on       mode step,fast   fast=1:100k step=1:200
when /test:kernel task-sched hold
else mode fast

Machine tries the clauses in order and the last match wins, so a later clause overrides an earlier one. else covers whatever nothing matched.

Each when names a state of a program (/test:on is the program running its own code, and /test:kernel is the program running in the kernel). After that come the things to choose for each stretch of execution it covers, such as the mode, how many conditional branches to run, or what to do with the scheduler. machine plan --help lists all of them.

1
2
machine plan app plan.txt
machine run app

A clause draws any of these, once per dispatch, and irq-delay once per interrupt:

Axis Draws
mode how the machine runs: fast, step, step-into, free, fence or keep, with the length of a stretch as fast=1:100k
task-sched what the task on the vCPU does: keep, hold, yield, drain, drain-all
task-int whether to preempt that task: none or preempt
task-slice how long that task runs between decisions: keep,default,10us:600us
task-pin where a task goes when it next wakes: a vCPU set, free or none
vtime-rate how fast the machine’s clock passes, as in Clock rate
vtime-add time added to the clock before the dispatch, such as 10us:2ms
timer-slack how late a timer may be, such as 0:20ms
irq-delay how long each interrupt is held, as in Holding interrupts

Every axis is a comma list of choices, and [N] makes a choice N times as likely: mode step[3],fast[1] draws step three times as often as fast. keep draws nothing and leaves the axis as it was, so a clause can leave an axis alone part of the time. A range such as 1:200 draws one value from it, and the plan’s --seed makes every draw repeatable.

A when names what a task is doing: /path:on for a program running its own code, /path:kernel in the kernel, /path:syscall=openat in one system call. The subject can also be tid=N, tgid=N, comm=NAME or cgroup=PATH, and vcpu=1 narrows it to one vCPU. Braces may group a clause, # starts a comment, and a file named - is read from standard input.

1
2
3
4
# Find the race: interrupt the writer, and move it between vCPUs.
when /usr/bin/app:on mode step[3],fast[1] step=1:200 fast=1:50k
when tid=56:on task-int none[4],preempt[1]
else mode fast task-pin 56=golden:2[3],3[1]

The plan applies only to that machine plan. A later machine run does not use it.

From Python, run.plan("plan.txt", seed=3) runs a plan from a file, and the RunResult it returns lists each clause with the iterations it applied to. The plan_file.py example shows one.

Interrupts

Inject an interrupt by name or number.

1
machine irq app preempt --vcpu 1

The interrupt is queued, then delivered at the next boundary and recorded there, so a replay delivers it at the same place.

--nmi delivers a non-maskable interrupt instead, which reaches a vCPU even while the guest has interrupts turned off, so you can test how a program or the kernel handles one:

1
machine irq app --nmi --vcpu 0

Holding interrupts

machine run can hold each interrupt a drawn time before it is delivered, at most one second:

1
2
machine run app --irq-delay 0:2ms
machine run app --irq-delay timer=0:2ms,blk=1ms

A bare span holds every interrupt. Name a target to hold one kind: timer, ipi, init, blk, rng, console, serial, keyboard, device or all. The most specific target wins. An interrupt is never delivered before one held earlier on its line, its MSI vector, or its vCPU’s timer, IPI vector, or INIT and SIPI.

machine irqs lists every interrupt the machine owes or holds, then the vectors machine irq injects. An owed interrupt was raised by a device or a timer and is injected at the next dispatch. A held one waits for the time the draw gave it. The state hash irq_pending is of the held ones, and machine state shows how many are held.

1
machine irqs app

The run log keeps each hold, and a checkpoint keeps what is held. At the control prompt the same option is irq-delay on /run. From Python it is run.run(irq_delay="0:2ms"), and run.irq.pending() lists them.

Machine time

Machine time is not the host’s. machine vtime app rate changes how fast it passes, and machine vtime app add pushes it forward without running anything.

1
machine vtime app add 5s

There is no way back. The machine’s clock only goes forward, so to return to an earlier time, branch there with machine go app 1.5s.

Starting date

Every machine’s clock starts at 2026-03-01 00:00 UTC, whatever the date is on your computer, so a program that reads the date reads the same one in every run. To test code that depends on the date, such as a certificate that expires or a job that runs at the end of a month, give the start as a Unix timestamp when you create the run:

1
2
machine create app --disk app --vtime-base 1893456000   # 2030-01-01 00:00 UTC
machine create app --disk app --vtime-base "$(date -d 2030-01-01T00:00:00Z +%s)"

The run keeps its starting date, and every branch of it has the same one.

Clock rate

The machine’s clock moves with the work the guest does: 10 nanoseconds for each conditional branch a vCPU retires, unless the run was created with --vtime-rate. A higher rate makes the same work take longer in the guest’s eyes, which brings timeouts closer, and a lower one makes it faster. The rate is a number or a fraction:

1
2
machine vtime app rate 20      # the same work takes twice as long
machine vtime app rate 1/2 # 0.5 ns a branch

The change is recorded, so a replay changes the rate at the same point. Idle time is a separate matter: Time compression skips it whatever the rate.

From Python, client.create("app", disk="app", vtime_base=1893456000) sets the starting date, run.vtime.rate("20") the rate, and run.vtime.add("5s") pushes the clock forward.

Fault injection makes things fail: system calls, the kernel and the disk, a service’s container, and the date the guest and its containers read.

Fault injection

Machine makes things fail inside the machine, at the point you choose or by chance, and records each fault with the run, so a replay fails exactly the same way. There are four kinds of fault, each with its own command:

Command What it makes fail Section
machine inject a program’s system calls and its functions, the kernel’s disk requests and memory allocations, and the disk’s data System call faults, Function faults, Kernel faults, Disk faults
machine containers one service’s container: stopped, killed, crashed, paused, throttled, or limited in memory, processes, open files or local ports Container faults, Resource exhaustion
machine clock the date the guest or one container reads: stepped, drifting, jittering, or a leap second Clock faults
machine net the links between containers: slow, lossy, out of order, rate limited, or cut Network faults

machine inject makes things fail inside the machine. A fault goes into a subsystem: a program’s system calls, or one of the kernel’s own places where things can fail, such as the disk or memory allocation, or the disk itself. You find out what your program does when openat says EACCES, write says ENOSPC, the disk returns an I/O error, the kernel runs out of memory, or the disk quietly returns the wrong bytes, at the call you choose or by chance.

Subsystem What fails
syscall a system call returns an error you choose, or waits first
function a function of a program, such as malloc, returns a value you choose without running
disk a disk read or write returns an I/O error
disk-timeout a disk request never completes, and its task waits forever
slab, pages a kernel memory allocation fails, as ENOMEM in most calls
usercopy the kernel fails to copy to or from the program’s memory: EFAULT
futex-access a futex call fails to reach the program’s memory: EFAULT
disk-corrupt a disk read or write succeeds with bytes changed
disk-torn a write succeeds, but only its first sectors reach the disk
disk-lost a write succeeds, and none of it reaches the disk

Every subsystem takes the same options: --in for whose calls, --prob for the chance, --after to let calls through first, --times to stop after a number, --seed, and --on and --off to make the fault cycle. The disk subsystems also take --file, --op and --sectors to say which requests. The guest kernel decides each call itself, without stopping the machine, and a replay fails exactly the same calls.

System call faults

A syscall fault names a system call and the errors it may return, and the program’s calls of it return one of them without running.

1
2
3
machine inject app add syscall connect ECONNREFUSED --in /usr/bin/app
machine exec app run -- /usr/bin/app
machine inject app

The last command lists the faults, with how many calls each one matched and fired on. For a program that connects once, it shows that one call failed. The seed is drawn, so yours differs:

1
2
#   INJECT                 ERRORS                   DELAY        IN               FIRES                 PROB       SEED  MATCHED   FIRED
1 syscall connect ECONNREFUSED - /usr/bin/app every call 1 1820473361 1 1

The guest kernel decides at each call, and the run records every fault you add, so a replay fails the same calls with the same errors. A checkpoint keeps the faults and their counts, and a branch has its parent’s.

Which calls fail

--in says whose calls fail, and you always give it:

  • A program, by its absolute path: --in /usr/bin/app. The fault applies to every task running that program, and to the processes it forks without starting another program, such as a server’s workers. It starts with the program, so you can add it before the program runs. Use the path machine tasks app shows as EXE: a symlink counts as the file it points to, so busybox’s /bin/cat is /bin/busybox. A program that loads shared libraries opens and reads them after it starts, so add a fault on openat, read or stat once it is up, or it fails the loading instead.

  • A task’s name: --in comm=cat. The kernel names a task after the file the command named, up to 15 bytes. That picks out one tool of a multi-call binary such as busybox, and a script, whose program is its interpreter but whose name is its own. The interpreter’s first open is of the script itself, so --after 1 lets it read the script and fails what the script opens:

    1
    2
    machine inject app add syscall open EACCES --in comm=s.sh --after 1
    machine exec app run -- /tmp/s.sh

    That is busybox’s shell, which calls open. A glibc shell calls openat and opens its shared libraries first: count them in --after, or add the fault once the script runs.

  • One process: --in tgid=412, its pid. Add it once the program is running. Before then, that process is still being set up to run it.

  • One thread: --in tid=41.

  • Every task in the machine, the system’s own included: --in all.

To make sure something runs before you act on it, ask first. machine tasks app exist takes the same words, lists the tasks that match, and fails when none does. It only reads, so it changes nothing:

1
machine tasks app exist /usr/bin/app && machine inject app add syscall openat ENOENT --in /usr/bin/app

A rule’s --after and --times count calls across every task it matches: for a program, the 100th write is the 100th of all its processes together.

1
2
3
4
machine inject app add syscall write ENOSPC --in /usr/bin/app --after 99 --times 1   # the 100th write
machine inject app add syscall read EINTR --in /usr/bin/app --prob 0.1 # a read in ten
machine inject app add syscall read EIO --in /usr/bin/app --prob 0.5 --times 2 # by chance, twice at most
machine inject app add syscall read EIO --in /usr/bin/app --prob 0 # count the calls, fail none

--after lets that many matching calls through first. --prob is the chance a call fails, from 0 to 1. --times is the most calls that fail; without it there is no end.

The chance belongs to each call, not to a stretch of time. A fault stands from when you add it until you delete it, and every matching call draws on its own: 1,000 calls at --prob 0.1 fail about 100 of them, however long they take. To fault a stretch, such as the next 10 seconds of machine time, add the fault, run for that long, and delete it:

1
2
3
machine inject app add syscall read EIO --in /usr/bin/app --prob 0.1
machine run app --within 10s
machine inject app delete 1

To fail one particular call, run until the program makes it, then add the fault for one call. The stop is at the start of the call, so that same call fails:

1
2
3
machine run app --until tgid=412:syscall=write
machine inject app add syscall write EIO --in tgid=412 --times 1
machine run app

Name several system calls at once and each gets a fault of its own:

1
machine inject app add syscall open,openat,creat EACCES --in /usr/bin/app

Which errors

Any Linux error goes, by name or by number, whether or not the system call’s manual lists it. Give a set and each failing call returns one of them. The examples here fail openat of /usr/bin/app, so add them once it runs, after machine tasks app exist /usr/bin/app:

1
2
3
machine inject app add syscall openat ENOENT,EACCES,EMFILE --in /usr/bin/app   # one of three, each call
machine inject app add syscall openat valid --in /usr/bin/app # one the manual documents
machine inject app add syscall openat valid,-EINTR --in /usr/bin/app # those, but not EINTR

valid is the errors the system call’s manual page lists. all is every error Linux names, which is more than the 30 a fault holds, so use --pick with it.

An error the manual does not list is one Linux would not return from that call. getpid failing with ENOSPC never happens on a real system. Machine allows it on purpose: it tests whether the program handles an error it did not expect. Use valid for failures a real system produces, and an error by name or all to test the program’s handling of any error.

Picking errors at random

--pick takes a number of errors from the set when you add the fault, and the fault keeps those. This is swarm testing: each run fails with its own few errors.

1
2
machine inject app add syscall openat,read,write valid --pick 1 --in /usr/bin/app   # one error each
machine inject app add syscall openat all --pick 3 --in /usr/bin/app # three of any

Every fault has a seed, and the list shows it. The seed decides the picks, which calls fail by chance and which error each one returns. Give --seed to draw as another run did:

1
machine inject app add syscall openat valid --pick 2 --seed 7 --in /usr/bin/app

Slow calls

A fault can hold a call instead of failing it. --delay takes a time, or a span to draw one from for each call, up to a second:

1
2
3
machine inject app add syscall read --delay 0:20ms --in /usr/bin/app            # each read waits 0 to 20ms, then runs
machine inject app add syscall connect ECONNREFUSED --delay 500ms --in /usr/bin/app # waits, then fails
machine inject app add syscall write --delay 5ms --prob 0.1 --in /usr/bin/app # one write in ten waits 5ms

The wait is on the machine’s own clock, and each call draws its time from the fault’s seed, so a replay holds the same calls for the same time. --prob, --after, --times and --in work as they do for errors.

Function faults

function makes a function of a program return a value without running. A malloc that returns 0 is an allocation that failed, which no system call fault reaches: the C library serves most allocations from memory it already has, without asking the kernel.

1
2
3
machine inject app add function malloc --return 0 --prob 0.01 --in /usr/bin/app
machine inject app add function malloc --in comm=sort --at 628 # found in task 628's files
machine inject app add function fopen --after 3 --times 1 --in /usr/bin/app

The function is found by its name in the files a running task maps, its C library or the program itself, and the fault acts on every task that maps the same file and that --in names. With --in naming a program that runs now, the function is found in that program. With one that does not run yet, name with --at a task that maps the same library, such as a shell of the same container.

For example, --in comm=sort with --return 0 makes BusyBox’s sort exit 2 on its first allocation, while seq and tail in the same container run as before. With --prob 0.05, four runs of sort match 60 calls and fail 4. The seed decides which calls fail, so a replay fails the same calls.

  • errno is not set. The C library keeps errno in each thread’s own memory, so a caller that checks the return sees the failure, and one that reads errno reads what was there before.
  • The function’s first instruction. The kernel puts a breakpoint there, in the tasks --in names alone, and a fire returns to the caller before the function sets up its frame. Each call traps into the kernel, which costs time in a function called often.
  • The task --at names. It must be a task inside the container, not the docker exec that started it: machine tasks app shows each task’s program as the container sees it, such as /bin/sh.

Memory faults

Memory goes wrong in four places: a program’s allocator returns nothing, the kernel refuses memory, a service runs out of it, or the data in it is wrong. Each has its own fault:

You want to know Fault
Does the program handle malloc returning NULL? (errno is not set) inject add function malloc --return 0 --after 5 --times 1 --in /usr/bin/app
Does it survive one allocation in a hundred failing? inject add function malloc --return 0 --prob 0.01 --in /usr/bin/app
Does it handle the kernel refusing to grow its heap? inject add syscall brk ENOMEM --in /usr/bin/app
Does a system call fail cleanly when the kernel cannot allocate? inject add slab --after 50 --times 1 --in /usr/bin/app
What happens when a service runs out of memory? containers memory db 64m
How does a service behave under memory pressure, short of a kill? containers memory db 48m --soft
Does the program notice a flipped bit in its data? mem 412 'checksum^=0x1'
What does a pointer set to 0 do? mem 412 'head=0'
What does a buffer of garbage do? mem 412 buf --fill 0xde --length 4096

A program’s allocations

A program asks its C library for memory, and the library serves most requests from memory it already has, without a system call. A function fault on malloc reaches every allocation:

1
2
machine inject app add function malloc --return 0 --after 5 --times 1 --in comm=sort --at 628
machine exec app run --container api -- sort -n /tmp/numbers
1
sort: out of memory

sort failed on its sixth allocation, and the next run, with the fault used up, sorted as before. calloc, realloc and strdup are functions of their own: name each that the program calls.

The kernel’s answer

brk grows the heap and mmap maps new memory. A fault on them is the kernel refusing, which the C library sees first: it turns a refused brk into an mmap, and a refused mmap into malloc returning NULL.

1
machine inject app add syscall brk ENOMEM --in /usr/bin/app

A refused brk returns the break unchanged, as the kernel’s own refusal does, and the C library moves to mmap: with every brk of sort failing, all 12, it sorted 100,000 lines and exited 0. That is the test of a program that assumes its heap is contiguous.

A program’s first mmap calls are its dynamic loader’s, mapping its libraries, and a fault on those stops the program before it runs: sort failed with error while loading shared libraries: libc.so.6: failed to map segment from shared object with mmap failing after its 13th call. A program with a small heap makes no mmap of its own, so function malloc is the fault that reaches its allocations.

The kernel’s own allocations

slab and pages fail allocations inside the kernel, for work a system call or a page fault does for the program, as Kernel faults describes. The program sees what the kernel does with the failure: a system call that returns ENOMEM, or a page fault that cannot be served. In one test, failing the 51st slab allocation of sort killed it with SIGSEGV.

A service out of memory

machine containers memory limits a container’s memory. Past the limit the guest kernel reclaims memory, and its out-of-memory killer kills a process of the container when it cannot:

1
2
machine containers app memory db 32m
machine exec app run --container db -- sh -c 'x=$(head -c 64000000 /dev/zero | tr "\0" a)'
1
⚠ Exit status 137: killed by SIGKILL, as Docker reports a signal

--soft sets the limit past which the kernel reclaims and slows the container’s processes, and kills none of them, as memory pressure does. What the kernel can reclaim decides what the program sees, and the machine has no swap:

  • Files the program reads are cached in memory the kernel can drop and read again. Past a soft limit the program keeps running, slower. For example, reading a 100MB file takes 0.20 seconds of machine time against a soft limit of 16MB and 0.02 seconds without. Its cache falls from 103MB to 8MB, the cgroup counts 202 times over the limit, and nothing is killed.
  • Memory the program allocates has nowhere to go without swap, so past a soft limit its processes slow to a crawl. For example, a shell holding 40MB against a soft limit of 16MB makes no progress in 2 seconds of machine time. The cgroup counts 27,935 times over the limit and kills nothing, and the shell goes on once the limit is lifted:
1
2
machine containers app memory db 16m --soft
machine containers app memory db 0 --soft # the soft limit lifted

machine containers app shows each container’s limits in its LIMITS column, such as memory 64MB soft 48MB.

Wrong data

machine mem writes a program’s memory with the machine stopped, ^= flips bits of a value, as C’s ^= does, and --fill writes a pattern over a span. Stop the program where the data is live first, at a breakpoint or a run --until. The names below stand for the program’s own variables and function:

1
2
3
4
5
6
machine break app add 412 process_record
machine run app
machine mem app 412 'checksum^=0x1' # one bit of checksum flipped
machine mem app 412 'head=0' # a pointer set to NULL
machine mem app 412 '$sp+8=-1' g # a word of the stack overwritten
machine mem app 412 buf --fill 0xde --length 4096 # a buffer of garbage

The answer names the bits, the address and the value before and after. Here optind, a variable of the C library a shell of a container maps, is named by its file, as machine symbols lists it, since the shell has one too:

1
2
machine mem app 628 '/lib/libc.so.6:optind^=0x1'
machine mem app 628 '$sp-64' --fill AB --length 5
1
2
✔ Flipped 0x1 at 0x7fe1a2385248 <optind> of tid 628: 0x00000001 became 0x00000000
✔ Filled 5 bytes at 0x7ffe41563260 of tid 628 with AB

Each write is recorded with the run, so a replay corrupts the same value at the same point, and a fork from before it runs the program unharmed.

Resource exhaustion

A program runs out of a resource in two ways, and Machine tests both:

  • The resource runs out. A Linux limit on one container makes its processes meet the limit, as they would on a full production host. Use it to see how a service behaves as a whole when it hits the limit. Memory, processes and CPU are shared by the container’s processes. The open-file limit is each process’s own count.
  • One call fails. A system call fault makes one program get the error the resource gives, at the call chosen, while everything else runs on. Use it to test one error path, exactly and repeatably.
Resource Run it out Fail one call The error
CPU containers throttle db --cpus 0.1 none: everything slows
Memory containers memory db 64m inject add function malloc --return 0 SIGKILL, or malloc returns NULL
Processes containers pids db 50 inject add syscall clone EAGAIN EAGAIN
Open files containers files api 64 inject add syscall openat EMFILE EMFILE
Ephemeral ports containers ports api 40000-40009 inject add syscall connect EADDRNOTAVAIL EADDRNOTAVAIL
Disk space, inodes fill a volume of its own inject add syscall write ENOSPC ENOSPC

A fault that fails one call also takes --in, the program it fails, as One failed call shows. Memory faults covers memory. The rest of this section covers the other resources.

You want to know Command Example
Does a service survive when it cannot start a process? containers pids db 10 Processes
Does a program cope when it runs out of file descriptors? containers files api 32 Open files
Does a client cope when it runs out of local ports, or churns through them? containers ports api 40000-40001 Ephemeral ports
Does the system cope with one slow service? containers throttle db --cpus 0.1 CPU
Does a service that crashes on a limit recover once the limit is lifted? containers memory hog 64m, then memory hog 0 A service that crashes on its limit
Does a service survive its disk filling up? a volume of 64MB, filled with dd A full disk
Does a program report a failed fork, a full disk or no inodes? inject add syscall clone EAGAIN, write ENOSPC, mkdir ENOSPC One failed call

Limits on a container

machine containers sets each limit on one container. The limit holds when the container starts again, until it is lifted, so a service that crashes on the limit and restarts meets it again. Every start has the limit before the container’s program runs its first instruction. A limit set on a stopped container applies when it starts. A container that Compose recreates starts without it. Use --for 30s to set the previous limit back after 30 seconds of machine time, on whichever start of the container runs then. A disk built before the guest kept container limits is refused, with a hint to build the disk again.

A limit of 0 sets back the container’s own: what its Compose file gives it, such as mem_limit or pids_limit, or no limit. A container with mem_limit: 512m and memory api 128m goes back to 512MB on memory api 0, running or stopped.

After pids db 10, machine containers app shows the limit in the LIMITS column:

1
2
machine containers app pids db 10
machine containers app
1
2
3
4
SERVICE  CONTAINER       ID            STATE    IMAGE            ADDRESS     LINK         LIMITS
api compose-api-1 0474e6006f79 running clockprobe:test 172.18.0.3 veth3ede11a -
db compose-db-1 8ef3a4936249 running clockprobe:test 172.18.0.4 veth31535d2 pids 10
nest compose-nest-1 6830a8b2a2fc running clockprobe:test 172.18.0.2 vetha032499 -

The other limits, and lifting one:

1
2
3
4
5
6
7
machine containers app pids db 20                # 20 processes and threads
machine containers app files api 64 # 64 open files a process
machine containers app ports api 40000-40009 # 10 local ports for each destination
machine containers app pids db 20 --for 30s # set back after 30 seconds of machine time
machine containers app pids db 0 # the container's own limit
machine containers app files api 0 # the container's own limit
machine containers app ports api 0 # the container's own range

Each limit is set by a command the run records, so a replay sets it at the same point, the restarts included.

A service that crashes on its limit

A service that dies of a limit and restarts by its Compose file’s restart: policy meets the limit again, and the run goes on. Here hog grows to 100MB five seconds after it starts, under restart: always:

1
2
3
4
5
services:
hog:
image: clockprobe:test
restart: always
command: ["sh", "-c", "sleep 5; x=$$(head -c 100000000 /dev/zero | tr '\\0' a); echo grew; sleep 100000"]
1
2
3
4
machine containers app memory hog 64m
machine containers app stop hog
machine containers app start hog
machine run app --within 40s

The out-of-memory killer takes it each time it grows, 3 times in 40 seconds of machine time, and the console says each time:

1
=== hog exited by itself: code 137. Its restart policy starts it again ===

memory hog 0 while it restarts lets it grow, and it prints grew. A service with no restart: policy that exits by itself ends the run, with its code.

Processes

pids limits the processes and threads of a container. Past the limit, a fork or a new thread fails with EAGAIN, and the processes already running go on. Here a shell in db starts background jobs under a limit of 10:

1
2
machine containers app pids db 10
machine exec app run --container db -- sh -c 'for i in $(seq 20); do sleep 30 & done'
1
sh: can't fork: Resource temporarily unavailable

After pids db 0, running that command again starts all 20.

Open files

files limits the files each process of a container may hold open. Sockets and pipes count as files. Past the limit, an open, a socket or an accept fails with EMFILE, and files already open stay open.

The limit is set on the processes running now, and the processes they start take it. A process that docker exec starts, machine exec --container included, takes the container’s own limit. So set the limit while the program runs. Here a shell in api was already running when the limit was set:

1
2
3
4
machine exec app start --container api -- sh -c \
'cd /tmp; for i in $(seq 100); do : > f$i; done; sleep 10; ulimit -n; timeout 2 tail -f f*'
machine containers app files api 32
machine run app --within 15s

The shell prints 32. Then tail opens the files f1 to f100, in the order f1, f10 to f19, f100, f2. It holds 29 of them and its standard input, output and error, 32 files, and every file after that fails:

1
tail: can't open 'f35': Too many open files

A later machine exec --container api -- sh -c 'ulimit -n' prints 1024, because a process docker exec starts does not take the limit set on the processes already running. It takes the container’s own limit. files api 0 sets that own limit back, both its soft and its hard number.

Ephemeral ports

ports narrows the range of local ports a container’s connections take from. A connection is told apart by its local and remote address and port, so connections to different servers can share a local port. The range is not a count of connections: it runs out for one destination at a time. Here db runs a server on port 9000, and api connects to db’s address from the listing above, 172.18.0.4, with 2 ports:

1
2
3
4
machine exec app start --container db -- tcpsvd 0 9000 cat
machine containers app ports api 40000-40001
machine exec app run --container api -- sh -c \
'for i in 1 2 3; do (sleep 5 | telnet 172.18.0.4 9000 >/dev/null) & done; wait'
1
telnet: can't connect to remote host (172.18.0.4): Cannot assign requested address

Two connections take both ports, and the third fails with EADDRNOTAVAIL. Four things follow from how the kernel gives out ports:

  • A closed connection keeps its port for 60 seconds. Its side stays in TIME_WAIT. Short connections in a loop therefore run out of a small range even though no more than one is open at a time. Right after two connections close, netstat -tan shows both ports in TIME_WAIT, and 65 seconds later neither.
  • Name lookups need a port too. A DNS query takes a UDP port from the same range. With both ports held, a client connecting to db by name failed with telnet: bad address 'db', before it tried to connect.
  • A program that binds port 0 before it connects gets EADDRINUSE. BusyBox nc does this: nc: bind: Address already in use.
  • A program that binds port 0 uses an even count of the range. Of a range of 3 ports, nc used 2.

The range belongs to the container’s network. The command refuses a container on the machine’s network, whose range is every process’s. It also refuses a container on another container’s network, with a hint to set the range on the container whose network it is:

1
2
✗ Not set: sidecar is on the network of compose-db-1.
Narrow the ports of compose-db-1 machine containers app ports compose-db-1 40000-40009

A full disk

A container’s own files are on the machine’s disk, shared by every container, so filling them fills the machine. A volume that is a filesystem of its own fills alone: only the services that mount it meet the full disk. A Compose file makes one in two ways:

Volume Full as a disk is After a restart Costs
A volume named in x-machine, with its size yes: an ext4 filesystem on a block device, with a journal, metadata and fsync its files are kept room on the disk, up to its size
A tmpfs volume, driver_opts with size= no: in memory, where fsync does nothing and no write is delayed its files are gone the machine’s memory, up to its size

Use the x-machine volume to see what a service does on a full disk. A tmpfs volume answers whether it handles ENOSPC on a write, with no change to how the disk is built. Here db has a volume of 64MB at /data:

1
2
3
4
5
6
7
8
9
services:
db:
image: clockprobe:test
volumes: [db-data:/data]
volumes:
db-data: {}
x-machine:
volumes:
db-data: 64M

Filling it with one file runs it out of space, and many empty files run it out of inodes:

1
2
3
4
5
6
machine exec app run --container db -- dd if=/dev/zero of=/data/fill bs=1M
machine exec app run --container db -- dd if=/dev/zero of=/data/more bs=4k count=10 conv=fsync
machine exec app run --container db -- rm /data/fill /data/more
machine exec app run --container db -- sh -c \
'mkdir /data/many; cd /data/many; i=0; while : > f$i; do i=$((i+1)); done'
machine exec app run --container db -- rm -r /data/many

The fill stops at 54.7MB of the filesystem’s 57MB, and the write after it stores one block of 10 and fails. The files stop at the 16,373rd, with blocks still free:

1
sh: can't create f16372: No space left on device

Meanwhile db writes its own files outside /data, api writes its files, and the machine’s disk is untouched. Deleting the files ends the full disk. A tmpfs volume of 40MB stops a dd at 40MB with No space left on device, and is empty after its container restarts.

Filling the machine’s own disk instead makes every container meet it, and the container runtime with them: machine exec --container then fails with OCI runtime exec failed. machine exec without --container still runs.

CPU

throttle gives a container a share of a CPU. Nothing fails: every process of the container runs slower. For example, a loop in db that counts to 200,000 takes 4.58 seconds of machine time, and 46.80 seconds with --cpus 0.1:

1
2
machine containers app throttle db --cpus 0.1
machine exec app run --container db -- sh -c 'i=0; while [ $i -lt 200000 ]; do i=$((i+1)); done'

One failed call

A system call fault gives one program the error a resource gives, at the call chosen. Nothing runs out, and every other program runs on. Each fault below was added, then the program after it was run, then the fault was deleted with machine inject app clear. The programs are BusyBox’s, in the containers of the ports example, and tail reads the files of the open-files example. worker.sh starts 3 background jobs and waits for them, and machine put copies it into db:

1
2
3
4
#!/bin/sh
for i in 1 2 3; do sleep 1 & done
wait
echo all started
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
machine put app ./worker.sh /tmp/worker.sh --container db --mode 755

# A failed fork: the script's second
machine inject app add syscall clone EAGAIN --after 1 --times 1 --in comm=worker.sh
machine exec app run --container db -- /tmp/worker.sh
machine inject app clear

# Out of descriptors: the third file of 8
machine inject app add syscall openat EMFILE --after 14 --times 1 --in comm=tail
machine exec app run --container api -- sh -c 'cd /tmp; timeout 1 tail -f f1 f2 f3 f4 f5 f6 f7 f8'
machine inject app clear

# Out of local ports
machine inject app add syscall connect EADDRNOTAVAIL --times 1 --in comm=telnet
machine exec app run --container api -- sh -c 'sleep 2 | telnet 172.18.0.4 9000'
machine inject app clear

# A full disk: the 11th write
machine inject app add syscall write ENOSPC --after 10 --times 1 --in comm=dd
machine exec app run --container db -- dd if=/dev/zero of=/tmp/out bs=4k count=100
machine inject app clear

# Out of inodes
machine inject app add syscall mkdir ENOSPC --in comm=mkdir
machine exec app run --container db -- mkdir /tmp/new
machine inject app clear

Each program reported its error:

1
2
3
4
5
/tmp/worker.sh: line 0: can't fork: Resource temporarily unavailable
tail: can't open 'f3': Too many open files
telnet: can't connect to remote host (172.18.0.4): Cannot assign requested address
dd: error writing '/tmp/out': No space left on device
mkdir: can't create directory '/tmp/new': No space left on device

fork in the C library is the clone system call. A script run by its own path, /tmp/worker.sh, has its file name as its comm, so the fault takes that script and no other shell. Running out of inodes gives ENOSPC from the call that creates a file or a directory, as running out of space does.

A count of calls includes the program’s start. The dynamic loader opens the program’s libraries with openat before main runs: 12 calls for tail here, of 20 in all for 8 files. --after 9 made the loader fail with error while loading shared libraries: libm.so.6, and --after 14 failed the third file. Read MATCHED in machine inject app to count a program’s calls, or use --prob 0.01 to fail calls at random.

Kernel faults

The other subsystems fail work inside the kernel, after a system call has started, where a syscall fault never reaches. They take no errors: each fails the way the kernel would.

1
2
3
4
5
machine inject app add disk --times 1 --in /usr/bin/app        # one disk request fails with EIO
machine inject app add disk --prob 0.01 --in all # a disk request in a hundred
machine inject app add slab --after 50 --times 1 --in /usr/bin/app # its 51st allocation fails
machine inject app add usercopy --after 100 --times 1 --in /usr/bin/app
machine inject app add disk-timeout --times 1 --in /usr/bin/app # its next disk request never completes

Five things decide whether you see the failure:

  • Who sends a disk request. --in names the task that sends the request, not the program the data belongs to. Your program sends its reads, its O_DIRECT writes and the writes fsync forces, so --in /usr/bin/app fails those. Ordinary writes wait in memory and the kernel’s flusher writes them later, so only --in all reaches them. To fail a program’s writes, name its files with --file and use --in all, as Disk faults does.
  • A program’s first moments. When a program starts, the kernel copies its arguments in and allocates for it. A usercopy or allocation fault that fires then kills the program before your code runs. Use --after, or add the fault once the program is up.
  • The kernel copes with some failures. A failed page allocation is often retried, and the program never sees it. MATCHED and FIRED in the list still show that the fault fired.
  • The kernel reads a page more than once. A buffered read goes out as readahead, and when that fails the kernel reads the pages again before it gives up. How many requests it makes depends on how it splits the read: for a file of 8KB, cat read the file fine with --times 1 and --times 2, and got I/O error with --times 3, all 3 of its requests failed. Leave out --times for the program to get EIO. An O_DIRECT read fails on its first request.
  • disk-timeout hangs. The request is lost, and the task waiting for it stays stuck until the machine stops. This is a hung device: no timeout fires and no reset follows. That is the point: you find out what else in the program notices. Name your program in --in: with --in all the lost request may be the kernel’s own, and whatever waits on that disk hangs with it.

Disk faults

The faults above are errors: your program is told something failed. Real disks also fail silently. A write reports success and the data is wrong, half there, or not there at all. disk-corrupt, disk-torn and disk-lost are those failures. The request succeeds, and the damage is found later, or never.

You want to know Fault
Does the database notice a bad block when it reads its data files? disk-corrupt --file '/var/lib/db/*/data.bin' --in all
Does the program trust bytes it read without a checksum? disk-corrupt --op read --file /srv/app/state.json --in all
Does recovery handle a log record only half written? disk-torn --file /var/lib/db/wal --times 1
Does it notice a committed write that never arrived? disk-lost --file /var/lib/db/wal --after 10 --times 1
Does the filesystem survive damage to its own metadata? disk-corrupt --sectors LO:HI, the sectors of its inode table or journal
1
2
3
4
machine inject app add disk-corrupt --file '/var/lib/db/*/data.bin' --op write --bytes 8 --times 1 --in all
machine inject app add disk-torn --file /var/lib/db/wal --times 1 --in all
machine inject app add disk-lost --file /var/lib/db/wal --after 10 --times 1 --in all
machine inject app add disk-corrupt --op read --prob 0.01 --in /usr/bin/app
  • --file is a pattern. * matches within a directory and ** across directories. Programs often name their files as they run, and a pattern names them before they exist. A file the program already holds open or maps, such as a running database’s WAL, matches too. A file inside a container is named by its path in the container.
  • --in all with --file. Ordinary writes reach the disk from the kernel’s flusher, not from your program, so --in all is what reaches them. --file keeps the fault on your file.
  • The page cache hides a damaged write. The kernel keeps the bytes your program wrote in memory, so the program reads them back clean until they are read from the disk again: with O_DIRECT, after echo 3 > /proc/sys/vm/drop_caches, or in a branch forked later. A damaged read is the other way round: the bad bytes go into the page cache, and every reader sees them until the kernel drops the page.
  • A tear is whole sectors. Machine tears a write at sector granularity: each 512-byte sector is written whole or not at all, so a write of one sector does not tear. A torn write keeps at least one sector of the new data and loses at least one. Real disks differ in what they write atomically; this is the model Machine applies.
  • --sectors aims below the files. It names 512-byte sectors of the whole disk, so it reaches what --file cannot: the filesystem’s inodes, bitmaps and journal. The filesystem’s own tools say where those are, such as dumpe2fs and debugfs for ext4, in blocks of its partition. With 4 KiB blocks, block N starts at sector 8 × N plus the partition’s first sector, which /sys/class/block/vda1/start holds, or 0 for a filesystem on the whole disk. Or stop at a write with catch add io, below, and read its sectors off the stop.
  • disk-lost is one write, not a crash. The lost write reports success, and the writes after it still land. A power cut loses every write since the last flush, and the disk faults do not model one. For that, use machine containers crash: it loses the writes a service’s volumes had not synced.

A replay damages the same bytes, so a bug you find this way is found again on every replay of the run.

A worked example: ClickHouse. ClickHouse writes each part of a table into a directory it names as it goes, such as all_12_12_0, and checks the part’s checksums when it reads it back. To damage one column file of the next part written and see what ClickHouse does:

1
2
3
4
5
6
machine inject app add disk-corrupt --file '/var/lib/clickhouse/data/db/t/*/data.bin' \
--op write --bytes 8 --times 1 --in all
machine exec app run -- clickhouse-client -q "INSERT INTO db.t SELECT number FROM numbers(1000000)"
machine exec app run -- sync
machine exec app run -- sh -c 'echo 3 > /proc/sys/vm/drop_caches'
machine exec app run -- clickhouse-client -q "CHECK TABLE db.t"

To damage one part you have already seen, find its name first on a branch, and keep your run where it is:

1
2
3
4
5
machine fork app --name peek
machine exec app/peek run -- clickhouse-client -q "INSERT INTO db.t SELECT number FROM numbers(1000000)"
machine exec app/peek run -- sync
machine checkpoint app/peek
machine fs ls app/peek /var/lib/clickhouse/data/db/t --at now

Then add the fault on app with the exact path. The run is deterministic, so it writes the same part. An insert’s part name holds; a merge’s can change, because any fault changes how the machine runs a little. A pattern covers both.

Faults that cycle

Real outages come in bursts: a disk that is flaky for two seconds, a network that drops for five. --prob fails calls one at a time and independently, which rarely exercises retries, backoff or a circuit breaker. --on and --off make a fault active for a stretch of machine time, rest for another, and repeat:

1
2
machine inject app add disk --file /var/lib/db/data --on 2s --off 5s --in all
machine inject app add syscall open,openat ENOENT --in tgid=63 --on 2s --off 3s --seed 5

The first fails the data file’s disk requests for 2 seconds in every 7. The cycle starts when the fault is added, active first, on the machine’s own clock, so a slow host changes nothing and a replay crosses every edge at the same instruction. The machine’s clock is not the date the guest reads, so a machine clock change does not move a window. While a fault rests it matches no call and counts none, so --after and --times count only the calls in its active windows. Both options are needed, each a whole number of milliseconds from 1ms to a day.

The listing says which window each fault is in and how long it has left. After the second command it reads:

1
2
3
4
5
#   INJECT                 ERRORS                   DELAY        IN               FIRES                 PROB       SEED  MATCHED   FIRED
1 syscall open ENOENT - tgid=63 every call 1 5 0 0
on 2s off 3s now on, 2s left
2 syscall openat ENOENT - tgid=63 every call 1 5 0 0
on 2s off 3s now on, 2s left

machine inject app later shows the window moving: now on, 996ms left one second in, and now off, 2496ms left at two and a half seconds.

A shell loop that opens a file twice a second under the second command sees 4 failures, then 6 successes, over and over: 2 seconds failing, 3 seconds working. Each window has a fixed length. To try outages of other lengths, fork branches from one checkpoint and give each its own --on and --off.

Stopping at a fault

To look at the program at the moment a fault fires, add a catchpoint for it:

1
2
3
4
machine catch app add fault openat
machine catch app add fault disk
machine catch app add fault disk-corrupt
machine run app

The run stops where the fault fires, before a system call’s wait and error, and says which: Stopped at catchpoint 1: fault openat EACCES, in tid 229, or fault disk, in tid 77. See Debugging for catchpoints.

Stopping at a file’s I/O

An io catchpoint stops the run at a disk request, with no fault. --file takes the same patterns as a disk fault, and --op takes read or write:

1
2
machine catch app add io --file '/var/lib/clickhouse/data/db/t/*/data.bin' --op write
machine run app

The run stops as the request is sent, before any disk fault decides on it, and names the request: Stopped at catchpoint 1: io write 1058144:1058151, in tid 266. A fault added there decides on that same request. Name it by its file:

1
2
3
machine inject app add disk-lost --file '/var/lib/clickhouse/data/db/t/*/data.bin' --times 1 --in all
machine catch app delete 1
machine run app

Or by its sectors and its task, which name this request and no other:

1
machine inject app add disk-corrupt --sectors 1058144:1058151 --times 1 --in tid=266

An ordinary write is sent by the kernel’s flusher, so the stop names a kernel task such as kworker/u4:1. --in tid=N names that task as well as any other.

Changing faults

1
2
machine inject app delete 2       # by its number in the list
machine inject app clear

At the control prompt it is /inject, with the same words, and /tasks exist checks first:

1
2
/tasks exist /usr/bin/app
/inject add syscall openat ENOENT --in /usr/bin/app

Faults and plans work together. Add the faults, then run the plan: the faults apply to every stretch of execution, whatever the plan chooses. To change them partway, stop, change them, and run on.

From Python, run.inject.add(...) takes the same subsystem, errors and options, with in_= for --in, and run.inject() lists the faults with their counts. The SDK’s Faults has an example.

What a fault misses

  • A C function is not always the system call of its name, and C libraries differ. glibc’s open() calls openat, musl’s calls open, and glibc’s stat() calls newfstatat. Name both, open,openat, to cover either. machine syscalls app lists the names the kernel uses.
  • Give --seed in a script that records again. Without it each run draws its own seed. A replay needs none: the run keeps the fault as it was added.
  • 64-bit system calls only. A 32-bit call, made with int 0x80, never fails.
  • Calls the vDSO answers never reach the kernel. clock_gettime, gettimeofday, time and getcpu return from the vDSO without a system call, so a fault on them matches nothing and MATCHED stays 0.
  • 16 faults, and 30 errors in one. Delete a fault to add another, and use --pick to take 30 or fewer of all.
  • A fault that fired is not a failure the program saw. The kernel retries an allocation, rereads a page and recovers a filesystem, and a program retries a call. FIRED says the fault fired, and MATCHED 0 says it never met a call, so nothing was tested. Whether the program failed is a question for the program: check its output, its exit status or its data after the fault, as well as the counts.
  • A fault matches the system call, whose call it is and the count of calls. It does not look at the arguments, so “openat of /etc/passwd” is not a fault. Stop at the call and fail that one instead.

Container faults

A project run as containers is a small cluster, and each service is a node. machine containers puts a fault on one container while the others keep running:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
machine containers app stop db                   # SIGTERM, then SIGKILL after 10s
machine containers app stop db --grace 2s # SIGKILL after 2s of machine time
machine containers app kill db # SIGKILL, no cleanup
machine containers app crash db # a power cut: its volumes lose what was not synced
machine containers app pause db --for 5s # frozen for 5s, then thawed
machine containers app throttle db --cpus 0.1 --for 30s
machine containers app throttle db --cpus 0 # the container's own limit again
machine containers app memory db 64m --for 30s # past 64MB, its out-of-memory killer
machine containers app memory db 0 # the container's own limit again
machine containers app memory db 48m --soft # past 48MB, reclaimed and slowed, not killed
machine containers app pids db 20 # past 20 processes, fork fails with EAGAIN
machine containers app files api 64 # a process past 64 open files: open fails with EMFILE
machine containers app ports api 40000-40009 # 10 local ports: connect fails with EADDRNOTAVAIL
machine containers app start db
You want to know Fault
Does the cluster keep serving while one node restarts cleanly? stop db --restart 10s
Does a client retry when its server dies mid-request? kill db --restart 5s
Does the database keep what it committed through a power cut? crash db --restart 0
Do the other services notice a node that hangs without dying? pause db --for 30s
Does the system cope with one slow node? throttle db --cpus 0.1 --for 1m
What happens when a node runs out of memory? memory db 64m
What happens when a node cannot start a process, open a file or connect? pids db 20, files api 64, ports api 40000-40009; see Resource exhaustion
Does the service keep state it never put in a volume? kill db --restart 0 --fresh

A container you stop, kill or crash stays down. --restart 5s brings it back after 5 seconds of machine time, and --restart 0 at once. The other containers keep running through the outage:

1
2
3
machine containers app kill db --restart 5s
machine containers app crash db --restart 0 # a power cycle
machine containers app kill db --restart 0 --fresh # back from its image
1
2
✔ Killed compose-db-1 (db)
Starts again in 5s of the machine's time, process 28493p1

A restart is a reboot: the container comes back with every file it had. --fresh replaces it instead, as a failed node is replaced by a new one from the same image. The files it wrote outside its volumes are gone, and its volumes are kept. Use it to find state a service keeps in its own files when it should keep it in a volume.

A limit holds when the container starts again, as a limit in its Compose file would: a service the out-of-memory killer stops comes back to the same limit. Every start has the limit before the container’s program runs. Use throttle db --cpus 0 to set back the container’s own limit, its Compose file’s or none, or --for 30s to restore the previous limit after 30 seconds of machine time, on whichever start of the container runs then. A limit set on a stopped container applies when it starts. A container that Compose recreates, or start --fresh replaces, is a new container and starts without the limit.

memory limits the memory a container’s processes may use, as its cgroup’s memory.max. A limit below what they use makes the guest kernel reclaim memory, and its out-of-memory killer kill a process of the container when it cannot. A process killed that way exits 137, by SIGKILL:

1
2
3
4
machine containers app memory db 32m
machine exec app run --container db -- sh -c 'x=$(head -c 64000000 /dev/zero | tr "\0" a)'
echo $? # 137: killed by its out-of-memory killer
machine containers app memory db 0

memory takes --for, and its limit holds through restarts, as throttle‘s does. machine containers app lists each container’s limits in its LIMITS column, cpus 0.1 memory 64MB or - for none. A disk built before container limits is refused, with a hint to build it again. An out-of-memory kill is part of the run: a replay kills the same process at the same point.

Each fault is recorded with the run, so a replay faults the same container at the same point. Put one where the run is stopped, at a breakpoint, a catchpoint or a run --until, to fault a service at the moment that matters.

A crash

kill loses nothing the program wrote: the kernel inside the machine keeps running and writes it out later. crash is a power cut of the service’s node. Its volumes keep what was synced and lose the writes still in memory. When the service starts again, each volume replays its journal and the service runs its own recovery. That finds a commit acknowledged before its data was synced, a rename whose directory was never synced, and a log record that survived while the data it describes did not.

A crash cuts the volumes the project names in x-machine; see Volumes a crash cuts. The cut is when the volumes stop writing to the disk, just after the container is frozen; a write the kernel finishes in between is kept, as one a disk finished before the power went. A volume another running container also mounts is refused, since cutting it would cut that container too.

1
machine containers app crash db /var/lib/postgresql/data --restart 0   # that volume alone
1
2
3
✔ Crashed compose-db-1 (db)
Cut compose_db-data
Started again

The volume is named as Docker names it, after the Compose project.

A crash test, start to end. Stop where the database syncs, keep the run there, and crash a branch of it. The branch loses what the database had not synced yet; the query tells you whether it had already told a client the data was safe:

1
2
3
4
5
6
machine catch app add syscall fsync fdatasync --in comm=postgres   # where it syncs
machine run app # stops at the next sync
machine fork app --name cut # the run keeps its place
machine catch app/cut delete 1 # the branch has its parent's
machine containers app/cut crash db --restart 0
machine exec app/cut run --container db -- psql -U postgres -c 'select count(*) from t'

The run is deterministic, so the branch crashes at exactly that sync every time. Run on to the next sync and fork again to cut at another point.

From Python, run.containers.crash("db", restart="0") and the other verbs take the same options as keywords. The SDK’s Container faults has an example.

Clock faults

Software reads two kinds of time. Elapsed time, CLOCK_MONOTONIC and the TSC, is the machine’s own, and Machine time controls it. The date, CLOCK_REALTIME, is what date, time() and gettimeofday return, and on a real fleet it jumps, drifts and disagrees between hosts. machine clock changes the date inside the machine, for the whole guest or for one container, and leaves elapsed time alone.

Command What the date does
jump +5s steps forward or back, as an NTP step or date -s does
set 2027-03-28T00:59:50Z goes to that date
drift +200ppm --container db gains 200 parts per million of machine time, 0.72 seconds an hour
jump +4s --slew 30s --container db moves 4 seconds over 30 seconds of machine time, with no step
jitter -200ms:200ms --every 10m is set to a drawn offset every 10 minutes of machine time
leap insert repeats a second, 23:59:60, at the next UTC midnight
reset returns to machine time, for the guest and every container

The guest kernel makes each change, so a change takes effect when the run next executes. The run records every change: a replay makes it at the same instruction, a checkpoint keeps the date, and a branch starts with its parent’s date.

The guest’s date

jump, set and reset change the date every process in the guest reads, in every container. The step is a real step of the guest kernel’s clock: a timer armed for an absolute date sees it, and a timerfd with TFD_TIMER_CANCEL_ON_SET is woken, which is how systemd and cron notice a step.

1
2
machine clock app jump +59m50s
machine run app --within 30s --until 'console~token refresh'

The answer shows the date as it stands and the change waiting for the run:

1
2
3
date:    2026-03-01T00:01:29.614903749Z
offset: +0s from machine time
waiting: 1 change, taken when the run executes

Steps add up: jump +5s and then jump -2s leave the date 3 seconds ahead. set puts the date at a value, whatever steps came before it. Machine refuses a date before 1970, or one on or after 2232-04-18T23:47:16Z, because the kernel cannot set either.

A container’s date

With --container, a change moves one container’s date alone, as an offset from the guest’s date. Every process of the container reads it, including those docker exec and machine exec --container start. The other containers keep the guest’s date.

1
2
machine clock app jump -6m --container client
machine exec app run --container client -- curl -sS https://api:8443/health

The client now reads a date 6 minutes behind the server, so a certificate the server issued a minute ago is not valid yet for the client.

The offset reaches every way a program reads the date. clock_gettime stays as fast as it is outside the machine, because the container’s processes still read the date in the vDSO, without a system call. System calls, POSIX timers, timerfd and clock_nanosleep agree with the container’s date, and so do the futex waits behind pthread_cond_timedwait and sem_timedwait and message queue timeouts. A restarted container keeps its offset, while a container created again starts on the guest’s date.

--container takes a container’s id, its name or its Compose service. Machine finds the container without running the machine, so a clock change made at a breakpoint leaves the run at that breakpoint. Prefer the id: Machine knows it as soon as Docker creates the container, and learns the name shortly after the container starts. machine containers app lists both.

A stopped container that has an offset keeps it, and takes it again when it starts. machine clock app reset --container db returns it to the guest’s date, stopped or running. A container Docker removes, with docker rm or a --fresh restart, loses its offset, and the container that replaces it starts on the guest’s date. At most 8 containers have dates of their own at once; the next is refused until one is reset or removed.

Drift and slew

A clock that runs fast or slow is the fault a jump cannot fake: nodes do not jump apart, they wander apart until an assumption breaks. drift makes one container’s date gain or lose time at a rate of machine time:

1
2
3
machine clock app drift +5000ppm --container node2      # 0.5% fast: 3s in 10 minutes
machine run app --within 10m --until 'console~two leaders'
machine clock app drift 0 --container node2 # stopped; the offset reached stays

No read sees a jump, and a timer armed in the container agrees with its drifting date: it fires when the container’s own date reaches the deadline, never before. At +100000ppm, a timerfd armed for 2 seconds ahead by the container’s date fired after 1.821 seconds of machine time, which is 2 seconds of a clock running 10% fast.

--slew moves a container’s date to a new value over a time instead of in one step, as NTP slews a small correction:

1
2
machine clock app jump +4s --for 2m --container db               # off and back, two steps
machine clock app jump +4s --for 2m --slew 30s --container db # drifts off and back, 30s each

A slew runs until it arrives. A later change to the same container, a drift included, stops it where it has got to, so set a drift before you start a slew. A slew moves the date at most half a second for each second of machine time, so jump +4s --slew 8s is the fastest slew of 4 seconds. Machine refuses a shorter one:

1
✗ A slew of +4s takes 8s at the least.

The guest’s date has no drift. To make the whole machine’s time run fast or slow, use the clock rate.

Jitter

An NTP-disciplined fleet is never exactly in sync: each host’s offset wobbles within a bound. jitter sets the date to a value drawn around it each period:

1
2
3
machine clock app jitter -200ms:200ms --every 10m --container api
machine clock app jitter -50ms:50ms --every 1m --container worker
machine run app --within 2h

Every 10 minutes of machine time, api’s offset is set to a value from -200ms to 200ms around the date without the jitter. The first step is taken when the guest takes the command. A weight in brackets draws a choice more often: -1s[1],0[8],1s[1] is in sync most of the time and a second off now and then. A jitter takes 4 choices at most, each a whole number of microseconds.

The draws come from --seed, or from a seed drawn at random and shown. Give that seed with --seed to step the date the same way again. A replay steps by the same amounts at the same points, and a branch draws on from where its parent was. A jump or a set moves the date under the jitter, which goes on around the new date. machine clock app jitter off stops the jitter and returns the date to where it would be without it.

Leap seconds

In 2012 a leap second livelocked Linux futexes, and services across the industry went down with it. leap insert makes the guest kernel repeat a second, 23:59:60, at the next UTC midnight of the guest’s date, and leap delete makes it skip 23:59:59:

1
2
3
machine clock app set 2026-12-31T23:59:50Z
machine clock app leap insert
machine run app --within 20s

After the 20 seconds the date is a second behind where it would have been without the leap. With leap delete it is a second ahead. A leap second moves the guest’s date and every container with it, so leap takes no --container. leap off cancels a leap still pending, and so does reset.

Undoing a change after a time

--for 2m takes a change back after 2 minutes of machine time, at the same instruction in every replay. jump +4s --for 2m is a step forward and, 2 minutes later, a step back: the NTP correction that follows a host running with a bad clock. jump, set, drift and jitter take --for.

The undo takes back the command’s own change. What else moved the date meanwhile stays: a drift’s accrual, a leap second, or another program’s step. A later change cancels an undo still pending.

Reading the date

machine clock app reads the date without running the machine. It shows the guest’s date and its offset from machine time, then each container whose date is off the guest’s, with its offset, drift, slew and jitter. To see the date a program reads, ask from inside its container, which runs the machine:

1
machine exec app run --container db -- date -u

The reading looks like this:

1
2
3
4
5
6
7
8
9
date:    2026-12-31T23:59:54.977514123Z
offset: +305d23h58m18.362410928s from machine time
jitter: -200ms:200ms every 10m, seed 42, at -0.022376s after 1 draw
leap: 23:59:60 inserted before 2027-01-01T00:00:00Z
container: compose-db-1
date: 2026-12-31T23:59:57.645176188Z
offset: +2.667662065s from the guest's date
drift: +200ppm
slew: to +6.00600183s at 122.130465170s of machine time

--container db shows one container, and machine -o json clock app gives every field.

A change is taken when the run next executes, so the command that asks for it cannot say whether the guest kernel accepts it. Read the clock after the run that takes it. The reading names each change of the last batch the guest kernel refused, as you typed it, and why:

1
2
3
4
machine clock app set 1970-01-01T00:00:01Z
machine clock app jitter -5ms:5ms --every 1s --seed 1
machine run app --within 1s
machine clock app
1
2
3
date:    2026-03-01T00:00:40.556767815Z
jitter: -5ms:5ms every 1s, seed 1, at +0.000356s after 1 draw
refused: clock set 1970-01-01T00:00:01Z: the date, the guest's or a container's, would leave the range the kernel takes: from 1970 plus the time since the guest booted, to 2232-04-18T23:47:16Z

The jitter applied and the set did not. A refused change changes nothing.

Clock fault use cases

Each block below is a question a clock fault answers, and the commands that ask it. Run the machine afterwards, with machine run app --within 2m or with an --until the program prints, and read what it did.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# Does the program refresh a token before it expires?
machine clock app jump +59m50s
machine run app --within 30s --until 'console~token refresh'

# Does a client accept a certificate issued in its future?
machine clock app jump -6m --container client

# Do two nodes both lead when one clock runs fast?
machine clock app drift +5000ppm --container node2

# Does last-write-wins lose the write from the slow data center?
machine clock app jump +3s --container dc2

# Does the program survive the clock going backwards?
machine clock app jump -2s

# Does the system cope with the NTP correction that follows a bad clock?
machine clock app jump +4s --for 2m --slew 30s --container db

# Does a node shut itself down on skew, and come back?
# At +1000ppm the offset passes a 500ms tolerance after 500s of machine time.
machine clock app drift +1000ppm --container roach3
machine run app --within 10m
machine clock app reset --container roach3

# Does the program survive 23:59:60?
machine clock app set 2026-12-31T23:59:50Z
machine clock app leap insert

# Does a scheduled job run twice or not at all across a time zone change?
# Set TZ in the container first.
machine clock app set 2027-03-28T00:59:50Z

# Does the program handle 2038 and February 29?
machine clock app set 2038-01-19T03:13:30Z
machine clock app set 2028-02-28T23:59:55Z

# Do services that compare timestamps agree when clocks wobble?
machine clock app jitter -200ms:200ms --every 10m --container api

# Does a cache or session expire when its day is over?
machine clock app jump +1d --container redis

How much skew the system tolerates

Fork the same checkpoint into branches, one skew each, and run each to the same point. Every branch starts from the identical machine, so the skew is the only difference:

1
2
3
4
5
6
machine checkpoint app steady
for skew in 50ms 100ms 200ms 400ms 800ms; do
machine fork app --at ^steady --name skew-$skew
machine clock app/skew-$skew jump +$skew --container node2
machine run app/skew-$skew --within 2m --until 'console~two leaders'
done

A jump inside one request

Most time bugs are check-then-use: the process checks a token’s expiry, the clock steps, and it uses what it checked. Stop the run where the check is done, jump, and go on:

1
2
3
4
5
machine containers app                       # auth is 3f2a91c0d4e7
machine break app add 412 check_expiry
machine run app --until 'tid=412:at=check_expiry'
machine clock app jump +1h --container 3f2a91c0d4e7
machine run app --within 30s

The jump finds the container by its id without running the machine, so it lands while the run is stopped at check_expiry.

A step back reorders events

A database that stamps rows with the time they were written sorts them by that stamp. Step its container’s date back between two writes, and the second row sorts before the first:

1
2
3
4
5
machine exec app run --container db -- psql -U postgres -tAc "create table ev(id serial, at timestamptz default clock_timestamp())"
machine exec app run --container db -- psql -U postgres -tAc "insert into ev default values"
machine clock app jump -10m --container db
machine exec app run --container db -- psql -U postgres -tAc "insert into ev default values"
machine exec app run --container db -- psql -U postgres -tAc "select id, at from ev order by at"
1
2
2|2026-03-01 00:50:42.230825+00
1|2026-03-01 01:00:41.531503+00

Any code that takes the newest stamp as the latest event, such as last-write-wins or a cursor over at, now picks row 1.

Because every clock change is part of the run’s recording, a failure found under skew comes back when you replay the recording; there is no timing to reconstruct. Investigate it as any other failure, with rewind, diff and the debugger.

What a clock change misses

  • File timestamps. The kernel stamps files with its own clock, so make in a container behind the guest reports clock skew.
  • Socket timestamps. SO_TIMESTAMP and SIOCGSTAMP carry the guest’s date, so ping in a skewed container reports the offset as round trip time.
  • Timers already armed. A container’s change does not move a deadline armed before it; a timerfd with TFD_TIMER_CANCEL_ON_SET is woken so the program can re-arm.
  • A container that sets the date. A container given CAP_SYS_TIME that calls settimeofday sets the guest’s date, as it would set a host’s; its own offset stays.

Network faults

A distributed system fails at its links: one that is slow, one that loses packets, one that stalls and then delivers everything at once, a node cut off from one peer while it still reaches the others, a connection reset in the middle of a transaction. machine net puts those faults on the links between the containers of a run.

Each container has one link: everything it sends. A fault on the whole link acts on every packet the container sends. A match narrows a fault to some of those packets: the ones to one peer, of one protocol, of one port, or the TCP SYNs that open a connection. Each match has faults, a seed and counts of its own. Every net command runs the machine until the guest has made the change, so the run moves forward.

1
2
3
4
5
6
7
8
9
10
11
machine net app                                        # each link and its faults
machine net app delay db 100ms --jitter 20ms # db's packets wait 80 to 120ms
machine net app loss db 10% # a tenth of them lost
machine net app duplicate db 5% # one in 20 sent twice
machine net app corrupt db 1% # one in 100 with a bit changed
machine net app reorder db 25% # a quarter sent ahead of the delayed ones
machine net app rate db 1mbit # db sends 1 megabit a second at most
machine net app partition db api # nothing between db and api
machine net app hold db --for 5s # db's packets held, then sent at once
machine net app reset api --to db # api's connections to db reset
machine net app heal # every link as it was

Use cases

Each command follows machine net <run>.

You want to know Command
Does a client time out and retry when its server answers slowly? delay db 2s
Does the protocol survive lost and duplicated packets? loss db 10%, then duplicate db 5%
Does a leader election hold when one node is cut off? partition node1 node2, then partition node1 node3
What happens when only one direction fails? partition db api --one-way
Does replication keep up over a slow link? rate primary 1mbit --to replica
Does the protocol recover when the network damages packets? corrupt db 1%
Does the client reconnect, and retry safely, when its connection is reset mid-transaction? reset api --to db
What happens when a server’s responses to one client are lost, and nothing else? loss db 100% --to api --port 5432
Do messages that arrive late, after a leader change or a timeout, do harm? hold node1 --for 10s
Does the cluster recover when the whole network stalls and comes back at once? hold, run --within 3s, release
What happens if the network stalls at one exact point? run --until <point>, then hold
Do established connections keep working while new ones cannot open? loss api 100% --to db --syn
Does the cluster settle when a link keeps going down and coming back? a loop of partition --for, below
What happens when a node cannot resolve a peer’s address? loss db 100% --proto arp --to api, then arp flush api and arp flush db
What happens when a node holds a wrong MAC for a peer? arp set api db 02:42:ac:12:00:99
Did the fault drop the packets, or did the program never send them? machine net app: each fault’s SENT and DROPPED, below
Which exact packet did the fault drop, and why? loss db 30% --log, then read the decision log, below
What exactly went between two containers, to open in Wireshark? tcpdump in the machine, in Capturing the packets, below

Other commands cover the failures around a network fault:

  • A refused connection, rather than one that times out: machine inject app add syscall connect ECONNREFUSED --in /usr/bin/app, in Fault injection.
  • A node that freezes rather than one that is cut off: machine containers app pause db --for 5s, in Container faults.
  • Clocks that disagree between nodes: machine clock app jump +2s --container db, in Clock faults.
  • A cluster split into two groups: one partition for each pair of containers across the split.

Without a match, a fault is on everything the container sends. With --to, it is on what the container sends to that one peer, and its other links stay as they are:

1
2
machine net app delay db 100ms --to api
machine net app
1
2
3
4
5
SERVICE  CONTAINER       TO             MATCH  FAULTS       SEED  SENT  DROPPED  QUEUED
api compose-api-1 - - - - - - -
db compose-db-1 - - - - - - -
db compose-db-1 compose-api-1 - delay 100ms - 0 0 0
nest compose-nest-1 - - - - - - -

A ping from api to db now takes 103ms, because db‘s replies to api wait 100ms. A ping from nest to db still takes 0.04ms.

Packets a match takes get only the match’s faults, not the whole link’s: after loss db 10% and then delay db 100ms --to api, what db sends to api is delayed and none of it is lost. A packet two matches take gets the faults of the one listed first. Each verb changes the fault it names and keeps the others, so delay and then loss give a link that is slow and lossy. A value of 0, or 0%, takes that fault away. A jitter also reorders packets, since each one leaves at its own time.

Matching packets: protocol, port and SYN

--proto tcp, --proto udp or --proto icmp takes the packets of one protocol. --port 5432 takes the packets with 5432 as their source or destination port, so on a server it takes its responses, and on a client its requests; without --proto it takes both TCP and UDP. --syn takes the TCP SYNs that open a connection. The options combine with each other and with --to, and a packet must satisfy all of them. Every verb that sets a fault takes them:

1
2
3
machine net app loss db 100% --to api --port 5432      # every packet db sends api on port 5432
machine net app delay db 1s --proto udp # db's UDP traffic, a second late
machine net app loss api 100% --to db --syn # api opens no new connection to db

The first loses db’s responses to api on that port, and its SYN-ACKs too, so api can open no new connection there either. The last leaves every connection api already has to db working: only the packets that open a new one are lost. A port belongs to TCP and UDP and a SYN to TCP, so --proto icmp --port 5432 and --proto udp --syn are refused. A container holds at most 15 matches besides its whole link.

Partitions

partition cuts two containers apart: everything each sends the other is lost, and both still reach the rest of the cluster. --one-way cuts what the first sends the second, and leaves the reverse. --proto, --port and --syn cut those packets alone:

1
2
3
machine net app partition db nest
machine exec app run --container nest -- ping -c 3 -W 1 db # 100% packet loss
machine exec app run --container api -- ping -c 3 -W 1 nest # 0% packet loss

With --for 10s, both directions come back together after 10 seconds of machine time.

A link that keeps failing and recovering is a loop of partitions, each followed by a stretch of the run:

1
2
3
4
for i in 1 2 3 4 5; do
machine net app partition node1 node2 --for 2s
machine run app --within 7s
done

Holding packets, then releasing them

hold keeps every packet the link sends, and release lets them all go at once, in the order they were sent. Nothing is lost, so the peer receives messages that were sent seconds ago: a stalled link that recovers, a message that arrives after a leader change or after its sender timed out:

1
2
3
machine net app hold node1 --to node2                   # node1's messages to node2 wait
machine run app --within 10s
machine net app release node1 # they all arrive now

--for 10s releases them after 10 seconds of machine time. A link holds 1000 full-size packets and drops the ones past them.

A held packet still counts against the send buffer of the socket that sent it, about 208KB by net.core.wmem_default. Once that buffer is full, the socket’s sends fail, or for TCP block, before the link’s queue is reached, so one socket cannot fill a hold by itself. For example, when api sends db 1200 pings of 1400 bytes during a hold of db, db receives all 1200 and its link holds 117 replies. The other 1084 replies fail in db’s kernel and count as OutErrors in its /proc/net/snmp. With pings of 56 bytes, 321 replies are held.

Without a container, hold holds every link at the instruction the machine is stopped at, and all networking between the containers stops there. release or heal without a container lets every packet go:

1
2
3
machine net app hold                                   # the whole cluster's network stops here
machine net app # says every link is held
machine net app release # everything held leaves at once

The hold is part of the recording, so a replay holds at the same point, and a checkpoint taken during a hold keeps it. Run the machine to an exact point first, and the hold lands there:

1
2
3
4
machine run app --until 'console~ready'                # stop where the cluster is ready
machine net app hold # hold every link at that point
machine net app loss db 100% --to api # change the faults while nothing moves
machine net app release # traffic resumes under the new faults

For example, with api pinging db every 100 milliseconds, db receives none of api’s requests while the links are held and the fault is set. After the release, db receives the 47 held and later requests, the new fault drops all 47 replies, and the listing’s DROPPED reads 47. No packet crosses a link under the old faults after the hold.

The guest’s programs and timers keep running during a hold. A retransmission or an application timeout can expire while the links are held. To hold for a time, run the machine for that time between the two commands:

1
2
3
machine net app hold
machine run app --within 3s # 3 seconds of machine time, all held
machine net app release

For example, with api pinging db every 250 milliseconds, the 3 replies sent before the hold arrive, none arrives in 3 seconds of hold, and all 20 arrive after the release. A container on another’s network, the machine’s or none has no link of its own and is left out, and so is a process outside every container. The listing names a container whose link could not be held.

Resetting connections

reset aborts TCP connections. The container’s side of each connection is aborted, and the peer receives a TCP reset, as from a middlebox that dropped the connection or a server that restarted. What a program sees depends on the call it makes. On the aborted side, the first call after the data it already received fails with ECONNABORTED; after that a read returns end of file and a write fails with EPIPE. On the peer, the first call fails with ECONNRESET, and the same follows. Unlike a partition, nothing waits for a timeout:

1
2
machine net app reset api --to db                       # api's connections to db
machine net app reset db --port 5432 # every connection to db's port 5432

A listening socket stays open, so the server takes new connections at once.

Address resolution

A container finds a peer’s MAC by ARP and keeps it in its neighbour table. --proto arp puts a fault on the ARP frames a container sends, and with --to on those about one peer: its requests for the peer’s address and its replies to the peer. A fault on ARP acts when a container resolves a peer again, so flush the tables to see it at once. With db’s replies lost, api cannot resolve db, while nest still reaches it:

1
2
3
4
5
machine net app loss db 100% --proto arp --to api
machine net app arp flush api # api resolves db again
machine net app arp flush db
machine exec app run --container api -- ping -c 3 -W 1 db # 100% packet loss
machine exec app run --container nest -- ping -c 3 -W 1 db # 0% packet loss

machine net app arp shows and changes a container’s neighbour table:

1
machine net app arp show api
1
2
INTERFACE  ADDRESS     PEER          MAC                STATE
eth0 172.18.0.2 compose-db-1 02:42:ac:12:00:02 reachable
1
2
3
4
5
machine net app arp set api db 02:42:ac:12:00:99       # api sends db's frames to a MAC nobody has
machine net app arp stale api db # used, and checked again by ARP soon after
machine net app arp delete api db # resolved again at the next packet
machine net app arp flush api # every peer resolved again
machine net app arp reachable api 1s # entries go stale after about a second

An entry made by arp set is permanent, so ARP never corrects it: api reaches db again only after arp delete. A stale entry is still used; at the next packet the kernel checks it by ARP within seconds, unless the connection’s own traffic confirms the peer first. The table is part of the machine, so a replay makes the same changes. A container that docker starts again begins with an empty table, as a host that restarts does.

The decision log

Run the machine for 10 milliseconds before reading a log just after a run stops: the last millisecond’s lines reach the run when the machine runs again.

--log writes each decision a fault makes to a log: which packet it took, and whether it dropped it, delayed it, duplicated it, corrupted it or sent it ahead. It answers which exact packet a fault dropped, which the counts and a capture do not. --no-log with the same fault stops it:

1
2
3
machine net app loss db 30% --to nest --seed 7 --log   # the link's decisions are logged
machine net app # FAULTS: loss 30% log
machine net app loss db 30% --to nest --seed 7 --no-log # the same fault, no longer logged

The guest kernel writes one line per decision, where the fault decides, to /output/drop/.machine/net/<container>.log. The .machine folder holds the files the machine itself writes, apart from a program’s own, and it is under drop, so a run stores the log only when it keeps that output. A run that keeps it from the start is read at any time:

1
2
3
4
5
machine create app --disk app.img --keep-output /output/drop/.machine/net
machine net app loss db 30% --to nest --seed 7 --log
machine exec app run --container nest -- ping -c 20 -i 0.05 db
machine run app --within 10ms # the last batch of lines reaches the run
machine output app --source /output/drop/.machine/net/compose-db-1.log --plain -n :

machine output prints the last 25 lines unless -n says otherwise; -n : prints every line. The guest kernel writes the lines in batches, at most 1 millisecond of machine time after the first line of a batch. Lines decided in the last millisecond before the machine stops reach the run when the machine runs again, and a checkpoint keeps them: run the machine for a moment, as machine run app --within 10ms, before reading the last packets’ lines.

A run that does not keep the log still makes it, and replay brings it back. Fork a branch from a checkpoint before the stretch, keeping that output, and the branch replays the stretch and stores the same lines:

1
2
3
4
5
machine checkpoint app before
machine net app loss db 30% --to nest --seed 7 --log
machine exec app run --container nest -- ping -c 20 -i 0.05 db
machine fork app --from ^before --keep-output /output/drop/.machine/net --name logged
machine output app/logged --source /output/drop/.machine/net/compose-db-1.log --plain -n :
1
2
3
4
2026-03-01T00:01:31.691986Z #8 to compose-nest-1 pass ICMP 172.18.0.4 > 172.18.0.3 echo reply id 25 seq 0 len 84
2026-03-01T00:01:31.742177Z #9 to compose-nest-1 drop ICMP 172.18.0.4 > 172.18.0.3 echo reply id 25 seq 1 len 84
2026-03-01T00:01:31.794040Z #10 to compose-nest-1 pass ICMP 172.18.0.4 > 172.18.0.3 echo reply id 25 seq 2 len 84
2026-03-01T00:01:31.846051Z #11 to compose-nest-1 drop ICMP 172.18.0.4 > 172.18.0.3 echo reply id 25 seq 3 len 84

Each line has the machine’s date, the packet’s number on its link, the match, the decision and the packet with its IP length. The number counts every packet the link carried since the container started, logged or not, so a later stretch starts at a higher number: here the link had carried 7 packets before the ping. The 8 drops of the 20 lines are the 8 pings the ping lost, and DROPPED in the listing reads 8. A replay from a checkpoint writes the same log, byte for byte.

The decisions are pass, drop, delay with the time the fault drew, ahead for a packet reorder sent before the delayed ones, and duplicate and corrupt beside the first word. drop limit is a packet dropped because the fault’s queue already held 1000 delayed packets, and drop hold one dropped because a hold already held 1000 full-size packets; DROPPED counts both. Every packet the link carries has its line and its number, and a packet a hold drops has a second line, drop hold, under the number of its first. A duplicated packet has two lines: the original says duplicate, and the copy, logged first, says copy. A delay is what the fault drew; the packet can leave later by the machine’s timer slack, as a timer can. A container on two networks writes both links’ decisions to the one file.

Every decision is logged; none is dropped however fast the packets come. For example, a 200MB TCP transfer under a logged fault decides 19,586 packets, the log holds 19,586 lines, and the listing’s SENT and DROPPED add up to them.

Each decision costs about 32 microseconds of machine time to write, and a decision is a packet as the kernel hands the link, whatever its size. A 50MB TCP transfer under a logged fault takes 0.66 seconds of machine time on average, against 0.47 seconds under the same fault without the log, because its packets are large. Small packets cost the same each, so a stream of them spends far more of its time on the log; 10,000 packets cost about 0.32 seconds. Log the stretch being examined, not a whole run. A decision is a packet’s, never a program message’s: see the limits below on TCP units and retransmission.

Seeds, replay and –for

Loss, jitter, duplicate, corrupt and reorder decide each packet by chance, from the link’s seed. Give one with --seed, or the link draws one and shows it. A delay or a rate alone draws nothing, and shows no seed. A link keeps its seed through later changes. The run records every change, so a replay delays and drops exactly the same packets, and so does a branch forked from a checkpoint before them.

A change to a link starts its draws again from the seed: loss db 10% --seed 7, then delay db 50ms, loses packets from the start of seed 7’s sequence again. Which packet corrupt changes, and which byte and bit of it, follow the seed.

A checkpoint keeps a link’s faults: a rewind to it brings them back. A container that docker starts again, after containers kill --restart or a crash, gets its faults back, toward each peer at the address it has now.

--for 30s sets back what its command set after 30 seconds of machine time, and leaves the rest of the link as it is then:

1
2
machine net app loss api 100% --for 3s       # everything api sends lost for 3 seconds
machine net app delay api 50ms # still there after the 3 seconds

A heal takes every fault off and releases what a hold keeps. A heal, and a --for that ends, act on new packets: a packet the link already delays still leaves at its time.

What a fault did

SENT, DROPPED and QUEUED count what each fault did with the packets it took. They count from when the match, or the whole link, first got a fault, and a later change to it keeps counting. SENT is the packets it passed on, a duplicate counting once more. DROPPED is the packets it lost, by loss, or past what a hold has room for. QUEUED is the packets it holds now, delayed or held: neither sent nor dropped yet. After 50 pings from api to db with loss db 30% --to api --seed 9, the row of that match:

1
2
SERVICE  CONTAINER       TO             MATCH  FAULTS    SEED  SENT  DROPPED  QUEUED
db compose-db-1 compose-api-1 - loss 30% 9 36 14 0

The ping prints 50 packets transmitted, 36 packets received, so the 14 db dropped are the 14 api lost. A hold shows what it holds as QUEUED until the release. The counts are what the fault processed, packet by packet, not a measure of its probability: over many packets 30% loss drops close to 30% of them, and over a few the count can be anything. A container that docker starts again counts from 0, and so does one a network is disconnected from. The counts are packets as the fault decides them, the unit of the log’s lines: a large TCP write that reaches the fault as one packet is one.

The counts are packets on the link, not sends by a program. A TCP write can wait in the socket’s buffer, and the kernel retransmits without a system call. machine catch add syscall sendto --in /usr/bin/app shows what a program sends, and a capture shows the packets.

Capturing the packets

Every disk has tcpdump in the machine itself, at /usr/sbin/tcpdump, so a capture needs nothing in the containers’ images. Write the capture under /.machine/output/keep, and the run keeps it as its output, named /output/keep/db.pcap.

machine containers lists what a capture needs: each container’s ADDRESS, and its LINK, the machine’s interface at the other end of the container’s own:

1
machine containers app
1
2
3
4
SERVICE  CONTAINER       ID            STATE    IMAGE            ADDRESS     LINK
api compose-api-1 2aaef0f5d567 running clockprobe:test 172.18.0.4 veth60a586c
db compose-db-1 41b6ae3b5fb0 running clockprobe:test 172.18.0.2 veth45c3e2e
nest compose-nest-1 3da165f7aa0f running clockprobe:test 172.18.0.3 veth80fa747

To capture one container’s packets, capture on its LINK:

1
2
3
4
5
machine exec app start -- tcpdump --immediate-mode -U -i veth45c3e2e -w /.machine/output/keep/db.pcap
machine run app --within 1s # tcpdump starts listening
machine exec app run --container api -- ping -c 3 db
machine exec app run -- sh -c 'killall -INT tcpdump; while pidof tcpdump >/dev/null; do sleep 0.1; done'
machine output app bytes --source /output/keep/db.pcap > db.pcap

--immediate-mode and -U write each packet as it is captured: without them, the packets tcpdump has not written yet when it stops are lost. The stop waits for tcpdump to exit, so the file is whole before it is read. Open db.pcap in Wireshark or read it with tcpdump on this computer.

On its link each packet db sends or receives appears once, as an Ethernet frame, ARP included. Broadcasts from the other containers appear too, since the bridge sends them to every link. A packet a fault on db dropped or still holds is not there: the capture sees what db’s faults let through, and what reaches db. The LINK changes when docker starts the container again, and so do addresses from run to run, so read them in the run you capture.

--net db runs tcpdump on db’s own network instead, so its eth0 is db’s link, and the name stays the same across restarts:

1
machine exec app start --net db -- tcpdump --immediate-mode -U -i eth0 -w /.machine/output/keep/db.pcap

It captures the same packets as a capture on the LINK. For example, with db’s ICMP lost by loss db 100% --proto icmp, both show the pings reaching db and none of db’s replies.

To capture every container at once, use -i any and filter by ADDRESS:

1
machine exec app start -- tcpdump --immediate-mode -U -i any -w /.machine/output/keep/all.pcap host 172.18.0.2

With -i any, each packet between two containers appears twice: as it arrives from the sender’s link, P, and as it leaves toward the receiver’s, Out. The file records each packet’s interface by an index that means nothing on this computer, so filter by address rather than by interface.

machine containers reads both from the machine; they can be read by hand as well. Docker knows each container’s address by the container’s name:

1
machine exec app run -- docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' compose-db-1    # 172.18.0.2

A container’s eth0 is one end of a pair, and the container’s iflink is the number of the machine’s interface at the other end:

1
2
machine exec app run --container db -- cat /sys/class/net/eth0/iflink    # 6
machine exec app run -- ip -o link | grep '^6:' # 6: veth45c3e2e@if5: ...

A replay writes the same capture, byte for byte, with loss, delay and clock jumps in effect. The timestamps are the machine’s date: clock jump of the whole guest moves them, and a container’s offset does not. A link between containers carries TCP segments larger than its MTU, since the kernel segments them only where a real interface needs it.

What a network fault misses

  • Traffic inside one container, over its loopback, has no link to fault.

  • DNS between containers is answered on each container’s loopback, so no fault reaches it. Point the container’s resolver at an address that does not answer, and its lookups time out; write the file back to restore them:

    1
    2
    machine exec app run --container api --user root -- sh -c 'cp /etc/resolv.conf /etc/resolv.keep; echo nameserver 192.0.2.1 > /etc/resolv.conf'
    machine exec app run --container api --user root -- sh -c 'cat /etc/resolv.keep > /etc/resolv.conf'
  • Address resolution follows the whole link’s faults unless a match names --proto arp: a match of another protocol takes IP packets alone. With loss db 30%, db’s ARP requests are lost too, and right after a start, with nothing resolved yet, db can take seconds to reach a peer.

  • A fault is on what a container sends. To fault what db receives, put the fault on each container that sends to it, with --to db. To slow both directions between two containers, put a fault on each, as partition does.

  • A container on another’s network, with network_mode: service:db, host or none, has no link of its own, and a fault on it is refused. Put the fault on the container whose network it uses.

  • A process outside every container, on the machine itself, sends on no container’s link. What it sends to a container takes no fault, and only the container’s replies do.

  • A rate is per sender. Three containers each limited to 1mbit toward db give db 3 megabits a second, not one link of 1 megabit.

  • A damaged packet rarely reaches the program. corrupt changes one bit anywhere in a packet’s frame, and the IP, TCP and UDP checksums drop nearly every damaged packet; TCP then sends it again. A bit in the Ethernet addresses is not caught by a checksum. No fault hands a damaged payload to the program.

  • Losing exactly the Nth packet, bursts of loss, and delays other than a uniform jitter: a fault decides each packet on its own, by its chance.

  • IPv4 alone. Ports and SYNs are matched in packets without IP options, which Docker’s networks send.

  • The path’s MTU stays as Docker sets it.

  • Loss of TCP is per unit the kernel hands the link, not per wire segment. Between containers the kernel sends a large TCP write as one packet many times the MTU, segmented only where a real interface needs it: for example, most are 8244 bytes. loss 1% drops 1% of those units, each as many segments as it holds, and TCP sends the whole unit again. The log’s len shows each unit’s size.

Debugging

There are two ways to debug a program in a machine. The machine’s own debugger works from outside the guest: machine regs, mem, step, break and the rest read and change a task while the machine is stopped, and they work on any program, including one with no debugger on its disk. GDB works from inside: install it on the disk, attach it with gdb -p, and use it as on any Linux host.

Both are recorded. A replay repeats every step, every write and every stop, and a checkpoint taken before a session lets you run it again from the same state.

The machine’s debugger

Each command takes the run and a task by its tid, which machine tasks lists. A task’s tgid is its process. The commands work while the machine is stopped, between runs, on a task that is on a vCPU or off one.

Symbols, types and source lines come from the program files on the machine’s disk: the program’s symbol table, and its debug information when it has some.

Debug information

What the debugger shows depends on what the program’s files on the disk hold:

On the disk What you get
The symbol table alone Function names in a backtrace, a stop at a function by its name, registers, memory, and a global at its symbol’s address
Debug information too Source lines, a stop at FILE:LINE, line steps, arguments in a backtrace, print and locals by name, and a watch of a variable by its name

Frame 1 of the same stop, in pg_parse_query, of PostgreSQL as it ships and with its debug package:

1
2
1  0x56365898a1af  ??
1 0x556ccfc7d1af exec_simple_query+159 (query_string=0x556cd01e6838 "SELECT count(*) FROM generate_series(1, 100000);") postgres.c:1071

A program built with -g has its debug information in itself. A distribution ships it apart, in a package such as -dbg, -dbgsym or debuginfo, which puts a debug file under /usr/lib/debug. Install the package of the program’s exact build in the image the disk is built from: the debugger takes a debug file only when its build id is the program’s, and finds it in any layer of a container’s image.

A server is built optimized, as it ships. Its arguments and locals are where the compiler kept them at each instruction: on the stack, in registers, or in pieces of both, as clang keeps a C++ std::string_view in two registers. One kept nowhere is optimized out. A C++ function is named short, such as DB::executeQuery. machine --signatures names it by its whole signature.

Without debug information, read the arguments from the registers the ABI passes them in, and give a global a type with a cast:

1
2
machine mem app 412 '$rsi' s                                  # the second argument, a string
machine run app --until 'tgid=7:write=*(long *)&queries' # a global of no type, as a long

The debug examples build ClickHouse and PostgreSQL both ways, and show a session of each.

Locations

Every command that takes a location takes one of these. A location names an address in the task’s process:

Location What it names
0x401136 an address
*0x401136 the same address, written with a *
main a function or a variable, by its name
libc.so.6:malloc a name in one file of the process, by its path or its file name
init.c:106 a source line, at its first code
$rsp a register of the task, as it is now
main+4, $rsp-8 any of these, moved by +N or -N

When the program has debug information, a breakpoint at a function by its name is placed past its prologue. Without it, the breakpoint is at the function’s first instruction.

Registers and memory

1
2
3
4
machine regs app 412                   # the general registers of tid 412
machine regs app 412 rip rsp # two of them
machine regs app 412 --all # with x87, SSE and AVX
machine regs app 412 rax=0 rip=main+4 # write two of them

machine mem shows memory in the format given after the location: a count, a letter and a unit, such as 8xg, 4i or s:

1
2
3
4
5
6
7
8
9
10
machine mem app 412 '$sp'                   # eight giant words at the stack
machine mem app 412 '$pc' 4i # the next four instructions
machine mem app 412 main 16xb # sixteen bytes of main
machine mem app 412 counter=42 # write counter, at its size
machine mem app 412 'counter^=0x10' # flip bit 4 of counter, as ^= does in C
machine mem app 412 --mapped # the pages the tgid maps, as info proc mappings
machine mem app 412 buf --find GET --length 4096
machine mem app 412 buf --dump buf.bin --length 4096
machine mem app 412 buf --restore buf.bin
machine mem app 412 buf --fill 0xde --length 4096 # a pattern written over a span

A write is part of the run, and a replay makes it again. A page the tgid has not written yet is shared, and is refused for a write. A name two files of the program define is refused with each file to name, /lib/libc.so.6:optind, by its path in the container as machine symbols lists it. ^= flips the bits a mask sets, at the variable’s width or a unit’s, such as machine mem app 412 '$sp^=0x1' b; it reads, flips and writes with the machine stopped, and says the value before and after.

The stack, symbols and variables

1
2
3
4
5
6
7
8
9
10
11
machine backtrace app 412              # the stack, the innermost frame first
machine backtrace app 412 --frame 1 # one frame, as info frame
machine symbols app 412 main # where main is
machine symbols app 412 0x401ac6 # what is at an address
machine symbols app 412 --regex ^wait # the functions that match
machine libs app 412 # the program and the libraries its tgid maps
machine print app 412 'shape->at.x' # a member through a pointer
machine print app 412 index --frame 1 # in the caller's frame
machine print app 412 counter x # in hexadecimal, as print/x
machine print app 412 'counter = 3' # write it
machine locals app 412 --args # the arguments of the innermost frame

print takes a variable of the frame, a global or a $register, with .m, ->m, [i], *, & and casts, and prints it by its type. print and locals need the program’s debug information.

Stepping

machine step steps one task. The machine runs whichever task the guest schedules, so other tasks run meanwhile:

1
2
3
4
5
6
machine step app 412                   # one instruction, as stepi
machine step app 412 5 --over # five, a call being one, as nexti
machine step app 412 --line # a source line, as step
machine step app 412 --line --over # a line over calls, as next
machine step app 412 --out # until the function returns, as finish
machine step app 412 --line --skip memcpy # step out of memcpy, as GDB's skip

A step of an instruction that faults, such as a write to a copy-on-write page, ends on the next instruction.

Breakpoints, watchpoints and catchpoints

A breakpoint stops the run when a task of its tgid reaches its location. A watchpoint stops it after an instruction writes the bytes it watches. A catchpoint stops it at an event of a task: an exec, an exit, a fork or vfork, a system call, a signal, a file mapped or unmapped, an injected fault, a disk request or a C++ exception.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
machine break app add 412 main                    # stop tgid of 412 at main
machine break app add 412 init.c:104 --temporary # deleted at its first hit, as tbreak
machine break app add 412 main --condition '$rdi != 0'
machine break app add 412 --regex '^parse_' # at each function that matches, as rbreak
machine break app add 412 puts --print 'puts {$rdi:s}' # print and go on, as dprintf
machine break app add 412 '*0x7f35d1580000' --hardware # in a debug register, as hbreak
machine watch app add 412 counter # after a write of counter
machine watch app add 412 counter --on access # after a read or a write
machine catch app add fork --in tgid=1 # where tgid 1 forks
machine catch app add syscall openat write # either system call, by any task
machine catch app add signal SIGSEGV --in tid=57 # a signal of tid 57
machine catch app add throw --in tgid=57 # a C++ throw
machine catch app add exit --in comm=cat # a cat ends
machine catch app add fault openat disk # an injected fault fires
machine catch app add io --file /var/lib/db/wal --op write # a disk request of a file

watch add takes --on write, change, read or access, and --size for a location that is not a variable. An exit catchpoint takes --in tgid=N, since an exit is the end of a process. A throw, catch or rethrow catchpoint needs --in tgid=N.

--in comm=NAME stops at the events of the tasks with that name, the same tasks machine tasks app exist comm=NAME lists. That catches one tool of a busybox, or a script, before you know its ids. An exec catchpoint sees the new program’s name. An exit catchpoint sees the name of the process’s main thread, which ends last, so a worker thread named cat inside a differently named process does not stop it. A syscall catchpoint with --in comm=NAME makes the machine report every task’s calls while it is set, which costs time on a busy machine.

Each stop has a number, and the same verbs change any kind:

1
2
3
4
5
6
7
8
9
machine points app                   # every breakpoint, watchpoint and catchpoint
machine break app condition 2 'n > 3' # a hit where it does not hold goes on
machine break app ignore 2 5 # pass the next 5 hits
machine break app disable 2 # kept, not armed
machine break app enable 2 --once # armed for one hit
machine break app delete 2
machine break app clear 412 main # every breakpoint at main, as GDB's clear
machine break app hits # the hits the run log keeps
machine break app save stops.sh # the commands that add each again

machine break, watch and catch each list their own kind.

A run that reaches a stop ends there and says which. The run log keeps the stops: a branch has the ones its parent had where it was forked, and a replay applies the adds and deletes. The stops are armed while the run records, and not while it replays.

Running to a place

1
2
3
4
5
machine run app --iters 0                      # until the next hit, as continue
machine run app --until 'tid=412:at=init.c:106' # until tid 412 reaches a line, as until and advance
machine run app --until 'tgid=7:at=0x401136' # until any task of tgid 7 reaches an address
machine run app --until 'tid=412:write=counter' # until tid 412 writes counter, as watch
machine run app --until 'tgid=7:access=buf[0]' # until any task of tgid 7 reads or writes buf[0], as awatch

tid=N:at=LOC and tgid=N:at=LOC take any location. write=LOC and access=LOC take a variable or an expression, watched at its size of 1, 2, 4 or 8 bytes. A write the kernel makes into the program counts, as read makes into its buffer. The address is resolved when the run starts, and the stop lasts for that run alone.

The run says the tid it stopped in and where that task stands. A watch also says the value the access left in the bytes:

1
2
✔ Written counter at main+116, watched.c:63, in tid 191
Value = 4, 0x4

Going back

Going back puts you on a new branch, at an earlier point of the run, and leaves the run as it was:

1
2
3
4
machine run app --back                 # onto a branch at the newest hit, as reverse-continue
machine step app 412 --back 3 # three executions back, as reverse-stepi
machine step app 412 --back --line # back a line, as reverse-step
machine step app 412 --back --out # back to the call, as reverse-finish

The branch keeps the run’s stops as they are now.

Calling, returning and signals

1
2
3
machine call app 412 square 3          # call a function in the task, as call
machine return app 412 0 # leave the innermost function, returning 0
machine tasks app signal 412 SIGTERM # the guest sends it, the run log keeps it

call puts the task’s registers back after the call. Each of these is part of the run, and a replay makes it again.

At the control prompt

Every command here has a slash command at the control prompt, with the run left out: /regs 412, /mem 412 '$pc' 4i, /step 412 --line, /break add 412 main, /run until 'tid=412:at=main'. The prompt also has /display, an expression printed after each run and step, and /skip, what a line step of the session steps out of:

1
2
3
4
/display 412 counter          # printed after every /step and /run
/skip function memcpy # /step --line steps out of memcpy
/step 412 --line
/undisplay 1

/help lists them all.

From Python

The Python SDK has the same debugger: run.task(tid) gives a task with registers(), read(), backtrace(), print(), step(), breakpoint() and watch(), and run.breakpoints lists and changes the stops.

How many stops a run holds

A run holds 64 breakpoints. The machine puts each one into the tgid’s own copy of its code as an int3 instruction. Another process that maps the same library does not stop, and other threads of the tgid stop at it too. A read of the task’s memory, with mem or the instruction a step shows, shows the program’s byte, not the int3. A write over that byte is refused while the breakpoint is there.

The throw, catch and rethrow catchpoints are breakpoints too. The other catchpoints are events the machine reports, and a run holds 8 of them.

A run has 3 debug registers. These take one each:

  • a watchpoint of machine watch add
  • a breakpoint added with --hardware
  • the stop of run --until tid=N:at=LOC, write=LOC or access=LOC, for that run
  • the stop of a step: step --over and step --out take one, and two in a C++ program, where the second stops at a catch the exception reaches
  • a hardware breakpoint or watchpoint of GDB inside the machine, hbreak or watch

When the 3 are taken, the next is refused. A machine watch add refused while GDB holds some says how many:

1
No room: a run holds 3 hardware breakpoints and watchpoints. The debugger in the guest holds 1.

A step of one instruction takes none, from a breakpoint’s hit too. The one exception is a task that returns into a signal handler. The step stops where the handler returns and takes a debug register for that stop.

A breakpoint in a library every process maps

A breakpoint at malloc stops one process, though every process of the machine calls libc’s malloc. The others run on, and pay nothing for it:

1
2
3
4
5
6
7
8
9
$ machine break app add 218 malloc
✔ Breakpoint 2 at malloc, in tgid 218
$ machine run app
✔ Stopped at breakpoint 2: malloc, in tid 218
$ machine backtrace app 218
# ADDRESS FUNCTION SOURCE FILE
0 0x7f998847d930 malloc /usr/lib/x86_64-linux-gnu/libc.so.6
1 0x5654d1e931b4 __cxa_allocate_exception+20 /usr/bin/cxxprobe
2 0x5654d1e92d51 work+24 (round=20) cxxprobe.cpp:12 /usr/bin/cxxprobe

A breakpoint in a library the program loads later is placed when it loads, and again each time it is loaded after being unloaded.

Many breakpoints at once

A pattern adds a breakpoint at each function that matches. For example, 38 of libc’s string functions:

1
machine break app add 251 --regex '^str[a-n]'

A pattern that matches more functions than the run has room for adds none, and says how many it matched and how many the run holds:

1
2
3
✗ 79 functions match: strcasecmp, strcasecmp_l, strcasestr, strcat, strchr and 74 more. The run holds 64 more breakpoints.
List the matches machine symbols app 251 --regex ^str
See the breakpoints to delete machine break app

Threads

Every thread of the tgid stops at a breakpoint, each with its own tid. For example, two worker threads that run the same function:

1
2
3
4
5
6
$ machine break app add 251 worker_tick
✔ Breakpoint 1 at worker_tick+7, bpprobe.c:20, in tgid 251
$ machine run app
✔ Stopped at breakpoint 1: worker_tick+7, bpprobe.c:20, in tid 303
$ machine run app
✔ Stopped at breakpoint 1: worker_tick+7, bpprobe.c:20, in tid 302

A child the process forks, or vforks, is another tgid and runs on, though it shares the parent’s copy of the code until it writes.

When to use –hardware

A software breakpoint is the right one almost always. --hardware is for the cases an int3 in the program cannot cover:

  • Code in memory the program allocated, as a JIT writes. The listing says the breakpoint is not placed, with the command that puts it in a register:

    1
    2
    3
    4
    $ machine break app
    # KIND ADDRESS SIZE WHERE TGID HITS WHEN
    1 breakpoint 0x7f35d1580000 1 *0x7f35d1580000 251 /usr/bin/bpprobe 0 not placed: the address maps no file
    Put it in a debug register machine break app add 251 '*0x7f35d1580000' --hardware
  • Code the program reads. The program sees the int3, so a checksum of its own code changes, for example. A hardware breakpoint changes no byte.

  • An instruction the kernel does not let a breakpoint sit on, such as syscall. The listing says not placed: the kernel refused the instruction.

A hardware breakpoint lists as hw breakpoint and takes one of the 3 debug registers.

A GDB hbreak or watch refused for the same reason makes GDB say Couldn't write debug register: No space left on device at the continue that inserts it. An x86 computer has 4, so a GDB session that uses all 4 there gets 3 in the machine.

GDB inside the machine

GDB runs in the machine as it does on any Linux host: breakpoints, stepi, next, finish, hbreak, watch and detach.

Getting GDB onto the disk

Put GDB in the image the disk is built from. For a server in a container, put it in the server’s image, so GDB sees the server’s files:

1
2
FROM clickhouse/clickhouse-server:26.8.31.2
RUN apt-get update && apt-get install -y --no-install-recommends gdb procps

GDB reads the program’s symbol table for function names. For arguments, locals and source lines it needs the program’s debug information too, in a format it reads. ClickHouse’s debug information is DWARF 5, which GDB 12 and later read. An older image’s GDB may not: GDB 9.2, Ubuntu 20.04’s, does not.

GDB holds what it reads in memory, and the machine needs room for it beside the program. ClickHouse’s 1.7GB of debug information grows to about 3.8GB in GDB, so that machine needs --mem 12288; with 4GB the guest’s out-of-memory killer ends GDB.

The ClickHouse example builds both kinds of image for ClickHouse 26.8: one with the symbol table alone, and one with the debug information. Its server prints the commit it was built from as it comes up, with a link to that source on GitHub, and the example shows how to put a source file where GDB finds it.

A batch of commands

Build the example’s disk from its server image, and create a run on it. The example’s README builds the image:

1
2
machine disks build --image clickhouse-server:26.8.31.2 clickhouse-server
machine create ch --disk clickhouse-server --mem 4096 --cpus 2

Run the machine to the point you want to debug, and keep a checkpoint there. Then run GDB with machine exec. Here the example’s server prints clickhouse: ready once it answers queries:

1
2
3
4
machine run ch --until 'console~clickhouse: ready'
machine checkpoint ch ready
machine exec ch run --container app --user clickhouse \
-c 'gdb -batch -ex "handle SIGUSR1 SIGUSR2 nostop noprint" -ex "break DB::executeQuery" -ex continue -ex "bt 3" -p $(pidof clickhouse-server)'
1
2
3
4
5
6
7
8
9
10
[New LWP 78]
...
Breakpoint 1 at 0x1d386340 (3 locations)
[Switching to Thread 0x7f844cbde640 (LWP 89)]

Thread 9 "TCPHandler" hit Breakpoint 1, 0x000000001d386340 in DB::executeQuery(std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) ()
#0 0x000000001d386340 in DB::executeQuery(std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) ()
#1 0x000000001ed12ec3 in DB::TCPHandler::runImpl() ()
#2 0x000000001ed37d64 in DB::TCPHandler::run() ()
[Inferior 1 (process 7) detached]

exec run runs the machine until GDB exits, and prints what GDB printed. -batch makes GDB exit after its commands, and an attached process is detached as GDB exits. A file of commands works the same way, with -x FILE. -c gives the line to the container’s shell, which finds the process id.

ClickHouse signals its own threads with SIGUSR1 and SIGUSR2. Without the handle line, continue stops at the first of those signals instead of at the breakpoint, and bt shows whichever thread got it.

At the control prompt the line after -c needs no outer quotes:

1
2
3
/run until 'console~clickhouse: ready'
/checkpoint ready
/exec run --container app --user clickhouse -c gdb -batch -ex "handle SIGUSR1 SIGUSR2 nostop noprint" -ex "break DB::executeQuery" -ex continue -ex "bt 3" -p $(pidof clickhouse-server)

Use --user to run GDB as the user of the process it attaches to. A container has no CAP_SYS_PTRACE by default, so root in it cannot attach to another user’s process.

Keeping GDB attached

-batch runs a list of commands chosen in advance. To decide the next command from what GDB just printed, keep GDB attached and give it its commands through a named pipe in the container.

This works on the server’s own image with GDB added, the example’s server image: ClickHouse 26.8 with no debug information. GDB finds functions by name in the program’s symbol table, and attaches in about 16 seconds. With the debug information of the debug image it takes minutes instead.

Make the pipe, keep it open, and start GDB with its input on it. exec start returns at once and prints the id of the process it started:

1
2
3
machine exec ch run   --container app --user clickhouse -- mkfifo /tmp/gdb.in
machine exec ch start --container app --user clickhouse -c 'sleep infinity > /tmp/gdb.in'
machine exec ch start --container app --user clickhouse -c 'gdb -q -p $(pidof clickhouse-server) < /tmp/gdb.in'
1
2
✔ Sent 3615877d to ch (whvR8O5Y1otCgBTbNi3rc)
Read the process machine exec ch proc 3615877d

The sleep holds the pipe open. Without it, GDB reads the end of its input after the first command and exits.

Every step below is the same three commands. exec start queues a printf that writes the step’s GDB commands to the pipe, the last of them an echo of a marker. run --until runs the machine until GDB prints that marker, which is when GDB has done every command before it. proc reads what GDB printed, by the id exec start gave:

1
2
3
machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "handle SIGUSR1 SIGUSR2 nostop noprint" "echo step-1\n" > /tmp/gdb.in'
machine run ch --until '/output/exec/3615877d/stdout~step-1' --within 60s
machine exec ch proc 3615877d stdout
1
2
3
4
5
6
7
8
9
10
11
Attaching to process 7
[New LWP 78]
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f84884760d7 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) Signal Stop Print Pass to program Description
SIGUSR1 No No Yes User defined signal 1
SIGUSR2 No No Yes User defined signal 2
(gdb) step-1
(gdb)

The handle line keeps continue from stopping at ClickHouse’s own signals, as in the batch session. This first step waits for GDB to read the program’s symbols, 10 seconds here. The others take a second or two.

A few things keep the steps right:

  • Each step has a marker of its own. run --until ends on text the run prints from its start, so step-1 printed before never ends a later run.
  • Use exec start, not exec run, to write to the pipe. exec run runs the machine until the printf exits, and GDB may print the marker during it. The run --until after it then waits for a marker that is already there.
  • printf "%s\n" writes each argument as a line and leaves \n in it as it is, so GDB’s echo ends the marker’s line. A $ in a GDB command is written \$ inside the double quotes, as "x/s \$rsi" below.
  • --within 60s bounds a step whose marker never comes. Without it a mistyped marker runs the machine until it is cancelled.

Each read shows the whole session so far. The output below is what each step added.

A breakpoint. GDB writes an int3 into the program, as on any host, and uses no debug register:

1
2
3
machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "break DB::executeQuery" "info breakpoints" "echo step-2\n" > /tmp/gdb.in'
machine run ch --until '/output/exec/3615877d/stdout~step-2' --within 60s
machine exec ch proc 3615877d stdout
1
2
3
4
5
6
7
(gdb) Breakpoint 1 at 0x1d386340 (3 locations)
(gdb) Num Type Disp Enb Address What
1 breakpoint keep y <MULTIPLE>
1.1 y 0x000000001d386340 <DB::executeQuery(std::__1::basic_string_view<char, ...>, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum)>
1.2 y 0x000000001d394300 <DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, ...)>
1.3 y 0x000000001d394640 <DB::executeQuery(std::__1::unique_ptr<DB::ReadBuffer, ...>, DB::WriteBuffer&, ...)>
(gdb) step-2

Continue to it. The example’s server sends itself a query every second, and a thread that handles it stops. GDB reads the marker after c stops, so the marker follows the hit:

1
2
3
machine exec ch start --container app --user clickhouse -c 'printf "%s\n" c "echo step-3\n" > /tmp/gdb.in'
machine run ch --until '/output/exec/3615877d/stdout~step-3' --within 60s
machine exec ch proc 3615877d stdout
1
2
3
4
5
(gdb) Continuing.
[Switching to Thread 0x7f844cbde640 (LWP 89)]

Thread 9 "TCPHandler" hit Breakpoint 1, 0x000000001d386340 in DB::executeQuery(std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) ()
(gdb) step-3

Every later continue works the same way. Each hit prints a new hit Breakpoint line, so --until '/output/exec/3615877d/stdout~hit Breakpoint' stops at the next hit too.

Read the query. executeQuery returns its result by value, so rdi holds where to put it, and the query follows in rsi and rdx, a pointer and a length:

1
2
3
machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "info registers rsi rdx" "x/s \$rsi" "echo step-4\n" > /tmp/gdb.in'
machine run ch --until '/output/exec/3615877d/stdout~step-4' --within 60s
machine exec ch proc 3615877d stdout
1
2
3
4
(gdb) rsi            0x7f8482a32cc0      140207104142528
rdx 0x28 40
(gdb) 0x7f8482a32cc0: "SELECT sum(number) FROM numbers(1000000)"
(gdb) step-4

Instructions and single steps:

1
2
3
machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "x/4i \$pc" stepi "nexti 3" "echo step-5\n" > /tmp/gdb.in'
machine run ch --until '/output/exec/3615877d/stdout~step-5' --within 60s
machine exec ch proc 3615877d stdout
1
2
3
4
5
6
7
(gdb) => 0x1d386340 <_ZN2DB12executeQuery...>:	jmp    0x1d38634b <_ZN2DB12executeQuery...+11>
0x1d386342 <_ZN2DB12executeQuery...+2>: nopw 0x200(%rax,%rax,1)
0x1d38634b <_ZN2DB12executeQuery...+11>: push %rbp
0x1d38634c <_ZN2DB12executeQuery...+12>: push %r15
(gdb) 0x000000001d38634b in DB::executeQuery(...) ()
(gdb) 0x000000001d386350 in DB::executeQuery(...) ()
(gdb) step-5

The stack, and running to the return:

1
2
3
machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "bt 3" finish "echo step-6\n" > /tmp/gdb.in'
machine run ch --until '/output/exec/3615877d/stdout~step-6' --within 60s
machine exec ch proc 3615877d stdout
1
2
3
4
5
6
7
(gdb) #0  0x000000001d386350 in DB::executeQuery(...) ()
#1 0x000000001ed12ec3 in DB::TCPHandler::runImpl() ()
#2 0x000000001ed37d64 in DB::TCPHandler::run() ()
(More stack frames follow...)
(gdb) Run till exit from #0 0x000000001d386350 in DB::executeQuery(...) ()
0x000000001ed12ec3 in DB::TCPHandler::runImpl() ()
(gdb) step-6

A hardware breakpoint. Delete the first breakpoint, and put one at the address finish returned to. It takes one of the run’s 3 debug registers, and stops where the next query returns:

1
2
3
machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "delete 1" "hbreak *0x1ed12ec3" c "echo step-7\n" > /tmp/gdb.in'
machine run ch --until '/output/exec/3615877d/stdout~step-7' --within 60s
machine exec ch proc 3615877d stdout
1
2
3
4
5
(gdb) (gdb) Hardware assisted breakpoint 2 at 0x1ed12ec3
(gdb) Continuing.

Thread 9 "TCPHandler" hit Breakpoint 2, 0x000000001ed12ec3 in DB::TCPHandler::runImpl() ()
(gdb) step-7

delete prints nothing, so its prompt stands alone at the start.

A hardware watchpoint. It takes another debug register, and stops after the next call writes its return address below the stack pointer:

1
2
3
machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "watch -l *(long *)(\$rsp - 8)" c "echo step-8\n" > /tmp/gdb.in'
machine run ch --until '/output/exec/3615877d/stdout~step-8' --within 60s
machine exec ch proc 3615877d stdout
1
2
3
4
5
6
7
8
9
(gdb) Hardware watchpoint 3: -location *(long *)($rsp - 8)
(gdb) Continuing.

Thread 9 "TCPHandler" hit Hardware watchpoint 3: -location *(long *)($rsp - 8)

Old value = 517025475
New value = 517025550
0x000000001b389000 in DB::BlockIO::operator=(DB::BlockIO&&) ()
(gdb) step-8

A system call. Delete every breakpoint and watchpoint, stop at the next sendto, and read its socket, its length and the first bytes it sends. GDB asks before it deletes them all, and answers itself, since its input is not a terminal:

1
2
3
machine exec ch start --container app --user clickhouse -c 'printf "%s\n" delete "catch syscall sendto" c "info registers rdi rdx" "x/16xb \$rsi" "echo step-9\n" > /tmp/gdb.in'
machine run ch --until '/output/exec/3615877d/stdout~step-9' --within 60s
machine exec ch proc 3615877d stdout
1
2
3
4
5
6
7
8
9
10
(gdb) Delete all breakpoints? (y or n) [answered Y; input not from terminal]
(gdb) Catchpoint 4 (syscall 'sendto' [44])
(gdb) Continuing.

Thread 9 "TCPHandler" hit Catchpoint 4 (call to syscall sendto), 0x00007f848850c98e in send () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) rdi 0x3a 58
rdx 0x22 34
(gdb) 0x7f8400e3c280: 0x01 0x00 0x01 0x00 0x02 0xff 0xff 0xff
0x7f8400e3c288: 0xff 0x03 0x00 0x00 0x01 0x00 0x0b 0x73
(gdb) step-9

That is the query’s result going back to the client, 34 bytes on socket 58.

Leave. Delete the catchpoint, detach and quit. GDB prints no marker after quit, so the run waits for the process’s status file to say it exited:

1
2
3
machine exec ch start --container app --user clickhouse -c 'printf "%s\n" delete detach quit > /tmp/gdb.in'
machine run ch --until '/output/exec/3615877d/status~^exited' --within 60s
machine exec ch proc 3615877d stdout
1
2
3
4
(gdb) Delete all breakpoints? (y or n) [answered Y; input not from terminal]
(gdb) Detaching from program: /usr/bin/clickhouse, process 7
[Inferior 1 (process 7) detached]
(gdb)

GDB has exited, and the server carries on.

The session is recorded with the run. Rewind to the checkpoint taken before it, and the same commands give the same answers.

From the checkpoint

The checkpoint holds the machine as it was before GDB attached. Rewind to it, and the next session starts from the same instruction:

1
machine rewind ch to ^ready

Or fork a branch at it and keep both:

1
machine fork ch --at ^ready --name again

Reading the disk

You can read files from a run’s disk without touching the run. machine fs reads the filesystem on the disk as a checkpoint saved it, so it never changes how the machine executes.

1
2
3
machine fs ls app /var/log --at ^done
machine fs cat app /var/log/app.log --at ^done
machine fs cp -r app /results ./results --at ^done

Every read names its checkpoint with --at, as a point: ^booted for a checkpoint by name, or '#4200' for the checkpoint at an entry. There is no default. This is not a view of the running machine. A listing that did not say how old it is would look like one.

1
2
3
4
5
machine fs ls app /tmp
✗ Checkpoint required: the disk is read where a checkpoint saved it.
List the checkpoints machine checkpoints app
Or take one machine checkpoint app
Then name it --at ^<name>

A point with no checkpoint at it is refused, and the refusal names the checkpoint before it.

Unsynced writes

Data the machine has not synced to its disk is not visible. A file a program wrote a moment before the checkpoint can be missing, short or empty, because the machine still holds it in memory.

To read the files as they are, sync in the machine, take a checkpoint, and read at now:

1
2
3
machine exec app run -- sync
machine checkpoint app
machine fs cat app /var/log/app.log --at now

now is the checkpoint the run stands at. It names one until the machine runs again; after that, name the checkpoint itself.

A run created with --fs-sync writes every change of a file to the disk as it is made, and one created with --fs-dirsync does so for every change of a directory. Both are options of machine create and are off by default.

To read the files of the machine as it runs, execute a command in it: machine exec app run -- ls /tmp. That reads now, synced or not, and is recorded in the run.

At the control prompt the command is /fs, and the point follows the word at:

1
2
/fs ls /var/log at ^done
/exec run sync;/checkpoint;/fs cat /var/log/app.log at now

From Python, run.fs.ls("/var/log", at="^done") lists a directory, run.fs.cat("/var/log/app.log", at="^done") returns a file’s bytes, and run.fs.cp("/var/log", "./logs", at="^done", recursive=True) copies it out.

Account and billing

Most of what the account page on the website does, the client does too.

Command Does
machine account show shows your account
machine account provision sets up account storage ahead of time; the first session or disk build does it otherwise
machine account credits shows your credits
machine account activity lists what they were spent on
machine account products list lists what is for sale
machine account buy-usage opens checkout for a pack of usage credits, as listed by products
machine account buy-plan subscribes to a plan, or switches tier if you already have a paid one
machine account upgrade-plan, machine account downgrade-plan moves you to a higher or lower plan
machine account cancel cancels the subscription: the plan stays until it renews, then drops to Free, and your usage credits stay
machine account invoices lists your invoices
machine account orders lists your purchases, and resumes or cancels a pending checkout
machine account payment updates your payment method
machine account portal opens your billing settings
machine account api-keys creates, lists, deletes and sets the expiry of API keys
machine account devices lists where you are signed in (--revoke signs one out)
machine account email changes your email address
machine account close closes the account, in two steps (--cancel stops it)

payment, portal, cancel, upgrade and downgrade open a browser. Add --link to print the address instead.

Sessions use credits. Machine writes an amount of credits with the sign ₡, so ₡0.15 is 0.15 credits. The price is per vCPU-hour, and depends on the hardware class. A session is created on standard-ip unless you pick another class, and it uses ₡0.15 a vCPU-hour. standard-ie uses ₡0.12. Each vCPU comes with 2G of memory and 16G of disk. A session that asks for more is charged for the vCPUs that would bring it. machine sessions classes lists the current price of each class. Your plan includes plan credits every month, and you buy usage credits at €1 a credit. A credit has no cash value.

Pricing lists the plans. For dedicated machines, reserved for your team alone, contact sales@testflows.com.

A running session is charged an hour ahead. When it stops, the unused part of its last hour is settled back. In the activity, a Charged row’s TIME is the hour and a Settled row’s is negative, the part given back, so a session’s rows add up to the time it ran, as their CHANGE adds up to what it used. machine account activity <session> lists a session’s charges and what they add up to. How credits are granted and used is in the usage credit terms.

Runs, logs and disks take up space in your account. machine storage show says how much, and machine storage prune frees the space that deleted runs and disks left behind. A prune does nothing while that space is less than 5% of your storage. machine storage prune --status shows the current or last prune.

machine storage check reads everything in your account’s storage back and reports what it could not read, showing its progress as it goes. It writes nothing.

Your plan’s storage quota counts TOTAL: what your runs and disks hold, and the space deleted ones have not given back yet. machine account show shows the same figure. Deleting a run or a disk moves its space to PRUNABLE, which still counts until you prune. The figures are from the last measurement, and machine storage --refresh measures again.

At the quota, a session starts no machine and creates no branch: machine create, machine start, machine fork, machine go, machine detach and machine disks build are refused. A machine that is already running keeps recording and saving. To get under the quota, delete runs or disks, stop your sessions, and prune.

Scripting

Everything the client does can be done from a script.

  • -o json prints machine-readable output. -o raw prints text without formatting values. Put the flag before the command, like machine -o json runs.
  • -q turns off progress output and spinners.
  • --timeout s gives up after that many seconds. It goes before the command too. The default is no limit, and 0 means check once.
  • --no-colors turns off colors.
  • A command exits with 2 when it fails or is refused, and with 1 when it cannot reach the service or a session. A waiting command exits with 3 on a timeout. Ctrl-C exits with 130.
1
machine -q --timeout 300 run app --until tasks || echo "boot did not finish"

Commands

machine --help lists these commands, in this order. The options apply to every command. Add --help after a command to list its own options. Where a command takes further commands, those follow it, so each one has a heading.

Options

Option Does
-o, --format Output format: text, raw, or json. The default is text. JSON carries every field.
--version Show version and license.
--no-colors Disable terminal color highlighting.
--signatures Name functions by their whole signature, parameters included. Without it a C++ function is named short, as GDB names it: DB::executeQuery.
--debug Show the messages exchanged with the service.
--api-url Cloud API base URL. Overrides TESTFLOWS_MACHINE_API_URL and the default.
-s, --session Target session, by name, id, or unique prefix. Goes before the command, as every option here does. Commands that need no session ignore it.
--timeout Give up after S seconds, for commands that wait. The default is no limit. 0 means check once.
-q, --quiet No progress or spinners.

machine login

Sign in. See Signing up.

A sign-in code is emailed. Use the account signup command if there is no account yet.

Or set TESTFLOWS_MACHINE_API_KEY instead.

1
machine login [-h]

Examples:

1
machine login  sign in

See also:

machine sessions

List and manage sessions. See Sessions.

A session is a reserved machine in the cloud. It holds CPUs, memory and disk space for runs. A run is created, started and stored in a session.

The create subcommand creates a session and uses it, so later commands in this terminal go to it. The use subcommand switches to another session.

Use -s to select a session for a single command. A session bills while it runs.

1
machine sessions [-h] [-w] command ...
Option Does
-w, --watch Watch for changes

Examples:

1
2
3
4
machine sessions                 list sessions
machine sessions create work create a session and use it
machine sessions show work details of work
machine sessions delete work -y stop it and delete it

See also:

machine sessions list

List sessions. See Sessions.

The current session is tagged. A session that stopped shows how it stopped. Use --all to list the stopped sessions of the account as well.

1
machine sessions list [-h] [-w] [-a]
Option Does
-w, --watch Watch for changes
-a, --all List stopped sessions as well, the newest 50

Examples:

1
2
3
machine sessions list        list sessions
machine sessions list --all with the stopped ones, and how each stopped
machine sessions list -w watch the list

See also:

machine sessions create

Create a session and use it. See Sessions.

The command uses the new session in this terminal, as sessions use does. Use --no-use to keep using the session in use before.

The command waits until the session is running. Use --no-wait to return once the request is accepted. A session created with --no-wait is not used.

1
2
3
machine sessions create [-h] [--cpus n] [--mem size] [--storage size] [--class type]
[--no-wait] [--no-use]
[name]
Argument Does
[name] Session name (default: auto-generated)
Option Does
--cpus n Number of vCPUs (default: 1)
--mem size Memory, MB or with a K/M/G/T suffix (default: the class’s memory for each vCPU, as sessions classes lists it)
--storage size The session’s storage, GB or with a K/M/G/T suffix (default: the class’s storage for the session’s size, up to the plan’s limit)
--class type Hardware class, as sessions classes lists it (default: the plan’s class, which sessions classes tags)
--no-wait Return without waiting
--no-use Create the session without using it

Examples:

1
2
3
4
machine sessions create work            create the session work and use it
machine sessions create work --cpus 4 with 4 vCPUs
machine sessions create work --no-use keep the session in use
machine sessions create work --no-wait return once accepted

Refuses, and the way past:

1
2
a class with no capacity now        machine sessions classes
a class with no machine in service machine sessions classes

See also:

machine sessions use

Make a session the default for later commands. See Sessions.

The command connects first if needed. It checks that the session responds before switching. With --no-wait it fails if a connect is needed and the session is not running yet.

1
machine sessions use [-h] [--no-wait] [session]
Argument Does
[session] Session name, id, or prefix (default: the only one)
Option Does
--no-wait Fail if a connect is needed and the session is not running yet

Examples:

1
2
machine sessions use work  use work in this terminal
machine sessions use use the only session

See also:

machine sessions show

Show session details. See Sessions.

1
machine sessions show [-h] [session]
Argument Does
[session] Session name, id, or prefix (default: the only one)

Examples:

1
machine sessions show work  details of work

See also:

machine sessions classes

List the hardware classes a session can be created on.

Each class names its region and its CPU architecture. A session runs in its class’s region, and its runs are stored there. A run starts only in a session of the architecture it was recorded on. Use the class name with create --class.

A session created without --class runs on the plan’s default class. The list tags that class default.

1
machine sessions classes [-h]

Examples:

1
2
machine sessions classes          list the classes
machine -o json sessions classes with every field, for a script

See also:

machine sessions delete

Delete a session. See Sessions.

The first operand names the session. Use -a to delete every session.

The command stops the machines of the session and syncs their runs to shared storage. The runs start again in another session. A session already gone counts as success. Its connection on this computer is cleared.

The command waits until a named session is gone. Use --no-wait to return once the request is accepted.

1
machine sessions delete [-h] [-a] [-y] [--no-wait] [session]
Argument Does
[session] Session name, id, or prefix (default: the only one)
Option Does
-a, --all Delete every session
-y, --yes Skip the confirmation prompt
--no-wait Return without waiting

Examples:

1
2
machine sessions delete work -y  stop and delete work
machine sessions delete -a -y delete every session

See also:

machine sessions rename

Rename a session. See Sessions.

Only the name changes. The session keeps running.

1
machine sessions rename [-h] session new-name
Argument Does
session Session name, id, or prefix
new-name New session name

Examples:

1
machine sessions rename work build  rename work to build

See also:

machine sessions keep

Keep a session running while it is idle.

A session is idle while it serves no command. An idle session stops after the plan’s idle time. Keep holds it for the duration, up to the plan’s limit. A later keep replaces the hold.

1
machine sessions keep [-h] session duration
Argument Does
session Session name, id, or prefix
duration How long to keep it, such as 30m or 1h

Examples:

1
2
machine sessions keep work 30m  keep work running idle for 30 minutes
machine sessions keep work 1h keep it for an hour

See also:

machine sessions connect

Connect to a session. See Sessions.

The command connects this computer to the session and uses it in this terminal. The use subcommand does the same and also checks a connection already open. Use sessions use instead.

The command waits until the session is running. Use --no-wait to return once the request is accepted.

1
machine sessions connect [-h] [--no-wait] [session]
Argument Does
[session] Session name, id, or prefix (default: the only one)
Option Does
--no-wait Fail if the session is not running yet

Examples:

1
machine sessions connect work  connect to work

See also:

machine sessions disconnect

Disconnect from a session. See Sessions.

The session keeps running. The connection is shared on this computer, so every terminal loses it. Use -a to disconnect every session.

1
machine sessions disconnect [-h] [-a] [session]
Argument Does
[session] Session (default: the current one)
Option Does
-a, --all Disconnect every session

Examples:

1
2
machine sessions disconnect work  disconnect from work
machine sessions disconnect -a disconnect every session

See also:

machine ping

Check reachability and latency. See The client.

The command pings the API and the connected sessions. It reports the state, round-trip time, jitter, setup time and certificate lifetime of each target.

1
machine ping [-h] [-c n]
Option Does
-c, --count n Round-trips per target (default: 3)

Examples:

1
2
machine ping       reachability and latency of the API and sessions
machine ping -c 5 five round-trips per target

Prints:

1
text  TARGET STATE RTT JITTER SETUP CERT LEFT

See also:

machine account

Show and manage the account. See Account and billing.

The account holds the plan, the credits, the API keys and the signed-in devices. The subcommands buy credits or a plan, manage invoices and payment, and close the account.

1
machine account [-h] command ...

Examples:

1
2
3
4
machine account           plan, credits and quotas
machine account signup create an account
machine account credits credit balance
machine account activity charges and credits

See also:

machine account devices

List signed-in devices. See Account and billing.

Use --revoke to sign out one device by id.

1
machine account devices [-h] [--revoke id]
Option Does
--revoke id Sign out one device by its id (or a unique id prefix)

Examples:

1
2
machine account devices             signed-in devices
machine account devices --revoke 3 sign out device 3

See also:

machine account buy-usage

Buy usage credits. See Account and billing.

The command opens checkout for the pack of the named number of credits. A credit costs one euro. A credit has no cash value.

1
machine account buy-usage [-h] [-y] credits
Argument Does
credits Credits to buy, one of the amounts account products list shows. A credit costs one euro
Option Does
-y, --yes Skip the confirmation prompt

Examples:

1
2
machine account buy-usage 10     buy 10 usage credits for €10
machine account buy-usage 10 -y without asking

See also:

machine account buy-plan

Subscribe to a plan. See Account and billing.

On a paid plan, the command upgrades or downgrades to the named tier instead.

1
machine account buy-plan [-h] [-y] plan
Argument Does
plan Plan to subscribe to: starter, plus or pro
Option Does
-y, --yes Skip the confirmation prompt

Examples:

1
2
machine account buy-plan starter     subscribe to starter
machine account buy-plan starter -y without asking

See also:

machine account upgrade-plan

Upgrade to a higher plan. See Account and billing.

An omitted tier is the next one up. Use --link to print the URL without opening a browser.

1
machine account upgrade-plan [-h] [--link] [-y] [plan]
Argument Does
[plan] Plan to move to (default: the next plan up)
Option Does
--link Print the link only (don’t open a browser)
-y, --yes Skip the confirmation prompt

Examples:

1
2
machine account upgrade-plan      upgrade to the next plan
machine account upgrade-plan pro upgrade to pro

See also:

machine account downgrade-plan

Downgrade to a lower plan. See Account and billing.

An omitted tier is the next one down. Use --link to print the URL without opening a browser.

1
machine account downgrade-plan [-h] [--link] [-y] [plan]
Argument Does
[plan] Plan to move to (default: the next plan down)
Option Does
--link Print the link only (don’t open a browser)
-y, --yes Skip the confirmation prompt

Examples:

1
2
machine account downgrade-plan          downgrade to the previous plan
machine account downgrade-plan starter downgrade to starter

See also:

machine account payment

Update the payment method. See Account and billing.

Use --link to print the URL without opening a browser.

1
machine account payment [-h] [--link]
Option Does
--link Print the link only (don’t open a browser)

Examples:

1
2
machine account payment         update the payment method
machine account payment --link print the link only

See also:

machine account portal

Open billing settings. See Account and billing.

Use --link to print the URL without opening a browser.

1
machine account portal [-h] [--link]
Option Does
--link Print the link only (don’t open a browser)

Examples:

1
2
machine account portal         open billing settings
machine account portal --link print the link only

See also:

machine account invoices

List and manage invoices. See Account and billing.

The listing shows the newest 25. Use --limit for a different page size.

1
2
3
4
machine account invoices [-h] [--since date|age] [--until date|age]
[--status {draft,open,paid,uncollectible,void}] [--limit n]
[--sort field] [--reverse]
command ...
Option Does
--since date|age Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--until date|age Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--status {draft,open,paid,uncollectible,void} Filter by invoice status
--limit n Cap to most recent n (default: 25, max 100)
--sort field Sort by created or amount
--reverse Reverse sort order

Examples:

1
2
machine account invoices                list invoices
machine account invoices --status paid paid invoices only

See also:

machine account invoices list

List invoices.

The listing shows the newest 25. Use --limit for a different page size.

1
2
3
machine account invoices list [-h] [--since date|age] [--until date|age]
[--status {draft,open,paid,uncollectible,void}] [--limit n]
[--sort field] [--reverse]
Option Does
--since date|age Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--until date|age Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--status {draft,open,paid,uncollectible,void} Filter by invoice status
--limit n Cap to most recent n (default: 25, max 100)
--sort field Sort by created or amount
--reverse Reverse sort order

Examples:

1
2
machine account invoices list              list invoices
machine account invoices list --since 30d the last 30 days

See also:

machine account invoices download

Open an invoice.

The hosted page has the PDF and receipt. Use --link to print the URL without opening a browser.

1
machine account invoices download [-h] [--link] invoice-id
Argument Does
invoice-id Invoice id (in_…)
Option Does
--link Print the link only (don’t open a browser)

Examples:

1
machine account invoices download in_123  open invoice in_123

See also:

machine account cancel

Cancel the subscription. See Account and billing.

The plan stays until renewal, then drops to Free. Usage credits stay. Use --link to print the URL without opening a browser.

1
machine account cancel [-h] [--link]
Option Does
--link Print the link only (don’t open a browser)

Examples:

1
2
machine account cancel         cancel the subscription
machine account cancel --link print the link only

See also:

machine account email

Change the account email. See Account and billing.

A code goes to the current inbox. A link goes to the new one.

1
machine account email [-h] [-y] email
Argument Does
email New email address
Option Does
-y, --yes Skip confirmation prompts

Examples:

1
machine account email me@example.com  change the email

See also:

machine account products

List products for sale. See Account and billing.

1
machine account products [-h] command ...

Examples:

1
machine account products list  products for sale

See also:

machine account products list

List products for sale.

1
machine account products list [-h]

Examples:

1
machine account products list  products for sale

See also:

machine account show

Show the account. See Account and billing.

1
machine account show [-h]

Examples:

1
machine account show  plan, credits and quotas

See also:

machine account credits

Show the account’s credits. See Account and billing.

1
machine account credits [-h]

Examples:

1
machine account credits  the credit balance

See also:

machine account activity

Show credit activity. See Account and billing.

A session is charged an hour ahead, as Charged rows. When it stops, a Settled row gives back the unused part of its last hour. Naming a session ends the listing with what the session used over all of its rows.

The listing shows the newest 25. Use --limit for a different page size.

1
2
3
4
5
machine account activity [-h] [--since date|age] [--until date|age] [--period date]
[--type {granted,purchased,charged,settled,expired,removed,adjusted}]
[--credit-type {plan,usage}] [--limit n] [--sort field]
[--reverse]
[session]
Argument Does
[session] Show one session’s transactions, by name, id or id prefix
Option Does
--since date|age Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--until date|age Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--period date Show the plan period that contains this date, such as 2026-10-01
--type {granted,purchased,charged,settled,expired,removed,adjusted} Filter by activity type
--credit-type {plan,usage} Credit type to filter by: plan or usage
--limit n Cap to most recent n (default: 25, max 500)
--sort field Sort by sequence or amount
--reverse Reverse sort order

Examples:

1
2
3
machine account activity             charges and credits
machine account activity idle-check one session's bill
machine account activity --since 7d the last 7 days

See also:

machine account orders

List and manage orders. See Account and billing.

The listing shows the newest 25. Use --limit for a different page size.

1
2
3
4
machine account orders [-h]
[--status {pending,processing,completed,cancelled,expired,failed}]
[--limit n]
command ...
Option Does
--status {pending,processing,completed,cancelled,expired,failed} Filter by status
--limit n Cap to most recent n (default: 25, max 200)

Examples:

1
2
3
machine account orders list    list orders
machine account orders resume resume the pending order
machine account orders cancel cancel the pending order

See also:

machine account orders list

List orders.

The listing shows the newest 25. Use --limit for a different page size.

1
2
3
machine account orders list [-h]
[--status {pending,processing,completed,cancelled,expired,failed}]
[--limit n]
Option Does
--status {pending,processing,completed,cancelled,expired,failed} Filter by status
--limit n Cap to most recent n (default: 25, max 200)

Examples:

1
2
machine account orders list                   list orders
machine account orders list --status pending pending orders

See also:

machine account orders resume

Resume a pending checkout.

The command resumes the pending order. An account has one at most. Use --link to print the URL without opening a browser.

1
machine account orders resume [-h] [--link]
Option Does
--link Print the link only (don’t open a browser)

Examples:

1
2
machine account orders resume         resume the pending order
machine account orders resume --link print the link only

See also:

machine account orders cancel

Cancel a pending order so another buy can start.

The command cancels the pending order. An account has one at most.

1
machine account orders cancel [-h]

Examples:

1
machine account orders cancel  cancel the pending order

See also:

machine account api-keys

List and manage API keys. See Account and billing.

A key controls runs, disks and sessions. It cannot change billing or the account.

Set TESTFLOWS_MACHINE_API_KEY to use one.

1
machine account api-keys [-h] command ...

Examples:

1
2
machine account api-keys list       list API keys
machine account api-keys create ci create the key ci

See also:

machine account api-keys list

List API keys.

1
machine account api-keys list [-h]

Examples:

1
machine account api-keys list  list API keys

See also:

machine account api-keys create

Create an API key.

The secret is shown once. A sign-in code is required.

1
machine account api-keys create [-h] [--expiry days|never] name
Argument Does
name Label to recognize this key later
Option Does
--expiry days|never Expire after N days (e.g. 30), or “never” (default: never)

Examples:

1
2
machine account api-keys create ci              create the key ci
machine account api-keys create ci --expiry 30 expire in 30 days

See also:

machine account api-keys delete

Delete an API key.

A sign-in code is required.

1
machine account api-keys delete [-h] id
Argument Does
id Key id, as account api-keys list shows it

Examples:

1
machine account api-keys delete 3  delete key 3

See also:

machine account api-keys update

Set the expiry of an API key.

A sign-in code is required.

1
machine account api-keys update [-h] id days|never
Argument Does
id Key id, as account api-keys list shows it
days|never Number of days from now (e.g. 30) or “never”

Examples:

1
2
machine account api-keys update 3 30     expire key 3 in 30 days
machine account api-keys update 3 never never expire key 3

See also:

machine account provision

Set up the account’s shared storage. See Signing up.

The first session create or disk build sets up storage itself. This command sets it up ahead of them. The command is safe to repeat while it is still working.

1
machine account provision [-h]

Examples:

1
machine account provision  set up shared storage

See also:

machine account signup

Create an account. See Signing up.

The command shows the Terms of Service and the Client EULA in full and asks for their acceptance. The account is made accepting them. A link is emailed to finish signing up.

1
machine account signup [-h]

Examples:

1
machine account signup  create an account

See also:

machine account close

Close the account. See Account and billing.

The command emails a code that starts the close. It waits until the machines of every session stop. It then emails a second code that confirms the close.

A close that is not confirmed stays in progress. Run the command again to finish it. Use --cancel to stop it.

1
machine account close [-h] [-y] [--cancel]
Option Does
-y, --yes Skip confirmation prompts
--cancel Abort closing and return the account to active

Examples:

1
2
machine account close -y        close the account
machine account close --cancel cancel a pending close

See also:

machine logout

Sign out. See Signing up.

The command revokes the access token. Use --everywhere to sign out of every device and browser.

1
machine logout [-h] [--everywhere]
Option Does
--everywhere Sign out of every device and browser, not just this one

Examples:

1
2
machine logout               sign out
machine logout --everywhere sign out of every device

See also:

machine upgrade

Install the latest client. See Download.

The command runs the installer at https://testflows.com/machine/install. The installer downloads the release and checks its signed checksum. It puts the client where it puts every install. It removes the files older versions unpacked.

The installer uses the client’s own sign-in. A client signed in asks for nothing and stays signed in. A client signed out signs in at the terminal for the download. It stays signed out after.

The latest is not downloaded again when it is installed already. A version named is installed even when it is installed already. An older version installs the same way. Use --list to see the versions there are.

1
machine upgrade [-h] [--list | version]
Argument Does
[version] Version to install, YYYYMMDD-HHMM (default: the latest)
Option Does
--list List the versions there are

Examples:

1
2
3
machine upgrade                install the latest client
machine upgrade --list the versions there are
machine upgrade 20261106-1233 install that version

See also:

machine completion

Print the shell completion script. See The client.

The script completes commands, options and their values as the client reads them. The client answers each Tab itself, so a newer client needs no newer script.

Add the line for the shell to its startup file:

1
2
3
bash  eval "$(machine completion bash)"   in ~/.bashrc
zsh eval "$(machine completion zsh)" in ~/.zshrc
fish machine completion fish | source in ~/.config/fish/config.fish

Run the same line in a shell to complete in that shell alone. The completion lasts until the shell exits.

machine-env loads the script itself. On macOS, machine outside machine-env is the system’s command. There the line belongs in no startup file.

1
machine completion [-h] {bash,zsh,fish}
Argument Does
{bash,zsh,fish} Shell to print the script for: bash, zsh or fish

Examples:

1
2
3
machine completion bash  the script for bash
machine completion zsh the script for zsh
machine completion fish the script for fish

See also:

machine disks

List and manage disks. See Disks.

A disk is the filesystem image that a run is created from. Build one from a static binary, a Docker image or a Compose project.

The listing shows the newest 25. Use --limit for a different page size.

1
2
3
machine disks [-h] [-q] [--no-trunc] [--deleted] [--label name=value] [--since date|age]
[--limit n] [--offset n] [-w]
command ...
Option Does
-q, --quiet Only show ids
--no-trunc Full disk id
--deleted Show deleted disks
--label name=value Filter by a label (repeatable; matches all)
--since date|age Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--limit n Show at most n disks (default: 25)
--offset n Skip the first n disks
-w, --watch Watch for changes

Examples:

1
2
3
4
machine disks                              list disks
machine disks build --binary ./prog hello build a disk from a program
machine disks show hello details of hello
machine disks transfers uploads in progress

Refuses, and the way past:

1
a disk with runs  machine disks delete hello -f -y

See also:

machine disks list

List disks. See Disks.

The listing shows the newest 25. Use --limit for a different page size.

1
2
machine disks list [-h] [-q] [--no-trunc] [--deleted] [--label name=value]
[--since date|age] [--limit n] [--offset n] [-w]
Option Does
-q, --quiet Only show ids
--no-trunc Full disk id
--deleted Show deleted disks
--label name=value Filter by a label (repeatable; matches all)
--since date|age Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--limit n Show at most n disks (default: 25)
--offset n Skip the first n disks
-w, --watch Watch for changes

Examples:

1
2
3
machine disks list                 list disks
machine disks list --label team=a disks with a label
machine disks list -w watch the list

See also:

machine disks build

Build a disk from a binary, image or Compose project. See Disks.

Exactly one source is required. --binary wraps a static executable in an image built around it. --from names a public image that the build pulls. No Docker is needed for --from.

--image takes a Docker image as a save tar or a name in the local Docker. --compose takes a project directory and the images its services name.

A binary or an image runs as one container. Use --entrypoint to set the executable. Pass arguments after –.

Use --env to set a variable in the environment of the container. The disk holds each --env value as written.

A project runs every service in it. It takes no --entrypoint, no --env and no arguments after –.

With --from or --binary, --add puts a file or a directory in the image, as one more layer. Without a destination the path lands in the working directory of the image, under its own name. Write path:dest to name another place. A dest ending in / is a directory.

The name defaults to what the source is called. The size defaults to the size of the images once loaded, plus 1 GB, rounded up to a whole GB. Use --dry-run to see those numbers without building anything.

The build writes the images into the Docker store of the disk. A machine created from the disk has every image present and loads nothing. Use --load-at-boot to keep them on the disk as save tars. Docker then loads them each time a machine starts from the disk.

A guest that runs docker save or docker push on one of the images needs the save tars. Such a disk holds each image twice and is sized for both.

1
2
3
4
machine disks build [-h] (--binary path | --image ref | --from ref | --compose dir)
[--size size] [--dry-run] [--load-at-boot] [--entrypoint path]
[--add path[:dest]] [--env name=value] [--label name=value]
[name] [-- args ...]
Argument Does
[name] Disk name
[-- args ...] Arguments after – become the service’s command
Option Does
--binary path Static x86_64 executable to wrap
--image ref Docker image, a save tar or a name
--from ref Public image, pulled by the build
--compose dir Compose project directory
--size size Disk size, MB or with a K/M/G/T suffix (default: the loaded images plus 1 GB, rounded up to a GB)
--dry-run Report what the disk would hold and build nothing
--load-at-boot Keep the images as tars on the disk and load them each time a machine starts from the disk
--entrypoint path Executable the service runs
--add path[:dest] File or directory to put in the image, repeatable; dest defaults to the image’s working directory
--env name=value Environment variable the service runs with, repeatable
--label name=value Attach a label, repeatable; an empty value removes one the name carried

Examples:

1
2
3
4
machine disks build --binary ./prog hello            from a binary
machine disks build --image app:1 hello from a Docker image
machine disks build --compose ./app app from a Compose project
machine disks build --binary ./prog hello --dry-run show name and size

See also:

machine disks show

Show disk details. See Disks.

1
machine disks show [-h] name
Argument Does
name Disk name

Examples:

1
machine disks show hello  details of hello

See also:

machine disks transfers

Show disk transfers. See Disks.

The listing shows the active transfers by default. Use --since to include history. Use --since with --active for transfers in flight that started in that window.

1
machine disks transfers [-h] [--active] [--since date|age] [-w]
Option Does
--active Only in-flight transfers
--since date|age Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m
-w, --watch Watch for changes

Examples:

1
2
machine disks transfers           uploads and downloads
machine disks transfers --active those in progress

See also:

machine disks delete

Delete disks. See Disks.

Deletion is permanent. The name is free at once. The bytes count against the storage quota until a prune removes them.

A disk that a run was created from is refused. Use --force to delete it anyway. Those runs stay and can no longer start.

1
machine disks delete [-h] [-y] [-f] name [name ...]
Argument Does
name [name ...] Disk names
Option Does
-y, --yes Skip the confirmation prompt
-f, --force Delete a disk that runs were created from

Examples:

1
2
machine disks delete hello -y     delete hello
machine disks delete hello -f -y delete it though runs boot it

See also:

machine disks rename

Rename a disk. See Disks.

1
machine disks rename [-h] name new-name
Argument Does
name Current disk name
new-name New disk name

Examples:

1
machine disks rename hello hello2  rename hello to hello2

See also:

machine disks verify

Check a stored disk against its id. See Disks.

The command reads the disk back out of shared storage. It computes the id, which is what the disk is stored under. A disk that reads as stored is verified. A disk that does not is reported as damaged.

The check is a storage operation and takes about as long as reading the disk. It keeps running if the command is interrupted. Run machine storage ops to list it. One check runs at a time for an account.

The command waits for the result. Use --no-wait to return once the request is accepted.

1
machine disks verify [-h] [--no-wait] name
Argument Does
name Disk name
Option Does
--no-wait Return without waiting

Examples:

1
machine disks verify hello  check hello against its id

See also:

machine disks cancel

Cancel a transfer in progress. See Disks.

1
machine disks cancel [-h] name
Argument Does
name Disk name

Examples:

1
machine disks cancel hello  cancel the transfer of hello

See also:

machine preload

Fetch a run or a disk onto this session. See Stopping and starting.

The operand is any run or disk of the account. A run fetched here starts later without a fetch. A disk fetched here is ready before a run is created from it.

Use -a to fetch every run that this session’s runs branched from, and their disks. A copy already here is skipped. A session is required.

1
machine preload [-h] [-a] command ...
Option Does
-a, --all Everything this session’s runs need

Examples:

1
2
machine preload run run1    fetch run1 onto the session
machine preload disk hello fetch the disk hello onto it

See also:

machine preload run

Preload a run into a session. See Stopping and starting.

The command fetches the run from shared storage onto the session. The run is there before it is needed. A run already there is skipped. The disk of the run is fetched with it.

1
machine preload run [-h] [-a] [run]
Argument Does
[run] Run name, required without -a
Option Does
-a, --all Every run this session’s runs branched from

Examples:

1
2
machine preload run run1  fetch run1 onto this session
machine preload run -a fetch every run this session needs

See also:

machine preload disk

Preload a disk into a session. See Stopping and starting.

The command fetches the account disk onto the session. The disk is there before a start. A disk already there is skipped.

1
machine preload disk [-h] [-a] [disk]
Argument Does
[disk] Disk name or id
Option Does
-a, --all Every disk this session’s runs need

Examples:

1
2
machine preload disk hello  fetch the disk hello
machine preload disk -a fetch every disk this session needs

See also:

machine fs

List, copy or print files of a run’s disk.

The command reads the filesystem on the disk as a checkpoint saved it. It never reads the live machine. It needs no program in the machine and does not change how the machine executes.

Every read names its checkpoint with --at. A point that is not a checkpoint is refused, and the refusal names the checkpoint before it. The point now is the checkpoint the run stands at, until the machine runs again.

Data the machine has not synced to its disk is not visible. A file a program wrote a moment before the checkpoint can be missing, short or empty.

To read the files as they are, execute sync in the machine with exec. Then take a checkpoint and read with --at now. Use exec run with cat to read a file of the live machine.

A run created with --fs-sync writes every change of a file to the disk as it is made. A run created with --fs-dirsync does so for every change of a directory. Both are options of create and are off by default.

1
machine fs [-h] command ...

Examples:

1
2
3
4
5
machine fs ls run1 /etc --at ^cp1             list a directory at cp1
machine fs cat run1 /etc/hostname --at ^cp1 print a file
machine fs cp run1 /etc/hostname . --at ^cp1 copy a file out
machine fs ls run1 / --at '#300' at the checkpoint at entry 300
machine fs ls run1 / --at now at the checkpoint just taken

Refuses, and the way past:

1
2
a read with no checkpoint named   machine checkpoints run1
a point with no checkpoint at it machine checkpoint run1

See also:

machine fs ls

List files on a run’s disk at a checkpoint.

The files are read from the disk as a checkpoint saved it, not from the live machine. Use --at to name the checkpoint. Data the machine has not synced to its disk is not visible.

1
machine fs ls [-h] [--at point] [-l] [-a] run [path]
Argument Does
run Run name
[path] Path (default: /)
Option Does
--at point Required. The checkpoint to read the disk at: ^name, ‘#entry’ of a checkpoint, or now
-l Long format (mode, size, mtime, name)
-a, --all Include the current and parent directory entries

Examples:

1
2
machine fs ls run1 /etc --at ^cp1        list a directory at cp1
machine fs ls -l -a run1 /etc --at ^cp1 long form

Prints:

1
2
text  checkpoint <entry> path <path>, then a name a line
json checkpoint path entries

See also:

machine fs cp

Copy a file or directory from a run’s disk at a checkpoint.

The files are read from the disk as a checkpoint saved it, not from the live machine. Use --at to name the checkpoint. Data the machine has not synced to its disk is not visible. Use -r to copy a directory.

1
machine fs cp [-h] [--at point] [-r] run path [dest]
Argument Does
run Run name
path Path to copy
[dest] Local destination file or directory (default: .)
Option Does
--at point Required. The checkpoint to read the disk at: ^name, ‘#entry’ of a checkpoint, or now
-r, --recursive Copy directories recursively

Examples:

1
2
machine fs cp run1 /etc/hostname . --at ^cp1  copy a file out
machine fs cp -r run1 /etc ./etc --at ^cp1 copy a directory out

See also:

machine fs cat

Print a file from a run’s disk at a checkpoint.

The files are read from the disk as a checkpoint saved it, not from the live machine. Use --at to name the checkpoint. Data the machine has not synced to its disk is not visible.

1
machine fs cat [-h] [--at point] run path
Argument Does
run Run name
path File path
Option Does
--at point Required. The checkpoint to read the disk at: ^name, ‘#entry’ of a checkpoint, or now

Examples:

1
2
machine fs cat run1 /etc/hostname --at ^cp1    print a file
machine fs cat run1 /etc/hostname --at '#300' at entry 300

See also:

machine sync

Publish a run to shared storage. See Stopping and starting.

Any session can then start the run. The command runs in the session that holds the run.

1
machine sync [-h] run
Argument Does
run Run name

Examples:

1
machine sync run1  publish run1 to shared storage

See also:

machine offload

List or drop this session’s local copies. See Stopping and starting.

Offload alone lists the copies this session can drop. It counts the copies in use.

The run and disk subcommands drop one kind. Use -a to drop every run and disk not in use. The copy in shared storage stays.

Start or preload brings a run back. The next start fetches a disk again. A session is required.

1
machine offload [-h] [--runs] [--disks] [-a] command ...
Option Does
--runs Only session run folders
--disks Only session disk files
-a, --all Every run and disk not in use

Examples:

1
2
3
machine offload             list the copies that can be dropped
machine offload run run1 drop the session's copy of run1
machine offload disk hello drop its copy of the disk hello

See also:

machine offload list

List the copies that can be dropped. See Stopping and starting.

The listing shows the copies on this session that are not in use. The rest are counted below the table. Use --runs or --disks to show one kind. Omit both, or pass both, to show both.

1
machine offload list [-h] [--runs] [--disks]
Option Does
--runs Only session run folders
--disks Only session disk files

Examples:

1
2
machine offload list         copies that can be dropped
machine offload list --runs runs only

See also:

machine offload run

Offload a run from a session. See Stopping and starting.

The command drops the session’s copy of the run and frees its storage. The run itself stays in shared storage. Stop does not drop the copy.

Start or preload brings the run back. The command does not drop the disk of the run.

1
machine offload run [-h] [-a] [run]
Argument Does
[run] Run name
Option Does
-a, --all Every run not in use

Examples:

1
2
machine offload run run1  drop the copy of run1
machine offload run -a drop every run not in use

See also:

machine offload disk

Offload a disk from a session. See Stopping and starting.

The command drops the session’s copy of the disk. The account disk stays. A disk in use by a live machine is refused. The next start fetches the disk again.

1
machine offload disk [-h] [-a] [disk]
Argument Does
[disk] Disk name or id
Option Does
-a, --all Every disk not in use

Examples:

1
2
machine offload disk hello  drop the copy of the disk hello
machine offload disk -a drop every disk not in use

See also:

machine storage

Show and manage storage. See Account and billing.

Storage holds the runs and disks of the account. The subcommands report its use, list its operations, reclaim space and read every object back.

1
machine storage [-h] [--refresh] command ...
Option Does
--refresh Measure again instead of using the last measurement

Examples:

1
2
3
4
machine storage show   storage the account uses
machine storage ops storage operations
machine storage prune reclaim space held by deleted runs
machine storage check check every object can be read

See also:

machine storage show

Show the storage the account uses. See Account and billing.

1
machine storage show [-h] [--refresh]
Option Does
--refresh Measure again instead of using the last measurement

Examples:

1
2
machine storage show            storage the account uses
machine storage show --refresh measure again

See also:

machine storage ops

List operations on the account’s storage.

The listing shows the newest 25 operations in every state. Use --limit for a different page size. Use --state to show some states.

STATUS says why an operation failed. It also says how many objects a running check has read.

These operations act on the account’s storage, not on a run. The ops command does not list them. The command needs no session.

1
2
machine storage ops [-h] [--op-id id] [--state [^]state] [--type [^]type]
[--since date|age] [--until date|age] [--limit n] [--offset n] [-w]
Option Does
--op-id id Show one operation, by its id (ignores --state and --type)
--state [^]state Filter by state: pending, running, done, failed (default: all; repeat = OR; ^ excludes)
--type [^]type Filter by type: measure, prune, verify, check (default: all; repeat = OR; ^ excludes)
--since date|age Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--until date|age Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--limit n Show at most n operations (default: 25)
--offset n Skip the first n operations
-w, --watch Watch for changes

Examples:

1
2
machine storage ops                  the newest operations
machine storage ops --state running those still running

See also:

machine storage prune

Reclaim space held by deleted runs and disks. See Account and billing.

The command does nothing while unused data is below 5% of storage.

1
machine storage prune [-h] [-y] [--status]
Option Does
-y, --yes Do not ask for confirmation
--status Show the current or last prune and exit

Examples:

1
2
machine storage prune -y        reclaim space without asking
machine storage prune --status the current or last prune

See also:

machine storage check

Check that every stored object can be read. See Account and billing.

The command reads everything in the account’s storage back and reports what could not be read.

Nothing is written. The check can run while sessions and a prune run. An object deleted while the check runs is counted apart, not as unreadable.

The command waits for the check. It shows the objects and bytes read as it goes. Use --no-wait to return once the request is accepted.

The check is an operation. It goes on if the command is interrupted. Run machine storage ops to list it.

The command fails when an object could not be read. It names each one with the id of the failed request.

1
machine storage check [-h] [--status] [--no-wait]
Option Does
--status Show the current or last check and exit
--no-wait Return once the check is started

Examples:

1
2
machine storage check           read every stored object
machine storage check --status the current or last check

See also:

machine create

Create a run and start it. See Creating a run.

Options configure the machine: disk, memory, CPUs, devices and virtual time. The RNG seed and the virtual time base, rate and costs fix the sources of nondeterminism.

Use --auto-checkpoints to set how often checkpoints are taken. Use --like to inherit the options of another run. The disk is not inherited, so --disk is always given.

The machine waits for commands. Use --no-daemon to run it to completion instead.

The run keeps its execution options. Start and fork repeat them unless given others.

A source is a device the machine prints on, such as ttyS1, or a file under /output. Use --keep-output to store a source that would be dropped. Use --drop-output to store less. The run keeps both lists for its whole life, and the describe command shows them.

A run stores every source but the files under /output/drop. A file there has a name and no bytes. A program writes under /output/drop through MACHINE_DROP_OUTPUT_DIR. It writes under /output/keep through MACHINE_OUTPUT_DIR.

A name given to either option may be a pattern, such as /output/drop//.log. A star matches within one directory name, a leading dot included, and never a slash. Two stars match any number of directories.

A name given to --keep-output or --drop-output decides before the directory does. When a kept name and a dropped name cover one source, a name spelled out decides over a pattern. Of two names spelled out, the longer decides, and a tie keeps the source. Of two patterns, the kept one decides.

The name none, given alone, is an empty list. Use it to keep or to drop nothing where the run would have the names of another run.

The command waits until the machine accepts commands. Use --no-wait to return once the request is accepted.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
machine create [-h] [--like run] [--label name=value] [--no-wait] [--keep-output name]
[--drop-output name] [--disk disk] [--mem size] [--cpus n]
[--console {serial,virtio}] [--mem-type {auto,hugepages}]
[--mem-mode {auto,map,lazy}] [--fs-sync | --no-fs-sync]
[--fs-dirsync | --no-fs-dirsync] [--fb type] [--rng | --no-rng]
[--rng-seed n] [--daemon | --no-daemon] [--single-step | --no-single-step]
[--rdtsc-exit | --no-rdtsc-exit] [--repeatable | --no-repeatable]
[--limit-margin n] [--timer-slack duration] [--instr-margin n]
[--dispatch-bound duration] [--fast-backstop n] [--irq-wait n] [--limit n]
[--backstop-limit n] [--pin-cpu n] [--vtime-base n] [--vtime-rate rate]
[--vtime-io-cost ns] [--vtime-rdtsc-cost ns] [--vtime-pause-cost ns]
[--auto-checkpoints n] [--side-dump | --no-side-dump]
[--side-trace | --no-side-trace] [--mem-hash | --no-mem-hash]
[--disk-hash | --no-disk-hash]
[name]
Argument Does
[name] Name for the new run, defaulting to a generated one
Option Does
--like run Inherit options from another run, its disk aside, overridden by explicit ones
--label name=value Attach a label, repeatable; an empty value removes an inherited one
--no-wait Return without waiting
--keep-output name Source that the run stores though it is dropped: a path under /output, a device such as ttyS1, a pattern, or none (repeatable)
--drop-output name Source that the run does not store: a path under /output, a device such as ttyS1, a pattern, or none (repeatable)
--disk disk Disk to create the run from
--mem size Memory, MB or with a K/M/G/T suffix (default: 256)
--cpus n Number of CPUs (default: 1)
--console {serial,virtio} Device the console is: serial for ttyS0 (default) or virtio for hvc0
--mem-type {auto,hugepages} Machine RAM backing: auto (default) or hugepages (local only)
--mem-mode {auto,map,lazy} How memory loads when the run starts from a checkpoint: auto (default), map to load it first, lazy to load each page at first use
--fs-sync, --no-fs-sync Mount filesystem with sync (default: off)
--fs-dirsync, --no-fs-dirsync Mount filesystem with dirsync (default: off)
--fb type Framebuffer type: vnc, gtk, sdl, or none
--rng, --no-rng Give the guest an RNG device, whose seed a branch can change (default: on)
--rng-seed n Seed of the RNG device, 0 to 2^64-1 (default: 0)
--daemon, --no-daemon Wait for commands instead of running to completion (default: on)
--single-step, --no-single-step Run in step mode, with --no-daemon. A run that waits for commands takes its mode from machine run --mode
--rdtsc-exit, --no-rdtsc-exit Record an entry for every RDTSC exit, with --no-daemon only
--repeatable, --no-repeatable Stop fast exactly at the limit, so recording again gives the same entries (default: the session’s)
--limit-margin n Conditional branches short of a stop where the CPU’s counter interrupts (default: the host’s)
--timer-slack duration How late a timer may be in code that takes no exit, 0 for none (default: 20ms)
--instr-margin n Instructions short of a stop placed by an instruction count where the CPU’s counter interrupts, 0 for the host’s (default: the host’s)
--dispatch-bound duration Host time a fast dispatch may run before it stops, 0 for 1s. A stop it places does not repeat (default: 1s)
--fast-backstop n Instructions a fast dispatch may run with no other stop, 0 for 100000000 (default: 100000000)
--irq-wait n Conditional branches a dispatch runs while an interrupt waits for the guest to take it, 0 for 200 (default: 200)
--limit n Conditional branches per dispatch in fast mode, steps in step mode, 0 to leave it to the machine (default: 0)
--backstop-limit n Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: 0)
--pin-cpu n Pin to this host CPU (default: the least-loaded)
--vtime-base n The vtime base (Unix timestamp, default: 2026-03-01)
--vtime-rate rate Nanoseconds of machine time for each conditional branch the guest retires, a number or num/den, and N=rate for vCPU N’s own, such as 10,1=40 (default: 10)
--vtime-io-cost ns IO exit vtime cost in ns (default: 2000)
--vtime-rdtsc-cost ns RDTSC exit vtime cost in ns (default: the IO cost)
--vtime-pause-cost ns Vtime cost of each PAUSE in ns (default: 2000)
--auto-checkpoints n Checkpoint every n entries, 0 for never (default: scaled by memory, 50000 per 512MB)
--side-dump, --no-side-dump Record a per-entry debug dump in the run log (default: off)
--side-trace, --no-side-trace Record the hypervisor’s trace of each entry, for diagnosis (default: off)
--mem-hash, --no-mem-hash Hash guest memory at every entry, for a replay to compare (default: off)
--disk-hash, --no-disk-hash Hash the guest’s disk at every entry, for a replay to compare (default: off)

Examples:

1
2
3
4
5
6
7
machine create run1 --disk hello                      create run1 from hello
machine create run1 --disk hello --mem 512 --cpus 2 with 512 MB, 2 CPUs
machine create run1 --disk hello --rng-seed 42 its RNG seeded from 42
machine create run1 --disk hello --no-daemon run to completion
machine create run1 --disk hello --repeatable so recording repeats
machine create run1 --disk hello --timer-slack 0 no timer late
machine create run1 --disk hello --drop-output ttyS1 store none of ttyS1

See also:

machine start

Start an existing run. See Stopping and starting.

Use --at to replay the tip from an earlier checkpoint. Use --read-only to open a checkpoint that cannot run or record. Use -f to stop a running machine first.

A stopped run resumes at its tip. A run that stopped past its last checkpoint replays its recorded tail to reach the tip. The replay reports how many entries it replayed. Use --no-recover to refuse instead.

A branch that never ran starts in its own machine. Its parent keeps running.

The machine runs with the execution options of the run. Options given here replace them for this start only.

The command waits until the machine accepts commands. Use --no-wait to return once the request is accepted.

1
2
3
4
machine start [-h] [-f] [--read-only] [--recover | --no-recover] [--at checkpoint]
[--parents mode] [--daemon | --no-daemon] [--single-step | --no-single-step]
[--limit n] [--backstop-limit n] [--pin-cpu n] [--no-wait]
run
Argument Does
run Run name
Option Does
-f, --force Stop a running instance first, then start
--read-only Start read-only at a checkpoint, unable to run or record
--recover, --no-recover Replay a tip that was never checkpointed from the last checkpoint (default: on)
--at checkpoint Checkpoint to replay to the tip from, or to open with --read-only: ^name, or #entry of a checkpoint
--parents mode Where to read the runs it branched from: local for this session, remote for shared storage
--daemon, --no-daemon Wait for commands instead of running to completion (default: the run’s)
--single-step, --no-single-step Run in step mode, with --no-daemon. A run that waits for commands takes its mode from machine run --mode (default: the run’s)
--limit n Conditional branches per dispatch in fast mode, steps in step mode, 0 to leave it to the machine (default: the run’s)
--backstop-limit n Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: the run’s)
--pin-cpu n Pin to this host CPU (default: the least-loaded)
--no-wait Return without waiting

Examples:

1
2
3
4
machine start run1              resume run1 at its tip
machine start run1 --at ^cp1 replay from cp1 to the tip
machine start run1 --read-only open it at its last checkpoint
machine start run1 -f stop a running one first, then start

Refuses, and the way past:

1
a run that is running  machine start run1 -f

See also:

machine pause

Pause a running machine. See Stopping and starting.

The command waits until the machine is paused. Use --no-wait to return once the request is accepted.

1
machine pause [-h] [--no-wait] run
Argument Does
run Run name
Option Does
--no-wait Return without waiting

Examples:

1
machine pause run1  pause run1

See also:

machine resume

Resume a paused machine. See Stopping and starting.

The command waits until the machine accepts commands. Use --no-wait to return once the request is accepted.

1
machine resume [-h] [--no-wait] run
Argument Does
run Run name
Option Does
--no-wait Return without waiting

Examples:

1
machine resume run1  resume the paused run1

See also:

machine stop

Stop a running machine gracefully. See Stopping and starting.

Use -r to stop a branch with its descendants. Use -a to stop every running machine. Use --dry-run to list what would stop.

A machine that refuses to stop stays up and is reported. A machine that does not stop in time stays up too. Use -f with -a or -r to kill those machines.

With -f, the command lists the machines it would kill and asks first. Use -y to kill without asking.

The command waits until the machine is gone. A machine that already exited counts as success. Use --no-wait to return once the request is accepted.

1
machine stop [-h] [-a] [-r] [-f] [-y] [--dry-run] [--no-wait] [run]
Argument Does
[run] Run name
Option Does
-a, --all Stop all running machines
-r, --recursive Stop branch and all descendants
-f, --force With -a or -r, kill a machine that refuses to stop or does not stop in time
-y, --yes With -f, kill without asking first
--dry-run List what would be stopped without stopping it
--no-wait Return without waiting

Examples:

1
2
3
4
5
6
7
machine stop run1               stop run1, wait until it is gone
machine stop run1 --no-wait return once the request is accepted
machine stop run1 -r stop run1 and every branch under it
machine stop run1 -r --dry-run list the machines of its tree
machine stop -a stop every running machine
machine stop -a --dry-run list what would stop
machine stop -afy stop all, kill any that do not stop

See also:

machine kill

Stop a machine, and end it if it does not stop. See Stopping and starting.

The machine is asked to stop and save, as stop does. It is ended if it does not stop.

The next start replays the tip of a machine that did not save. The command says when the machine did not save.

The command waits until the machine is gone. A machine already gone counts as success. Use --no-wait to return once the request is accepted.

1
machine kill [-h] [--no-wait] run
Argument Does
run Run name
Option Does
--no-wait Return without waiting

Examples:

1
2
machine kill run1            stop run1, end it if it hangs
machine kill run1 --no-wait return once the request is accepted

See also:

machine delete

Delete a run that is not running, with its log and checkpoints. See Stopping and starting.

Use -r to delete a branch with its descendants. Use --dry-run to list what would be deleted.

The first operand names the run to delete. Use -a to delete every run, or --disk to delete every run created from one disk. Use --cleanup -a to resume interrupted deletes.

The command waits until the run is gone. Use --stop to stop a running machine first. Use -f to also delete a run whose machine is gone. Use --no-wait to return once the request is accepted.

A delete is an operation. It goes on if the command is interrupted. Run machine ops to list it.

1
2
3
machine delete [-h] [-a] [--disk name|id] [-r] [-y] [--stop] [-f] [--cleanup] [--full-scan]
[--dry-run] [--no-wait]
[run]
Argument Does
[run] Run name
Option Does
-a, --all Delete all runs, or cleanup all interrupted deletes (with --cleanup)
--disk name|id Delete every run created from this disk
-r, --recursive Delete branch and all descendants
-y, --yes Skip the confirmation prompt
--stop Stop running machines before deleting
-f, --force Implies --stop, and deletes a run whose machine is already gone
--cleanup Resume interrupted delete operations
--full-scan With --cleanup, check every run, not only those recorded as interrupted
--dry-run List what would be deleted/recovered without acting
--no-wait Return once the delete of one run is accepted

Examples:

1
2
3
4
machine delete run1                    delete a stopped run
machine delete run1 -r --dry-run list what would go with it
machine delete run1 -r -f stop and delete a tree
machine delete --disk hello --dry-run list the runs on hello

Refuses, and the way past:

1
2
a run with branches    machine delete run1 -r
a run that is running machine delete run1 --stop

See also:

machine cleanup

Remove stale runs and their leftovers. See Stopping and starting.

A stale run is listed as running but has no machine. Use -f to kill machines that belong to no run, and to remove those runs from the list.

1
machine cleanup [-h] [-f]
Option Does
-f, --force Also kill machines that belong to no run, and remove runs whose machine is already gone

Examples:

1
2
machine cleanup     remove stale runs
machine cleanup -f also hard-kill orphaned machines

See also:

machine run

Run one or more vCPU iterations. See Driving it.

The machine executes and the run log records each iteration, so the run can be replayed.

The mode sets how an iteration executes. Fast mode stops near the limit, a little late. Step mode stops exactly. Step-into mode also steps into syscalls, gates and injected events.

An iteration is one dispatch of a vCPU. It ends where the guest next exits to the machine, or at the limit. Its length is not fixed. One may be five instructions and the next a million.

The run stops after --iters iterations or when the --until predicate holds. No count is right in advance, because how long a program runs is not known. Use --until to stop where something happens, with --iters or a vtime predicate as a bound.

A run that went too far is not lost. Use rewind or go to return to an earlier point.

A vtime predicate holds once the clock of the machine reaches a time. vtime>=T counts from the start of the run. vtime=+T counts from where the run is. The run stops at the end of the iteration that reaches the time.

Options vary the run. They draw the mode, the limit and the virtual time rate. They also draw a scheduling mark, slice and interrupt for a task. The same seed draws the same values.

The --until predicates are those of wait. The console~ predicate signals that boot is done. The halted and activity=idle predicates do not.

A console~ predicate takes a regular expression. The run stops at the entry that ends the line the match is on. A match on a line with no end, such as a prompt, stops the run where the machine goes idle.

The match is within a line, as output search finds it, and has at least one character. Use the point console~REGEX to go to the entry that printed the first byte of the match.

The name of another source takes the place of console, as in ttyS1~REGEX. The run then stops for a line that source prints. A line another source prints does not stop it.

The path of an output file takes the same place, as in /output/drop/app/debug.log~REGEX. A file that is dropped stops the run the same way. The run stores none of the file. A path that holds a ~ is quoted, as in ‘“/output/keep/app.log~“~ERROR’.

Only text the run prints from its start ends it. A line printed before does not. Use machine wait --for to ask whether text was printed.

tid=N:at=LOC runs until tid N reaches LOC in its own code. tgid=N:at=LOC runs until any thread of the process does. The address is resolved when the run starts. The stop takes one of the 3 debug registers the run’s watchpoints use.

tid=N:write=LOC runs until an instruction writes the bytes LOC names while tid N is on a vCPU. tid=N:access=LOC runs until one reads or writes them. tgid=N:write=LOC and tgid=N:access=LOC stop for any thread of the process.

The run stops after the access. A write the kernel makes for the task counts. A variable or an expression is watched at its size of 1, 2, 4 or 8 bytes. The watch takes one of the 3 debug registers the run’s watchpoints use.

An address is watched through a cast that gives it a type, as GDB’s watch takes one. The address is a multiple of the size. tgid=N also stops for a kernel thread that works in the process’s memory.

A run that stops at LOC says the tid that reached it and where it stands. A watch also says the value the access left in the bytes. The machine records both at the stop, so a task in the kernel still shows them.

LOC is one of these, in the task’s process:

1
2
3
4
5
6
7
0x401136            an address
*0x401136 an address, as GDB writes it
main a function or a variable, by its name
libc.so.6:malloc a name in one file, by its path or file name
init.c:106 a source line, from the line tables
$rip a register of the task, as it is now
main+4, $rsp-8 any of these moved by +N or -N

The run log records each iteration as an entry. It also records interrupts and device input, so entries outnumber iterations.

The global timeout cancels the run in flight. Without one there is no limit. A timeout of 0 checks once without batching. Exit 3 means the predicate was not reached.

--iters is an upper bound, not a count. A batch is the iterations one request of the command runs. It ends at every context switch of a focused task, a task the focus command names. It also ends where a task reaches its own code.

Without --until, a batch that ends early ends the run. With --until, the run goes on with the next batch. The response reports how many iterations ran.

Every parameter is drawn for one iteration. The mode, the limit, the task marks and the interrupt apply to the task on that one vCPU.

Without --vcpu, a scheduling cycle dispatches every vCPU. It spends up to one iteration for each. --iters 100 on four vCPUs buys 100 dispatches, not 400.

With --until, --iters and the predicate are both bounds. The first to fire wins. --iters 500 --until on stops at 500 iterations or when a focused task takes a vCPU.

Use --within to bound the run by the machine’s clock. The machine checks the bound at every iteration, so a slow host does not end the run sooner. With --until, the first of the two to hold ends it. Without --until, the run goes on until the clock has moved that far.

Without --iters, only the timeout bounds the run, so --until tasks runs a whole boot. A span is lo:hi, a range one value is drawn from for each dispatch. A span bounds nothing either, so --iters 100:500 --until on keeps going.

--when bounds nothing. It names what must be true for the clause to apply to a dispatch. A dispatch it excludes runs free, up to the crossing into the subject.

So --when /test:on --until /test:exit perturbs that program while its own code runs. It runs free while the program is in the kernel or off the CPU. It stops when the program is gone. --iters counts every dispatch, the ones that ran free too.

--when takes on, kernel and off. Each is a state, not an edge.

The options of run make one clause of a plan. Use machine plan <run> <file> for a run of more than one clause.

A hit of a breakpoint, watchpoint or catchpoint ends the run. Use --iters 0 to run until one does, as GDB’s continue does.

Use --back to go back to the newest breakpoint, watchpoint or catchpoint hit the run recorded, as reverse-continue does. The machine switches onto a branch at that entry, as go does. The branch keeps the run’s stops as they are now. The run stays as it was.

1
2
3
4
5
6
7
8
9
10
machine run [-h] [--back] [--limit span] [--backstop-limit n] [--retry n] [--strategy name]
[--seed n] [--vcpu span] [--iters span] [--until predicate] [--within duration]
[--irqblk] [--rdtsc-exit] [--no-idle-skip] [--auto-checkpoints n]
[--mode mode,...] [--fast span] [--step span] [--step-into span] [--fence span]
[--vtime-rate span] [--vtime-add span] [--timer-slack span]
[--irq-delay [target=]span,...] [--vtime-io-cost span,...]
[--vtime-rdtsc-cost span,...] [--vtime-pause-cost span,...]
[--task-pin [tid=]set,...] [--task-sched mark,...] [--task-slice slice,...]
[--task-int int,...] [--when predicate]
run
Argument Does
run Run name
Option Does
--back Go back to the newest hit before now, onto a branch
--limit span Default branches per dispatch under fast and fence, steps under step; 0 means no limit under fast, one step under step and the machine’s fence under fence; default 0
--backstop-limit n Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: 0)
--retry n Retries for an entry the host interrupted, default 2; 0 does not retry
--strategy name How spans resolve when they name none: random, golden, sweep (default: random)
--seed n Seed for the random and golden draw strategies (default: 0)
--vcpu span The vCPU, or a span to draw one per dispatch (default: all)
--iters span Iterations, an upper bound; 0 is unlimited (default: 1; with --until, unbounded)
--until predicate Drive until this is true: a predicate of wait --for, or tid=N:at=LOC, tid=N:write=LOC or tid=N:access=LOC, or the same with tgid=N
--within duration End the run when the machine’s clock has moved this far, such as 10s
--irqblk Block IRQs during run
--rdtsc-exit Record an entry for every RDTSC exit
--no-idle-skip Keep machine time crawling while every vCPU is idle
--auto-checkpoints n Auto-checkpoint every N entries; 0 turns it off, default the machine’s
--mode mode,... Execution mode: fast, step, step-into, free, fence, or a comma list to draw among, each with an optional [weight] (default: fast)
--fast span Limit fast draws when it wins (its member), overriding --limit
--step span Limit step draws when it wins (its member), overriding --limit
--step-into span Limit step-into draws when it wins (its member)
--fence span Conditional branches to stop at when fence mode is drawn, overriding --limit
--vtime-rate span Vtime rate for this run: n, n/d, or a span of either
--vtime-add span Nanoseconds to push the clock forward before each dispatch
--timer-slack span How late a timer may be in code that takes no exit, drawn per dispatch, at most 1s
--irq-delay [target=]span,... Hold each interrupt a time drawn from its target’s span, at most 1s. A target is timer, ipi, init, blk, rng, console, serial, keyboard, device or all, and a bare span is all
--vtime-io-cost span,... Machine time an I/O, MMIO or hypercall exit costs, drawn per dispatch from spans such as 2us or 50:20us, 1ns to 1s
--vtime-rdtsc-cost span,... Machine time an RDTSC; it follows the I/O cost unless given costs, drawn per dispatch from spans such as 2us or 50:20us, 1ns to 1s
--vtime-pause-cost span,... Machine time each PAUSE a spin loop executes costs, drawn per dispatch from spans such as 2us or 50:20us, 1ns to 1s
--task-pin [tid=]set,... Draw per dispatch where a task is placed when it next wakes: a vCPU set such as 1-3, free or none; tid= names the task
--task-sched mark,... Draw a scheduler mark per dispatch for the task on that vCPU: keep, hold, yield, drain, drain-all
--task-slice slice,... Draw that task’s slice per dispatch: keep, default, or a duration range such as 10us:600us
--task-int int,... Draw an interrupt for that task per dispatch: none, preempt
--when predicate Apply the clause only while this holds

Examples:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
machine run run1                             run one iteration
machine run run1 --iters 100 run 100 iterations
machine run run1 --iters 0 continue to the next hit
machine run run1 --mode step --iters 20 single-step 20 iterations
machine run run1 --mode fence --limit 50000 stop on 50000 branches
machine run run1 --until 'console~login:' until the console prints login:
machine run run1 --until 'entry>=500' until it reaches entry 500
machine run run1 --until vtime=+10ms until its clock moves 10 ms
machine run run1 --until tasks until its tasks can be listed
machine run run1 --until tid=41:at=main until tid 41 is at main
machine run run1 --until tgid=7:at=a.c:9 until tgid 7 reaches a.c:9
machine run run1 --until tid=41:write=sum until tid 41 writes sum
machine run run1 --until tgid=7:access=buf until tgid 7 touches buf
machine run run1 --within 10s until its clock has moved 10 s
machine run run1 --vtime-pause-cost 50:20us a PAUSE's cost drawn
machine run run1 --task-pin 56=1,2 tid 56 on vCPU 1 or 2
machine run run1 --until tasks --within 9s the first of the two
machine run run1 --irq-delay 0:2ms hold each interrupt up to 2 ms
machine run run1 --irq-delay blk=1ms hold disk interrupts 1 ms
machine run run1 --timer-slack 0:20ms draw the timer slack

Prints:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
progress       <done>/<batch> vtime=<time> entry=<id> ic=<instructions>
summary iterations: <n> run ic: <instructions in this run>
vtime: +<how far the clock moved in this run>
last dispatch VCPU EXIT ID RUN IC RCB TOTAL IC REGS HASH RIP RCX ITER
EXIT why it ended: IO, MMIO, HYPERCALL, HLT, INSTR_LIMIT
INSTR_LIMIT is the limit reached, or a cancel
IO with EXIT_SHUTDOWN is the guest powering off
ID the entry the dispatch wrote
RCB conditional branches retired, which replay matches on
REGS HASH a hash of the registers, which replay matches on
ITER which iteration of this command it was
task: the task event an --until predicate was met by
guest event a stop the guest asked for, such as a task switch
matched <n> with --when, how many dispatches it applied to
the rest of the iterations ran free
Stopped at the breakpoint, watchpoint or catchpoint hit, and tid

See also:

machine plan

Run a multi-clause plan from a file. See Plans.

A plan varies how the run executes. Its clauses draw the mode, the virtual time, and the scheduling and interrupts of tasks. Use --iters and --until to bound the run, as with run.

A plan file is a list of clauses. A clause is an optional when predicate and the per-dispatch axes it draws for a dispatch it matches. Clauses are tried in order. The last to match supplies each axis, so a later clause overrides an earlier one.

The else clause applies to a dispatch that no when matched. It is the last clause of the file.

Braces may group a clause. # begins a comment. Blank lines are ignored.

A file of - reads the plan from standard input. The plan applies to the run of this command only. A later machine run runs without the plan. Run machine plan again to go on with it.

Example:

1
2
3
when /test:on       mode step,fast   fast=1:100k step=1:200
when /test:kernel task-sched hold
else mode fast

Example: place tid 56 on vCPU 2 or 3 when it wakes, 2 three times as often as 3:

1
else  task-pin 56=golden:2[3],3[1]

A clause names one when predicate, then any of these axes. An axis is a comma list of choices, one drawn per dispatch. [N] is an optional weight on a choice:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
when <predicate>
apply the clause only while the predicate holds. A
predicate is [subject:]state. The state is on, kernel,
off, syscall[=name] or int[=vec]. The subject is /path,
tid=N, tgid=N, cgroup=P, comm=NAME or none for any
focused task.
vcpu=N[,N] ANDs a vCPU set onto it.
mode <mode>[N],...
the modes to draw among: fast, step, step-into, free, fence
or keep. A mode may carry the limit it draws when it wins,
<mode>=<span>, as in mode step[3],fast[1] step=1:200.
free takes no limit.
vtime-rate <rate>[N],...
the machine-time rate: a number N, a fraction N/D, a span
of numerators such as 1:10, or keep.
vtime-add <duration>[N],...
time added to the clock before each dispatch: a duration
such as 500 or 10us, a span such as 10us:2ms, or keep.
task-sched <mark>[N],...
a mark for the task on the drawn vCPU: keep, hold, yield,
drain, drain-all.
task-int <int>[N],...
an interrupt for that task: none or preempt. This axis has
no keep: none is its do-nothing choice.
task-slice <choice>[N],...
that task's time slice: keep, default, a duration or a
range of durations, as in keep,default,10us:600us.
task-pin [<tid>=]<set>[N],...
where a task is placed when it next wakes, each choice a
set of vCPUs: 1-3 is all three, 1,2,3 is three choices of
one, free is any vCPU and none is no vCPU, which stops it
running. A placement policy, not confinement: a task
already queued elsewhere keeps that vCPU until it wakes.
<tid>= names the task. Without it, the task on the
drawn vCPU is placed. none requires <tid>=.
timer-slack <duration>
how late a timer may be in code that takes no exit, drawn
per dispatch: a duration or a span such as 0:20ms, at
most 1s. keep repeats the last slack drawn; with none
drawn yet, the run's own --timer-slack.
irq-delay [<target>=]<duration>,...
how long each interrupt is held before it is injected,
drawn per interrupt: a duration or a span such as 0:2ms,
at most 1s. A target is timer, ipi, init, blk, rng,
console, serial, keyboard, device or all. A bare span is
all. The most specific target wins. An interrupt never
overtakes one held before it on its line, its MSI vector,
or its vCPU's timer, IPI vector or INIT and SIPI.
vtime-io-cost <duration>[N],...
what an I/O, MMIO or hypercall exit costs on the clock,
drawn per dispatch: a duration such as 2000 or 2us, a span
such as 1us:20us, or keep; 1ns to 1s. The run's own cost
comes back when the run ends.
vtime-rdtsc-cost <duration>[N],...
what an RDTSC costs, the same way. Without it the RDTSC
cost follows the I/O cost.
vtime-pause-cost <duration>[N],...
what each PAUSE of a spin loop costs, the same way.

A weight makes a choice win more often. mode step[3],fast[1] draws step three times as often as fast. keep is the choice that draws nothing. It holds the last value of the axis, so a clause can leave an axis alone part of the time.

A span is a range written lo:hi[:strategy[:step]]. The strategy is random, golden or sweep. One value is drawn from the span per dispatch.

A span is accepted by the mode member limit, vtime-rate, vtime-add, task-slice, timer-slack, irq-delay and the run-level --limit. The three exit costs take one too. The limit itself is not a clause axis. It is the default for a mode with no member limit of its own.

The options of this command are plan-level. --limit and --strategy are the defaults for a clause that names no span of its own. A clause axis overrides them for the dispatches it matches. A dispatch that no clause matches runs free.

--seed seeds the draw strategies. --until, --iters and --vcpu bound and steer the run as on run. The clause axes and when belong in the file, not here.

1
2
3
4
5
machine plan [-h] [--limit span] [--backstop-limit n] [--retry n] [--strategy name]
[--seed n] [--vcpu span] [--iters span] [--until predicate]
[--within duration] [--irqblk] [--rdtsc-exit] [--no-idle-skip]
[--auto-checkpoints n]
run file
Argument Does
run Run name
file Plan file of when and else clauses, and their axes, or - to read standard input
Option Does
--limit span Default branches per dispatch under fast and fence, steps under step; 0 means no limit under fast, one step under step and the machine’s fence under fence; default 0
--backstop-limit n Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: 0)
--retry n Retries for an entry the host interrupted, default 2; 0 does not retry
--strategy name How spans resolve when they name none: random, golden, sweep (default: random)
--seed n Seed for the random and golden draw strategies (default: 0)
--vcpu span The vCPU, or a span to draw one per dispatch (default: all)
--iters span Iterations, an upper bound; 0 is unlimited (default: 1; with --until, unbounded)
--until predicate Drive until this is true: a predicate of wait --for, or tid=N:at=LOC, tid=N:write=LOC or tid=N:access=LOC, or the same with tgid=N
--within duration End the run when the machine’s clock has moved this far, such as 10s
--irqblk Block IRQs during run
--rdtsc-exit Record an entry for every RDTSC exit
--no-idle-skip Keep machine time crawling while every vCPU is idle
--auto-checkpoints n Auto-checkpoint every N entries; 0 turns it off, default the machine’s

Examples:

1
2
3
machine plan run1 plan.txt                       run the clauses of plan.txt
machine plan run1 plan.txt --iters 100 --seed 7 100 iterations, seed 7
machine plan run1 - plan from standard input

Prints:

1
2
3
4
5
tally         one line for each clause, numbered from 1
<choice>×<n> how many times that choice of an axis was drawn
matched <n> how many dispatches the clause applied to
marks held tasks a clause held that are still held at the end
then what run prints

See also:

machine control

Control a run interactively. See Interactive mode.

The prompt takes slash commands. They map to the commands of machine, with a syntax of their own.

Type /help at the prompt to list the commands. Use -c to send a slash command from the command line, without opening the prompt. Use -c /help to list the commands that way.

A semicolon between two slash commands runs the second after the first, and stops when the first fails. The prompt reads quotes as a shell reads them. A semicolon inside quotes is part of the word.

Use control log with a run to read the command history instead of driving the machine. Use --entries to select the commands issued at given run entries. Use -n to select commands by number.

1
machine control [-h] [--entries range] [--wide] [-n range] [-c cmd] run|log [run]
Argument Does
run|log Run name, or log to read history
[run] With log: the run whose history to read
Option Does
--entries range With log: commands at one entry, or an inclusive range
--wide With log: show the entries, vCPU, opcode and size of each command
-n, --lines range With log: commands: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end
-c, --command cmd Slash command to run without opening the prompt, such as “/f@100”

Examples:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
machine control run1                            open the prompt for run1
machine control run1 -c /help list the slash commands
machine control run1 -c '/help output' the help of /output
machine control run1 -c /f@100 run 100 iterations fast
machine control run1 -c '/checkpoint name cp1' create checkpoint cp1
machine control run1 -c '/rewind to ^cp1' put run1 back at cp1
machine control run1 -c '/send ls -la' type ls -la and Enter
machine control run1 -c '/send "a;b"' send a;b and Enter
machine control run1 -c '/exec run ls -la /' execute ls -la /
machine control run1 -c '/exec run -c echo $0' a line for the shell
machine control run1 -c '/exec start sleep 5' send it, get its id
machine control run1 -c /exec the processes of run1
machine control run1 -c '/fs ls / at ^cp1' the disk's files at cp1
machine control run1 -c '/f;/d' run, then dump
machine control log run1 the commands issued to run1

See also:

machine wait

Wait until a run satisfies a predicate. See Waiting.

Use wait to script against a run. It waits for a state, an entry, a console line or a task event. Several runs, or every run of the session, may be waited for.

The predicates are running, ready, tasks, stopped, paused, halted, activity=<name>, entry>=N, entry=+N, vtime>=T, vtime=+T, console~REGEX and a task predicate. The tasks predicate holds once the operating system is up and its task table is readable. That is before it execs anything in userland.

A console~REGEX predicate holds once the console prints a match. The name of another source takes the place of console, as in ttyS1~REGEX.

A task predicate is [who:]what. The what is exec, exit, sched, desched, on, kernel or off. The who is /path, cgroup=path, comm=name, tgid=N or tid=N. Omitting it means any focused task.

A path, a cgroup or a comm must already be focused. Such a subject means its tgid. tgid=N means the tgid for exit and any of its tasks for exec. Use machine tasks <run> --focused to see which task is on a vCPU.

Four of the seven are edges. exec fires where a task matching who entered the task table. exit fires where one left it.

sched fires where a task is given the CPU. desched fires where one gives it up.

Three are states, and they partition time. on holds while the task is on cpu in its own code, at CPL 3. That is where a forced decision can land between two of its own instructions.

kernel holds while that task is on cpu in the kernel. off holds while it is not the task on cpu at all.

Named runs must be in the current session. Use --all to wait for every run in it.

The global timeout bounds the wait. Without one there is no limit. A timeout of 0 checks once.

Exit codes: 0 success, 2 error, 3 timeout.

1
machine wait [-h] [--all] --for predicate [--interval s] [run ...]
Argument Does
[run ...] One or more runs (omit with --all)
Option Does
--all Wait for every run in the session
--for predicate Predicate such as running, stopped, entry=+N or console~REGEX
--interval s Poll every S seconds (default: 1)

Examples:

1
2
3
4
machine wait --for running run1               until run1 runs
machine wait --for stopped run1 until it stops
machine wait --for 'entry>=500' run1 until it reaches entry 500
machine --timeout 60 wait --for stopped run1 give up after 60 s

See also:

machine cancel

Cancel the run or replay in progress. See Cancelling a drive.

A run stops where it is. The stop falls where the host’s clock put it, so another recording of the same state does not stop there. Describe reports the run as not repeatable from that entry. A replay stops before its next entry.

Use --soft to stop the run at the end of the dispatch in progress. That stop is an ordinary entry, and the run stays repeatable. It takes at most the run’s dispatch bound, 1s by default. Ctrl-C stops a run this way, and a second Ctrl-C stops it now.

1
machine cancel [-h] [--soft] run
Argument Does
run Run name
Option Does
--soft Stop at the end of the dispatch in progress, which keeps the run repeatable

Examples:

1
2
machine cancel run1         cancel what run1 is running
machine cancel run1 --soft at the end of its dispatch

Refuses, and the way past:

1
a run with nothing in progress  machine state run1

See also:

machine replay

Replay recorded entries from the parent lineage. See Replay.

The command executes the recorded entries again. During replay, Machine checks each entry against the original recording. If they don’t match, Machine reports a divergence. It names the entry, with the expected and actual values.

Without a line, the parent’s entries are replayed. A fork at an entry leaves that entry with the parent. The branch starts at the next entry, so the entries of the parent end at the fork. Replaying past them reports no forward entries.

Use --to to stop at a point of the line that is replayed. The point is an entry, a checkpoint or a time of that line. A checkpoint the parent took after the fork is such a point.

The line of a run is its entries from the root on, across every fork. Naming a run replays the entries of its line. The run must be in the same tree. The entry this run stands on must be on the line replayed.

An entry that diverges is replayed again from the newest checkpoint before it, at most twice. It is not replayed again once it diverges the same way twice. An entry that the host interrupted is replayed again the same way. It is not a divergence.

Use --retry to set how many retries. --retry 0 replays each entry once. Use --limit-margin to set how many conditional branches short of each recorded entry the CPU’s counter interrupts. A smaller margin makes a miss more likely.

The replay stores what the branch keeps of its output. A branch that keeps an output file its parent dropped is how the file is read. Use fork with --keep-output to make one. What is stored does not change what the machine executes.

1
2
3
machine replay [-h] [--to point [point ...]] [--trace] [--retry n] [--limit-margin n]
[--instr-margin n]
run [line]
Argument Does
run Run name
[line] Run whose line to replay, defaulting to the parent’s
Option Does
--to point [point ...] Point of the line to replay to: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, now, parent [n], root, or %mark (default: the end of the line)
--trace Trace each iteration one by one
--retry n Retries for an entry that diverges or the host interrupted, default 2; 0 does not retry
--limit-margin n Conditional branches short of each recorded stop where the CPU’s counter interrupts (default: the run’s)
--instr-margin n Instructions short of each stop placed by an instruction count where the CPU’s counter interrupts (default: the run’s)

Examples:

1
2
3
4
5
6
machine replay try2                        replay the parent's entries
machine replay try2 --to '#300' replay up to entry 300
machine replay try2 --to ^cp2 replay up to checkpoint cp2
machine replay try2 run1 --to '#420' replay the line to run1
machine replay try2 --limit-margin 100 100 branches short of each stop
machine replay try2 --instr-margin 100000 100,000 instructions short

Prints:

1
2
3
4
text          Replayed <n> entries to entry <id>. Converged.
then the last dispatch, as run prints it
json converged entry_id replayed diverged result misses
nothing left Nothing to replay: already at entry <id>

Refuses, and the way past:

1
a run off the line it would replay  machine replay try2 run1

See also:

machine play

Play recorded entries without checking divergence. See Replay.

Use --include and --exclude to select the event types to play. The entries no type names are always played. They are the clock’s rate and steps, the focus, the stops and the debugger’s writes. The RUN entries after them need them.

A play that leaves out irq delivers the interrupts the machine raises itself, as a recording does. The run records them as its own.

The line of a run is its entries from the root on, as replay says. The run named must be in the tree of this run, and need not be on its line. Divergence is not checked.

1
2
machine play [-h] [-n range] [--include type [type ...]] [--exclude type [type ...]]
run [line]
Argument Does
run Run name
[line] Run whose line to play, defaulting to the parent’s
Option Does
-n, --entries range Entries: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end
--include type [type ...] Play only these event types: run, irq, exception, random, device_in, block_done, input, exit_shutdown, system_shutdown, system_reset, checkpoint, cpuid, rng_seed
--exclude type [type ...] Leave these event types out

Examples:

1
2
machine play run1 -n 295:300  play entries 295 to 300 of run1
machine play try2 run1 play run1's entries on try2

See also:

machine diff

Compare a run log entry against another. See Replay.

The command compares the state recorded at two entries. It breaks the combined hash into machine state, memory and disk, with register detail, so the layer that differs shows. It reads the recorded log, so no replay is needed. Use --show to choose sections and --context to add the entries before.

With no second operand, the entry is compared against the run it was replayed from. The answer says whether the replay converged. Naming a run compares against that run, at its own entry when given as other:m. A difference between two recordings says that they differ, not that a replay diverged.

1
machine diff [-h] [--show sections] [--entry n] [--context n] run [other[:m]]
Argument Does
run Run name
[other[:m]] Run to compare against, at its entry m; m defaults to --entry
Option Does
--show sections One or more, comma-separated: code, regs, stack, pt, instr, trace, events, timers, debug, xsave, irq, pic, ioapic, devs, mem, disk, hash, counts, entry, io, msrs, fullhash, all; default hash
--entry n Compare specific entry N (0 = last entry)
--context n Also show the N entries leading up to it, on both sides

Examples:

1
2
3
machine diff try2                   compare try2 with the run it replayed
machine diff try2 --show code,regs include code and registers
machine diff try2 run1 compare try2 with run1 instead

See also:

machine checkpoint

Create a checkpoint of the machine’s state. See Checkpoints.

A checkpoint is a point to return to, written ^name. It saves the machine state. Fork, go and rewind return to it.

The machine also checkpoints on its own, at the interval that create sets.

1
machine checkpoint [-h] run [name]
Argument Does
run Run name
[name] Checkpoint name (auto-generated if omitted)

Examples:

1
2
machine checkpoint run1      checkpoint run1 under a generated name
machine checkpoint run1 cp1 checkpoint run1 as cp1

See also:

machine fork

Fork a branch and start it alongside the parent. See Starting a branch.

A branch shares the history of its parent up to the fork point. Then it runs on its own path. Use --rebase to copy the data of the ancestors instead of sharing it. Use --no-start to create the branch without starting it.

The parent keeps running. The terminal stays attached to the parent.

The branch forks at the current point and reuses a checkpoint there if one exists. Use --at to fork at another point.

The branch keeps the execution options of the parent. Options given here replace them for the branch.

The new run keeps and drops the output that the run it comes from does. Use --keep-output to replace the names it keeps. Use --drop-output to replace the names it drops. Each option changes its own list only.

Use --from to replay the new run from an earlier point to its point. The new run stores the output of that stretch by its own names. It is forked at the checkpoint at or before the earlier point.

A run stores every source but the files under /output/drop. A file there has a name and no bytes. A program writes under /output/drop through MACHINE_DROP_OUTPUT_DIR. It writes under /output/keep through MACHINE_OUTPUT_DIR.

A name given to either option may be a pattern, such as /output/drop//.log. A star matches within one directory name, a leading dot included, and never a slash. Two stars match any number of directories.

A name given to --keep-output or --drop-output decides before the directory does. When a kept name and a dropped name cover one source, a name spelled out decides over a pattern. Of two names spelled out, the longer decides, and a tie keeps the source. Of two patterns, the kept one decides.

The name none, given alone, is an empty list. Use it to keep or to drop nothing where the run would have the names of another run.

The command waits until the branch accepts commands. Use --no-wait to return once the request is accepted.

1
2
3
4
5
6
7
8
9
10
machine fork [-h] [--at point [point ...]] [--from point [point ...]] [--name name]
[--rebase] [--depth n] [--lean] [--no-start] [--side-dump | --no-side-dump]
[--side-trace | --no-side-trace] [--mem-hash | --no-mem-hash]
[--repeatable | --no-repeatable] [--limit-margin n] [--timer-slack duration]
[--instr-margin n] [--dispatch-bound duration] [--fast-backstop n]
[--irq-wait n] [--disk-hash | --no-disk-hash] [--rng-seed n]
[--keep-output name] [--drop-output name] [--no-wait] [--daemon | --no-daemon]
[--single-step | --no-single-step] [--limit n] [--backstop-limit n]
[--pin-cpu n]
run
Argument Does
run Run to fork from
Option Does
--at point [point ...] Point the new run is at: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, now, parent [n], root, or %mark
--from point [point ...] Earlier point to replay the new run from: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, parent [n] or root. The new run starts at the checkpoint at or before it
--name name Name for the new run (default: a name made from where it starts)
--rebase Copy the ancestors’ data into the new branch instead of sharing it
--depth n With --rebase, how many root-side ancestors to keep (default 1: the root)
--lean With --rebase, keep no history shared with the ancestors, so they can be deleted
--no-start Create the new run without starting it
--side-dump, --no-side-dump Record a per-entry debug dump, whatever the run it comes from records
--side-trace, --no-side-trace Record the hypervisor’s trace of each entry, for diagnosis, whatever the run it comes from records
--mem-hash, --no-mem-hash Hash guest memory at every entry, whatever the run it comes from records
--repeatable, --no-repeatable Stop fast exactly at the limit, so recording again gives the same entries, whatever the run it comes from does
--limit-margin n Conditional branches short of a stop where the CPU’s counter interrupts (default: the run’s)
--timer-slack duration How late a timer may be in code that takes no exit, 0 for none (default: the run’s)
--instr-margin n Instructions short of a stop placed by an instruction count where the CPU’s counter interrupts, 0 for the host’s (default: the run’s)
--dispatch-bound duration Host time a fast dispatch may run before it stops, 0 for 1s. A stop it places does not repeat (default: the run’s)
--fast-backstop n Instructions a fast dispatch may run with no other stop, 0 for 100000000 (default: the run’s)
--irq-wait n Conditional branches a dispatch runs while an interrupt waits for the guest to take it, 0 for 200 (default: the run’s)
--disk-hash, --no-disk-hash Hash the guest’s disk at every entry, whatever the run it comes from records
--rng-seed n Seed the new run’s RNG device from n once it is up, so its guest draws other random numbers
--keep-output name Source that the run stores though it is dropped: a path under /output, a device such as ttyS1, a pattern, or none (repeatable)
--drop-output name Source that the run does not store: a path under /output, a device such as ttyS1, a pattern, or none (repeatable)
--no-wait Return without waiting
--daemon, --no-daemon Wait for commands instead of running to completion (default: the run’s)
--single-step, --no-single-step Run in step mode, with --no-daemon. A run that waits for commands takes its mode from machine run --mode (default: the run’s)
--limit n Conditional branches per dispatch in fast mode, steps in step mode, 0 to leave it to the machine (default: the run’s)
--backstop-limit n Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: the run’s)
--pin-cpu n Pin to this host CPU (default: the least-loaded)

Examples:

1
2
3
4
5
6
7
8
9
10
machine fork run1                         fork here and start the branch
machine fork run1 --name probe name the branch probe
machine fork run1 --at ^cp1 fork at checkpoint cp1
machine fork run1 --at -10ms fork 10 ms before now
machine fork run1 --at '#120' --no-start create the branch, do not start it
machine fork run1 --keep-output /output store what is dropped
machine fork run1 --drop-output ttyS1 store none of ttyS1
machine fork run1 --from ^cp1 replay the branch from cp1 to now
machine fork run1 --at ^cp1 --rng-seed 7 a branch seeded from 7
machine fork run1 --no-repeatable a branch without exact stops

See also:

machine go

Branch at a point and switch onto it. See Switching the machine.

Go starts another path from a point. The branch shares the history of the run up to the point.

The machine moves onto the branch. The run it leaves is listed as switched. A switched run keeps its history and has no machine.

Go is the only command that branches and switches in one step. The point operand lists the forms a point takes. Each names a point on the own line of this run.

console~REGEX is where the console first printed a match. It is the entry that printed the first byte of the match. The name of another source takes the place of console, as in ttyS1~REGEX.

A time such as 10ms counts from the start of the run. A minus sign, as in -10ms, counts back from now. A time may carry +N, as machine now prints it, to name one entry among several at that time.

A minus sign before a number with no unit counts entries back from now. The point -10 is #-10, 10 entries before now.

Go reaches a point that is not a checkpoint by branching at the nearest checkpoint at or before it. It then replays the difference.

Every use creates a branch, also for a point gone to before. What runs from a point is a path of its own. Use switch to return to a branch that exists. Use rewind to put this run back at a point.

The new run keeps and drops the output that the run it comes from does. Use --keep-output to replace the names it keeps. Use --drop-output to replace the names it drops. Each option changes its own list only.

Use --from to replay the new run from an earlier point to its point. The new run stores the output of that stretch by its own names. It is forked at the checkpoint at or before the earlier point.

A run stores every source but the files under /output/drop. A file there has a name and no bytes. A program writes under /output/drop through MACHINE_DROP_OUTPUT_DIR. It writes under /output/keep through MACHINE_OUTPUT_DIR.

A name given to either option may be a pattern, such as /output/drop//.log. A star matches within one directory name, a leading dot included, and never a slash. Two stars match any number of directories.

A name given to --keep-output or --drop-output decides before the directory does. When a kept name and a dropped name cover one source, a name spelled out decides over a pattern. Of two names spelled out, the longer decides, and a tie keeps the source. Of two patterns, the kept one decides.

The name none, given alone, is an empty list. Use it to keep or to drop nothing where the run would have the names of another run.

1
machine go [options] <run> <point>
Argument Does
run The running machine that moves
[point ...] Where to branch: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, now, parent [n], root, or %mark
Option Does
--from point [point ...] Earlier point to replay the new run from: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, parent [n] or root. The new run starts at the checkpoint at or before it
--name name Name for the new run (default: a name made from where it starts)
--rebase Copy the ancestors’ data into the new branch instead of sharing it
--depth n With --rebase, how many root-side ancestors to keep (default 1: the root)
--lean With --rebase, keep no history shared with the ancestors, so they can be deleted
--side-dump, --no-side-dump Record a per-entry debug dump, whatever the run it comes from records
--side-trace, --no-side-trace Record the hypervisor’s trace of each entry, for diagnosis, whatever the run it comes from records
--mem-hash, --no-mem-hash Hash guest memory at every entry, whatever the run it comes from records
--repeatable, --no-repeatable Stop fast exactly at the limit, so recording again gives the same entries, whatever the run it comes from does
--limit-margin n Conditional branches short of a stop where the CPU’s counter interrupts (default: the run’s)
--timer-slack duration How late a timer may be in code that takes no exit, 0 for none (default: the run’s)
--instr-margin n Instructions short of a stop placed by an instruction count where the CPU’s counter interrupts, 0 for the host’s (default: the run’s)
--dispatch-bound duration Host time a fast dispatch may run before it stops, 0 for 1s. A stop it places does not repeat (default: the run’s)
--fast-backstop n Instructions a fast dispatch may run with no other stop, 0 for 100000000 (default: the run’s)
--irq-wait n Conditional branches a dispatch runs while an interrupt waits for the guest to take it, 0 for 200 (default: the run’s)
--disk-hash, --no-disk-hash Hash the guest’s disk at every entry, whatever the run it comes from records
--rng-seed n Seed the new run’s RNG device from n once it is up, so its guest draws other random numbers
--keep-output name Source that the run stores though it is dropped: a path under /output, a device such as ttyS1, a pattern, or none (repeatable)
--drop-output name Source that the run does not store: a path under /output, a device such as ttyS1, a pattern, or none (repeatable)
--no-wait Return without waiting

Examples:

1
2
3
4
5
6
7
machine go run1 ^cp1                 branch at checkpoint cp1 and switch
machine go run1 '#120' --name probe branch at entry 120 as probe
machine go run1 parent branch where run1 left its parent
machine go run1 -10ms branch 10 ms before now
machine go run1 -10 branch 10 entries before now
machine go run1 console:120 branch where line 120 was printed
machine go run1 now --from ^cp1 replay the branch from cp1

Prints:

1
2
3
result           Branched <parent>/<name> (<id>) at entry <n>
<parent>/<name> how a branch is listed, under the run it left
every command takes the name alone

See also:

machine switch

Switch a running machine onto a branch that exists. See Switching the machine.

The first operand names the run that switches, as with fork and detach. The second names where it goes: a branch, a mark written %mark, parent or root. Switch never creates a branch.

The run that the machine leaves is listed as switched. A switched run keeps its history and has no machine.

Go forks a branch and switches onto it. Fork forks a branch without switching.

1
machine switch [-h] [--no-wait] run [branch ...]
Argument Does
run The running machine that switches
[branch ...] Branch to switch onto: a name, a mark written %mark, parent [n], or root
Option Does
--no-wait Return without waiting

Examples:

1
2
3
machine switch run1 try2    switch run1's machine onto try2
machine switch run1 parent switch it onto the parent branch
machine switch run1 %good switch it onto the branch the mark names

See also:

machine rewind

Put a run back at a point and discard what follows. See Switching the machine.

The run keeps its id and its name. The entries, checkpoints and console output after the point are discarded.

The point is required. It is any entry from #1 to the tip. Write an entry id after #, a checkpoint name after ^, or a machine time. The run lands on it exactly.

The point becomes a checkpoint. Rewinding to the tip changes nothing unless a replay diverged there. A point before the run was forked puts the run back along the line it was forked from. A run with a branch forked after the point is refused.

1
machine rewind <run> to <point>
Argument Does
run The machine that rewinds
[point ...] Point to go back to, from #1 to the tip: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, now, parent [n], root, or %mark

Examples:

1
2
machine rewind run1 to ^cp1    put run1 back at checkpoint cp1
machine rewind run1 to '#120' put run1 back at entry 120

Refuses, and the way past:

1
a branch forked after the point  machine delete try2

See also:

machine detach

Detach a new root from a run and start it. See Switching the machine.

Without --at, the root is detached at the current point. The copy includes everything the root still owns. It shares nothing with the origin tree. It is slower than fork --rebase.

The new root is named for the run it copied and the point. A second copy of the same point gets a suffix.

The new root keeps the execution options of the run it copied. Options given here replace them for the root.

The new run keeps and drops the output that the run it comes from does. Use --keep-output to replace the names it keeps. Use --drop-output to replace the names it drops. Each option changes its own list only.

Use --from to replay the new run from an earlier point to its point. The new run stores the output of that stretch by its own names. It is forked at the checkpoint at or before the earlier point.

A run stores every source but the files under /output/drop. A file there has a name and no bytes. A program writes under /output/drop through MACHINE_DROP_OUTPUT_DIR. It writes under /output/keep through MACHINE_OUTPUT_DIR.

A name given to either option may be a pattern, such as /output/drop//.log. A star matches within one directory name, a leading dot included, and never a slash. Two stars match any number of directories.

A name given to --keep-output or --drop-output decides before the directory does. When a kept name and a dropped name cover one source, a name spelled out decides over a pattern. Of two names spelled out, the longer decides, and a tie keeps the source. Of two patterns, the kept one decides.

The name none, given alone, is an empty list. Use it to keep or to drop nothing where the run would have the names of another run.

1
2
3
4
5
6
7
8
9
10
machine detach [-h] [--at point [point ...]] [--from point [point ...]] [--name name]
[--no-start] [--side-dump | --no-side-dump] [--side-trace | --no-side-trace]
[--mem-hash | --no-mem-hash] [--repeatable | --no-repeatable]
[--limit-margin n] [--timer-slack duration] [--instr-margin n]
[--dispatch-bound duration] [--fast-backstop n] [--irq-wait n]
[--disk-hash | --no-disk-hash] [--rng-seed n] [--keep-output name]
[--drop-output name] [--no-wait] [--daemon | --no-daemon]
[--single-step | --no-single-step] [--limit n] [--backstop-limit n]
[--pin-cpu n]
run
Argument Does
run Run to detach from
Option Does
--at point [point ...] Point the new run is at: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, now, parent [n], root, or %mark
--from point [point ...] Earlier point to replay the new run from: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, parent [n] or root. The new run starts at the checkpoint at or before it
--name name Name for the new run (default: a name made from where it starts)
--no-start Create the new run without starting it
--side-dump, --no-side-dump Record a per-entry debug dump, whatever the run it comes from records
--side-trace, --no-side-trace Record the hypervisor’s trace of each entry, for diagnosis, whatever the run it comes from records
--mem-hash, --no-mem-hash Hash guest memory at every entry, whatever the run it comes from records
--repeatable, --no-repeatable Stop fast exactly at the limit, so recording again gives the same entries, whatever the run it comes from does
--limit-margin n Conditional branches short of a stop where the CPU’s counter interrupts (default: the run’s)
--timer-slack duration How late a timer may be in code that takes no exit, 0 for none (default: the run’s)
--instr-margin n Instructions short of a stop placed by an instruction count where the CPU’s counter interrupts, 0 for the host’s (default: the run’s)
--dispatch-bound duration Host time a fast dispatch may run before it stops, 0 for 1s. A stop it places does not repeat (default: the run’s)
--fast-backstop n Instructions a fast dispatch may run with no other stop, 0 for 100000000 (default: the run’s)
--irq-wait n Conditional branches a dispatch runs while an interrupt waits for the guest to take it, 0 for 200 (default: the run’s)
--disk-hash, --no-disk-hash Hash the guest’s disk at every entry, whatever the run it comes from records
--rng-seed n Seed the new run’s RNG device from n once it is up, so its guest draws other random numbers
--keep-output name Source that the run stores though it is dropped: a path under /output, a device such as ttyS1, a pattern, or none (repeatable)
--drop-output name Source that the run does not store: a path under /output, a device such as ttyS1, a pattern, or none (repeatable)
--no-wait Return without waiting
--daemon, --no-daemon Wait for commands instead of running to completion (default: the run’s)
--single-step, --no-single-step Run in step mode, with --no-daemon. A run that waits for commands takes its mode from machine run --mode (default: the run’s)
--limit n Conditional branches per dispatch in fast mode, steps in step mode, 0 to leave it to the machine (default: the run’s)
--backstop-limit n Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: the run’s)
--pin-cpu n Pin to this host CPU (default: the least-loaded)

Examples:

1
2
3
machine detach run1 --at ^cp1                 copy run1 at cp1 to a new root
machine detach run1 --at '#120' --name copy1 name the new root copy1
machine detach run1 --from ^cp1 replay the root from cp1

See also:

machine checkpoints

List the checkpoints of a branch. See Checkpoints.

Each checkpoint is a point for fork --at, go and rewind.

The listing shows the first 25 checkpoints. Use --limit for a different page size. No running machine is needed.

The durable column says whether a checkpoint is synced to shared storage. Use -o json for an object of run_id, hosted and checkpoints instead of a bare list.

1
2
3
machine checkpoints [-h] [--durable] [--since date|age] [--until date|age] [--sort field]
[--reverse] [--limit n] [--offset n] [-w]
run
Argument Does
run Run name
Option Does
--durable Show only checkpoints synced to shared storage
--since date|age Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--until date|age Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--sort field Sort field: entry, date, name (default: entry)
--reverse Reverse sort order
--limit n Show at most n checkpoints (default: 25)
--offset n Skip the first n checkpoints
-w, --watch Watch for changes

Examples:

1
2
machine checkpoints run1            list the checkpoints of run1
machine checkpoints run1 --durable only those in shared storage

Prints:

1
2
3
4
5
text      NAME ID VTIME DURABLE ANCESTOR
ID the entry the checkpoint is at
VTIME machine time there, with +N among entries at one time
DURABLE ticked when it is in shared storage, no when it is not
ANCESTOR the ancestor that took it, for a checkpoint this run did not take

See also:

machine marks

List and manage marks. See Marks.

A mark is a second name for a branch. It is unique within its tree. Written %mark, it names the branch anywhere a run is named.

Without a command, marks of every tree are listed. Use list with a run for the marks of that run’s tree. No running machine is needed.

1
machine marks [-h] command ...

Examples:

1
2
3
machine marks list              list marks
machine marks add run1 good put the mark good on run1
machine marks remove run1 good take it off

See also:

machine marks list

List marks. See Marks.

The listing covers every tree, or the tree of one run when a run is named.

1
machine marks list [-h] [run]
Argument Does
[run] Run name

Examples:

1
2
machine marks list       marks of every tree
machine marks list run1 marks of the tree of run1

See also:

machine marks add

Put a mark on a branch. See Marks.

Marking a branch again with the same word succeeds. A word that another branch in the tree already holds is refused. The mark stays with that branch.

1
machine marks add [-h] run mark
Argument Does
run Run name
mark The mark to add

Examples:

1
machine marks add run1 good  put the mark good on run1

See also:

machine marks remove

Take a mark off a branch. See Marks.

The command is refused when the branch does not carry the mark.

1
machine marks remove [-h] run mark
Argument Does
run Run name
mark The mark to remove

Examples:

1
machine marks remove run1 good  take the mark good off

See also:

machine branches

Show the branch tree. See The tree.

The tree shows how each branch forked from its parent. Use --running to show only running branches.

The listing shows the newest 25 branches. Use --limit for a different page size. Use --depth to keep one level of the tree, 0 for the roots. Naming any run in a tree, root or branch, limits the listing to that tree.

1
2
3
machine branches [-h] [--running] [--depth n] [--since date|age] [--until date|age]
[--sort field] [--reverse] [--limit n] [--offset n] [-w]
[run]
Argument Does
[run] Show only this run’s tree
Option Does
--running Show only running branches
--depth n Show only branches at this depth from the root, 0 is the root
--since date|age Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--until date|age Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--sort field Sort field: created, name (default: created)
--reverse Reverse sort order
--limit n Show at most n branches (default: 25)
--offset n Skip the first n branches
-w, --watch Watch for changes

Examples:

1
2
3
machine branches            the branch tree
machine branches run1 only the tree of run1
machine branches --running running branches only

Prints:

1
2
3
4
each run     <run> (<id>) [<first entry>, <end>)
the end is open while the run can still record
each branch @<entry> <vtime> before it: where it was forked
this run the run named on the command line

See also:

machine lineage

Show the ancestors of a branch, from the root down. See The tree.

1
machine lineage [-h] run
Argument Does
run Run name

Examples:

1
machine lineage try2  the ancestors of try2

Prints:

1
each ancestor  <run> (<id>) [<first entry>, <end>) <n> checkpoints

See also:

machine entries

Show run log entries. See Inspecting a run.

An entry is one recorded event of the run, such as a vCPU run, an interrupt or a device input.

Use -n to select entries, and -f to follow new output.

1
machine entries [-h] [-n range] [-f] run
Argument Does
run Run name
Option Does
-n, --entries range Entries: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end
-f, --follow Follow new output (like tail -f)

Examples:

1
2
3
machine entries run1 -n -20      the last 20 entries
machine entries run1 -n 100:120 entries 100 to 120
machine entries run1 -f follow new entries

Prints:

1
2
3
4
5
6
7
8
9
#<id>     the entry, then its fields as name=value
ic rcb instructions and conditional branches of this entry
total_ic instructions since the machine started
vtime machine time in nanoseconds
RUN one iteration: its mode, limit, exit and where it stopped
IRQ an interrupt delivered, with its source and vector
IPI is one sent to a vCPU, which is how machine irq arrives
flags what else the entry carries, such as MEM_HASH
exit why the iteration ended: IO, HYPERCALL, INSTR_LIMIT

See also:

machine runs

List runs, newest first. See Inspecting a run.

Options filter by state, label, flag, disk, sync state and age.

The listing shows the newest 25. Use --limit for a different page size. Naming a run limits the listing to its tree. Use --roots for one row per tree, and --wide for more columns.

1
2
3
4
5
machine runs [-h] [--state [^]state] [--running] [--roots] [--label name=value]
[--flag name[=value]] [--disk name|id] [--parents mode] [--synced |
--not-synced] [--since date|age] [--until date|age] [--sort field] [--reverse]
[--limit n] [--offset n] [--no-trunc] [--wide] [--size] [-w]
[root]
Argument Does
[root] Show one tree, by its root
Option Does
--state [^]state Filter by state: running, paused, recovering, read-only, stopped, killed, terminated, died, failed (repeat = OR; ^ excludes)
--running Show only running runs, the same as --state running
--roots One row per tree instead of every run, with its running N/M
--label name=value Filter by a label (repeatable; matches all)
--flag name[=value] Filter by an option the run was created with, such as daemon or rng; name=0 for one it was not (repeatable; matches all)
--disk name|id Filter by the disk a run was created from
--parents mode Filter by where a run reads the runs it branched from: local for this session, remote for shared storage
--synced Show only runs published to shared storage
--not-synced Show only runs with nothing published
--since date|age Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--until date|age Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--sort field Sort field: created, name, cpus, memory, rng-seed, limit (default: created)
--reverse Reverse sort order
--limit n Show at most n runs (default: 25)
--offset n Skip the first n runs
--no-trunc Show full run ids
--wide Show the DEPTH, DURABLE and PIN columns
--size Measure and show size
-w, --watch Watch for changes

Examples:

1
2
3
4
5
6
machine runs                     the newest runs
machine runs --running running runs only
machine runs --state '^stopped' all but stopped runs
machine runs run1 the tree of run1
machine runs --roots one row per tree
machine -o json runs --limit 5 five runs as JSON

Prints:

1
2
3
4
5
6
text      NAME STATE CPUS MEMORY DISK OPTIONS SESSION CREATED ID
json name run_id state home disk nrcpus ram_bytes created_at
states running paused recovering read-only stopped
killed terminated died failed switched
switched the machine left this run for a branch
no machine is on a switched run

See also:

machine describe

Show the full configuration of a run. See Inspecting a run.

The configuration covers the state, resources, options and lineage of the run.

1
machine describe [-h] run
Argument Does
run Run name

Examples:

1
2
machine describe run1  configuration and lineage of run1
machine describe try2 a branch is named like any run

Prints:

1
text  Name ID State Started Resources Options Lineage

See also:

machine state

Show the state of a run and of its machine. See Inspecting a run.

The command shows the activity, machine time, entry and resources of the machine.

1
machine state [-h] run
Argument Does
run Run name

Examples:

1
machine state run1  state, activity and position of run1

Prints:

1
2
3
4
5
6
7
running   state activity vtime entry cpus vcpu memory host cpu
stopped state vtime entry cpus memory, and halted for a halted machine
activity idle sched_wait guest io mmio halted stopping
paused starting, where sched_wait is waiting for a command
switch the task switch the machine stopped at, when at one
<tid> -> <tid>, with in or out for the focused side
plan the clauses of the plan and the tally of its last run

See also:

machine now

Show where a run is. See Inspecting a run.

The entry and the vtime are points for go, rewind, fork --at and detach --at. Nothing moves.

Machine time does not move for work too short to count. A +N after the time tells apart the entries that share it. 0.2s is the first entry at that time and 0.2s+2 is the third.

A time with +N is a point like any other. It names exactly one entry.

1
machine now [-h] run
Argument Does
run Run name

Examples:

1
machine now run1  entry, vtime and RIP of run1

Prints:

1
2
3
4
5
text  vtime=<time> entry=<id> vcpu=<n> rip=<address> <event> hash=<hash>
the event as entries names it, IO_OUT, and its state hash
json entry_id vtime vtime_offset vcpu rip event state_hash
vtime in nanoseconds, rip as a decimal number
vtime_offset is the N of +N

See also:

machine output

Show the output of a run. See Inspecting a run.

The output is everything the machine printed, in the order it was printed. Each line has a source. The console is one source, and it is the one shown.

The serial ports ttyS1 to ttyS3 are sources too. A program in the guest that writes to /dev/ttyS1 prints on ttyS1. Use --source to read another source. The sources subcommand lists the sources that printed.

An output file is a source, named by its path. A program in the guest writes its files under the directory that MACHINE_OUTPUT_DIR names. A file events.jsonl that the service app writes there is /output/keep/app/events.jsonl. The path of a directory is every file under it.

The directory that MACHINE_DROP_OUTPUT_DIR names is for output too large to store. A file under it has a name and no bytes. A branch forked with --keep-output stores the file when replayed.

The name all is every source in the order printed, each line after the name of its source. A source that prints ends a line another source left unfinished.

Use --entries to select the output of given entries, -n to select lines, and -f to follow new output. An entry that printed nothing contributes nothing.

The command prints lines and does not search them. Pipe it to grep to find text, or to less to page through it. Use -n : to hand them every line.

On a terminal each line shows the entries and the source that printed it, and its number. ENTRIES is the first and the last entry of the line. Into a pipe or a file the lines are printed alone. Use --wide or --plain to choose.

The entry subcommand gives the one entry that printed a byte. Use it for the exact entry of a match.

1
2
machine output [-h] [-n range] [--entries range] [-f] [--wide] [--plain] [--source name]
run command ...
Argument Does
run Run name
Option Does
-n, --lines range Lines: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end
--entries range Output of entry N, or of entries A:B, both ends included
-f, --follow Follow new output (like tail -f)
--wide Show the entries and the source that printed each line, and its number (default on a terminal)
--plain Print the lines alone (default into a pipe or a file)
--source name Source to read: console, all, ttyS0 to ttyS3, hvc0, debugcon, or the path of an output file or directory under /output (default: console)

Examples:

1
2
3
4
5
6
7
8
9
10
11
machine output run1                                    the last lines
machine output run1 --plain the lines alone
machine output run1 -f follow new output
machine output run1 -n : | grep -n login find text
machine output run1 -n : | less -RN page through it
machine output run1 -n : | grep -bo login find an offset
machine output run1 -n : --wide | grep login text and its entries
machine output run1 --source ttyS1 what ttyS1 printed
machine output run1 --source all every source
machine output run1 --source /output/keep/app/app.log an output file
machine output run1 --source /output/keep every file under it

Prints:

1
2
3
4
5
6
7
8
9
10
text              ENTRIES SOURCE LINE OUTPUT on a terminal
the lines as the machine printed them into a pipe or a file
no output stored No output stored for <path>, for a file with a name and no bytes
--source all each line after its source, such as ttyS0
--plain the lines alone, on a terminal too
ENTRIES <first>:<last>, the entries that printed the line
SOURCE the source that printed the line, such as ttyS0
LINE the line's number, as -n counts them
json lines entry_ranges sources start_line total_lines partial_last
with -f, an object a line: line text and its labels

See also:

machine output entry

Show the entry that printed a byte of the output. See Finding a line.

A line is printed over a range of entries. A byte is printed by one. The command takes the offset of a byte and prints its entry.

The offset counts bytes from 0 over what the console printed. The grep option -b prints the offset of a match. The first number of its output is the operand.

Use --source for a byte another source printed. The offset then counts within the output of that source.

The entry is a point. Use it with go, fork, rewind and the other commands that take an entry.

1
machine output run entry [-h] [--source name] offset
Argument Does
offset Byte offset in the output of the source, counted from 0
Option Does
--source name Source to read: console, all, ttyS0 to ttyS3, hvc0, debugcon, or the path of an output file or directory under /output (default: console)

Examples:

1
2
3
machine output run1 entry 48213             the entry of byte 48213
machine output run1 entry 0 the entry of the first byte
machine output run1 entry 0 --source ttyS1 the first byte ttyS1 printed

Prints:

1
2
text  the entry, alone
json entry_id source offset line

Refuses, and the way past:

1
an offset past the end  machine output run1 -n :

See also:

machine output sources

List the sources that printed and how much each printed.

A source is a serial port such as ttyS1, the virtio console, or an output file. The command prints a row for each source that printed, in the order each first printed.

The lines of a source are counted in its own output. Use the name with --source to read that source.

An output file with nothing stored is listed after them with 0 bytes. It is a file the run drops, or a file that was not written. Use its name with --keep-output on a fork to store it.

The list is made where the output is kept, so the output is not downloaded. The output of the ancestors of the run is counted.

1
machine output run sources [-h]

Examples:

1
2
machine output run1 sources          each source that printed
machine -o json output run1 sources the same as JSON

Prints:

1
2
3
4
5
6
text     SOURCE BYTES LINES ENTRIES, a row for each source
SOURCE the name --source takes, such as ttyS1
LINES the lines of the source in its own output
ENTRIES <first>:<last>, the entries of its first and last byte
- for an output file with nothing stored
json source bytes lines entry_id last_entry_id stored

See also:

machine output bytes

Write the bytes a source printed, exactly as printed.

The command writes the bytes to standard output and changes none. Use it for an output file that is not text.

With the json format the command prints a row for each write instead of the bytes. A row has the entry that wrote, the source, the offset and the length. The offset counts within the output of the source of the row. A row also names the task that wrote.

The bytes are read where the output is kept. The output of the ancestors of the run comes first.

1
machine output run bytes [-h] [--source name]
Option Does
--source name Source to read: console, all, ttyS0 to ttyS3, hvc0, debugcon, or the path of an output file or directory under /output (default: console)

Examples:

1
2
machine output run1 bytes --source /output/keep/app/a.bin  a file
machine -o json output run1 bytes --source /output/keep each write

Prints:

1
2
3
text  the bytes, unchanged
json entry_id source offset length tgid tid task
a row for each write, with the task that wrote it

See also:

Find a pattern in the output and the entry of each match. See Finding a line.

The pattern is a regular expression. A match is within a line. Each match is printed with the entry that printed its first byte and the time of that entry.

A match has at least one character, since an entry printed it. A pattern that matches without any text is refused. The pattern 6* is one, and 6+ is not.

The search is done where the output is kept, so a large output is not downloaded. Its lineage is searched, from the first line an ancestor printed.

Use --from and --to to search the output between two points. A match begins at --from or after it and ends at --to or before it.

Use -f to go on with what is printed next. Use --max to stop after a number of matches.

The console is searched. Use --source to search another source. With all, every source is searched and a match is still within the line of one source.

The entry and the time are points. Use either with go, fork, rewind and the other commands that take a point.

1
2
3
machine output run search [-h] [-f] [--max n] [--from point [point ...]]
[--to point [point ...]] [--source name]
regex
Argument Does
regex Regular expression to find
Option Does
-f, --follow Go on with new output (like tail -f)
--max n Stop after n matches
--from point [point ...] Point the search begins at: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, parent [n] or root (default: the start of the output)
--to point [point ...] Point the search ends at, as --from takes one (default: the end of the output)
--source name Source to read: console, all, ttyS0 to ttyS3, hvc0, debugcon, or the path of an output file or directory under /output (default: console)

Examples:

1
2
3
4
5
6
7
machine output run1 search login                each match and its entry
machine output run1 search 'error|panic' -f go on, live
machine output run1 search login --max 1 the first match
machine output run1 search x --from ^a --to 1s between two points
machine output run1 search x --from '#42' from an entry on
machine -o json output run1 search login matches as JSON
machine output run1 search ready --source all search every source

Prints:

1
2
3
4
5
6
7
8
text    ENTRY TIME SOURCE LINE:COL TEXT, a row for each match
ENTRY the entry that printed the first byte of the match
TIME the time of that entry, a point such as 1.194000000s+2
SOURCE the source that printed the match, such as ttyS0
TEXT the whole line: the match bold, the part past --to dim
json entry_id last_entry_id vtime vtime_offset point recorder
source line offset length text column column_end in_range
with -f, an object a line

Refuses, and the way past:

1
2
3
a pattern that matches no text, as 6*  machine output run1 search '6+'
-f with --to machine output run1 search x -f
--from after --to machine output run1 search x --to 1s

See also:

machine dump

Dump the state of a machine. See Inspecting a run.

The sections cover vCPU code, registers, stack and page tables, timers, interrupt controllers, devices, memory and hashes. Use --show to choose sections. Use --entry to dump the state recorded at an entry.

Without --show, every section the machine has is printed. Use --mem to print memory alone, or with --show beside other sections. Use -o json for one object per section.

The hash section is what the entry recorded, before its exit was handled. Memory and the disk are 0 there unless the run hashes them. Use machine hash for the hashes after the entry, with its exit handled.

1
machine dump [-h] [--entry n] [--show sections] [--mem addr] [--vcpu n] run
Argument Does
run Run name
Option Does
--entry n Entry whose recorded state to dump, instead of the state now
--show sections One or more, comma-separated: code, regs, stack, pt, instr, trace, events, timers, debug, xsave, irq, pic, ioapic, devs, mem, disk, hash, entry, io, msrs, fullhash, all
--mem addr Dump guest memory at an address: pADDR physical, vADDR virtual in the page tables of the vCPU
--vcpu n The vCPU whose state and page tables are read (default: 0)

Examples:

1
2
3
4
machine dump run1                      vCPU state at the current entry
machine dump run1 --entry 5 the state recorded at entry 5
machine dump run1 --show code,regs only code and registers
machine -o json dump run1 --show regs registers as JSON

See also:

machine hash

Show the state hashes of the machine. Each is computed as it stands. See Comparing two machines.

The command prints a hash of memory. It prints a hash of the disk. Each device has a hash too. The interrupts held by --irq-delay have their own hash, irq_pending.

Each vCPU has a hash of each part of its state. Each vCPU also has a hash of the whole machine. Each hash compares with another machine’s at the same point, such as a replay’s.

A hash is not the one dump --show hash prints. Dump prints the hashes the newest entry recorded, before its exit was handled. This command computes them after that entry, with its exit handled. Memory and the disk are in its hashes even when the entry hashed neither.

The hashes are computed whether or not the run hashes memory and the disk at each entry. Memory is read from all of RAM.

The vCPU hashes are of each vCPU now on a host kernel that computes them. Elsewhere they are of each vCPU’s last exit. The command then says so.

The command changes nothing. The run records nothing, and the machine stays where it is.

The command waits until every hash is computed, however long reading RAM takes. The machine refuses commands that would change it until then. Use Ctrl-C or machine cancel to stop it.

1
machine hash [-h] [--vcpu n] run [hashes]
Argument Does
run Run name
[hashes] One or more, comma-separated: vcpu, devices, mem, disk, irq_pending, or a component such as regs or rtc (default: all)
Option Does
--vcpu n Show the hashes of this vCPU alone

Examples:

1
2
3
4
machine hash run1                every hash of the machine now
machine hash run1 mem,disk memory and the disk
machine hash run1 vcpu --vcpu 1 the hashes of vCPU 1
machine -o json hash run1 the hashes as JSON

See also:

machine vcpu

Show what a vCPU is doing and how its last dispatch ended.

A dispatch is one entry into the machine. It runs until its limit or until the machine exits to the host.

1
machine vcpu [-h] [--vcpu n] run
Argument Does
run Run name
Option Does
--vcpu n The vCPU (default: 0)

Examples:

1
2
machine vcpu run1           run state of vCPU 0
machine vcpu run1 --vcpu 1 run state of vCPU 1

Prints:

1
2
3
4
5
6
7
first line     entry ready pending iter
ready the machine accepts commands
pending a run is in progress
last dispatch VCPU EXIT ID RUN IC RCB TOTAL IC REGS HASH RIP RCX ITER
RUN IC, RCB instructions and conditional branches of the dispatch
TOTAL IC instructions since the run started
ITER the iteration the dispatch was

See also:

machine ops

List operations on runs and how they finished. See Inspecting a run.

The listing shows the newest 25 operations in every state. Use --limit for a different page size. Use --state to show some states.

An operation outlives the command that started it. One interrupted with Ctrl-C is still listed here by its id. The next start of a run finishes its unfinished rewind.

Operations on the account’s storage are listed by machine storage ops.

1
2
3
machine ops [-h] [--op-id id] [--state [^]state] [--type [^]type] [--since date|age]
[--until date|age] [--limit n] [--offset n] [--no-trunc] [-w]
[run]
Argument Does
[run] Show operations on one run
Option Does
--op-id id Show one operation, by its id (ignores --state and --type)
--state [^]state Filter by state: running, unfinished, done, failed (default: all; repeat = OR; ^ excludes)
--type [^]type Filter by type: checkpoint, branch, switch, rewind, start, delete (default: all; repeat = OR; ^ excludes)
--since date|age Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--until date|age Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m
--limit n Show at most n operations (default: 25)
--offset n Skip the first n operations
--no-trunc Show full run ids
-w, --watch Watch for changes

Examples:

1
2
machine ops       the newest operations
machine ops run1 those of run1

See also:

machine debug

Show debug or error output from a run. See Inspecting a run.

The log subcommand shows the debug output. The err subcommand shows errors and crashes.

1
machine debug [-h] command ...

Examples:

1
2
3
4
machine debug log run1         the log of run1
machine debug log run1 -n -20 its last 20 lines
machine debug log run1 -f follow it
machine debug err run1 its error log

See also:

machine debug log

Show the debug log of a run. See Inspecting a run.

Use -n to select lines, and -f to follow new output.

1
machine debug log [-h] [-n range] [-f] run
Argument Does
run Run name
Option Does
-n, --lines range Lines: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end
-f, --follow Follow new output (like tail -f)

Examples:

1
2
3
machine debug log run1         the log of run1
machine debug log run1 -n -20 its last 20 lines
machine debug log run1 -f follow it

See also:

machine debug err

Show the error and crash log of a run. See Inspecting a run.

Use -n to select lines, and -f to follow new output.

1
machine debug err [-h] [-n range] [-f] run
Argument Does
run Run name
Option Does
-n, --lines range Lines: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end
-f, --follow Follow new output (like tail -f)

Examples:

1
2
3
machine debug err run1         the error log of run1
machine debug err run1 -n -20 its last 20 lines
machine debug err run1 -f follow it

See also:

machine tasks

List the tasks the guest runs. See Looking at tasks.

A task is what the guest kernel schedules: a thread of a program, or the kernel’s own. Each task has a tid. Its tgid names its thread group, the process it belongs to.

Each row is one task, by its tid, the thread id ps -L shows. Its tgid is the pid of its process. Use --vcpu to list the tasks of one vCPU. Use --focused to list the tasks that focus steers.

The subcommands steer the scheduler, so an order of tasks can be tried. The hold, yield and release subcommands control whether a task runs.

The slice subcommand sets how long a task runs. The pin subcommand places it on a vCPU. The cgroups subcommand groups the list. The exits subcommand lists the tasks that left.

1
machine tasks [-h] [--vcpu n] [--focused] run command ...
Argument Does
run Run name
Option Does
--vcpu n Only tasks on this vCPU
--focused Only the tasks focus is steering

Examples:

1
2
machine tasks run1            the tasks in run1's machine
machine tasks run1 --focused only the focused tasks

Prints:

1
2
3
4
5
6
7
8
9
10
11
TID TGID        the task, and the tgid it belongs to
PID the tgid as its container names it, blank for the machine's own
COMM EXE the name of the task, and the path of its program
CPU the vCPU the task is on, blank when it is on none
NEXT whose code runs next on that vCPU: user or kernel
KIND user for a program, kernel for the kernel's own
FOCUS ticked for a task that focus steers
HOLD YIELD PIN ticked when hold, yield or pin is set on the task
SLICE the time set with slice
json also cgroup, and exe under the same name
generation a count that changes when the tasks do

Refuses, and the way past:

1
a machine not yet up  machine run run1 --until tasks

See also:

machine tasks list

List the tasks.

The listing is the one the command prints with no verb. Use --vcpu to list the tasks of one vCPU.

1
machine tasks run list [-h] [--vcpu n] [--focused]
Option Does
--vcpu n Only tasks on this vCPU
--focused Only the tasks focus is steering

Examples:

1
2
machine tasks run1 list           every task
machine tasks run1 list --vcpu 0 the tasks of vCPU 0

See also:

machine tasks cgroups

List the tasks grouped by cgroup. See Holding threads.

Use --vcpu to list the tasks of one vCPU.

1
machine tasks run cgroups [-h] [--vcpu n]
Option Does
--vcpu n Only tasks on this vCPU

Examples:

1
2
machine tasks run1 cgroups           tasks grouped by cgroup
machine tasks run1 cgroups --vcpu 1 tasks of vCPU 1

See also:

machine tasks exits

List the tasks that exited and how each one ended. See Holding threads.

The list is kept across checkpoints and replays. It is bounded and says how many it dropped.

1
machine tasks run exits [-h]

Examples:

1
machine tasks run1 exits  tasks that left, newest last

Prints:

1
2
3
4
5
text     TID TGID COMM OUTCOME
OUTCOME status <n>, or killed by <signal>
json tid tgid comm exit_code signal status outcome
head how many tasks have exited since the machine started
lost how many of the oldest are no longer kept

See also:

machine tasks exist

Assert that the machine runs a program, a comm, a tid or a tgid.

The command lists the tasks that match and is refused when none does. It takes the words inject add takes for --in, except all.

A program is the absolute path of the file it runs, the EXE that tasks shows. A symlink’s path matches no task.

The command reads the task table. It changes nothing in the machine and the run records nothing.

1
machine tasks run exist [-h] path|comm=name|tid=n|tgid=n
Argument Does
path|comm=name|tid=n|tgid=n Program by absolute path, comm, tid or tgid

Examples:

1
2
3
machine tasks run1 exist /usr/bin/app  is /usr/bin/app running
machine tasks run1 exist comm=cat a task named cat
machine tasks run1 exist tgid=412 tgid 412

See also:

machine tasks hold

Take a task off the runqueue. See Holding threads.

The scheduler cannot pick a held task. A hold takes effect at the next scheduling decision of the vCPU. Holding the last runnable task leaves nothing to run. Release it, then use the irq command with preempt to go on.

1
machine tasks run hold [-h] tid
Argument Does
tid Task, by its tid

Examples:

1
machine tasks run1 hold 56  hold tid 56

See also:

machine tasks yield

Put a task behind its peers. See Holding threads.

The task stays runnable. It does nothing when no peer is runnable.

1
machine tasks run yield [-h] tid
Argument Does
tid Task, by its tid

Examples:

1
machine tasks run1 yield 56  put tid 56 behind its peers

See also:

machine tasks release

Undo a hold or a yield. See Holding threads.

A release takes effect at the next scheduling decision of the vCPU. Use the irq command with preempt to force one.

1
machine tasks run release [-h] tid
Argument Does
tid Task, by its tid

Examples:

1
machine tasks run1 release 56  undo hold and yield

See also:

machine tasks slice

Set how long a task runs between decisions. See Holding threads.

A decision is a scheduling decision of the vCPU. Give a duration with a unit, such as 10us, 0.6ms or 750000ns. Use default for the kernel’s own. A slice survives release.

1
machine tasks run slice [-h] tid duration|default
Argument Does
tid Task, by its tid
duration|default Duration with a unit, or default

Examples:

1
2
machine tasks run1 slice 56 600us    run tid 56 for 600 us
machine tasks run1 slice 56 default the kernel's own slice

See also:

machine tasks pin

Place a task on one or more vCPUs when it next wakes. See Holding threads.

A set of vCPUs is one number, 1, or a range, 1-3, both ends included.

Free means any vCPU. None means no vCPU, which stops the task running.

Pin is a placement policy, not confinement. A task already queued elsewhere keeps that vCPU until it wakes.

1
machine tasks run pin [-h] tid set|free|none
Argument Does
tid Task, by its tid
set|free|none Set of vCPUs to place the task on, such as 1 or 0-2; free for any, none for no vCPU

Examples:

1
2
machine tasks run1 pin 56 2     place tid 56 on vCPU 2
machine tasks run1 pin 56 free place it on any vCPU

See also:

machine tasks signal

Send a task a signal.

The guest sends the signal at its next entry from user mode, as GDB’s signal command delivers one. The task acts on it when it next runs. The run log keeps the request, so a replay sends it at the same instruction.

1
machine tasks run signal [-h] tid signal
Argument Does
tid Task, by its tid
signal Signal by name, such as SIGTERM or TERM, or by number

Examples:

1
2
machine tasks run1 signal 56 SIGTERM  send tid 56 SIGTERM
machine tasks run1 signal 56 USR1 the name without SIG

See also:

machine syscalls

List the system calls, by number. See Looking at tasks.

A name filters the list to calls whose name contains that text.

A task shows its syscall histogram instead. Name a task by tid, tgid, absolute path, cgroup=path or comm=name. The histogram counts calls made while the focus entry of the task was armed with focus add <path> +syscalls.

1
machine syscalls [-h] run [name|tid|tgid|path]
Argument Does
run Run name
[name|tid|tgid|path] Filter the list, or a task to show its histogram

Examples:

1
2
3
machine syscalls run1        the system calls of the machine
machine syscalls run1 read calls with read in the name
machine syscalls run1 /test the calls counted for /test

See also:

machine ints

List the interrupt vectors. See Looking at tasks.

One row per IDT vector that the machine has a handler for. The vector is in hex, as the table is indexed. A name filters to handlers whose name contains that text.

A task shows its interrupt histogram instead. Name a task by tid, tgid, absolute path, cgroup=path or comm=name. The histogram counts interrupts taken while the focus entry of the task was armed with focus add <path> +ints.

1
machine ints [-h] run [name|tid|tgid|path]
Argument Does
run Run name
[name|tid|tgid|path] Filter the list, or a task to show its histogram

Examples:

1
2
3
machine ints run1        the interrupt vectors of the machine
machine ints run1 timer handlers with timer in the name
machine ints run1 /test the interrupts counted for /test

See also:

machine focus

Show and change which tasks a machine steers. See Focus.

The focus list names the programs the machine steers. A run stops at each context switch of a focused program, and where the program reaches its own code. The tasks subcommands then decide which task runs next.

A program is named by its absolute path, matched exactly. Write cgroup=<path> to match the cgroup of a task. Write tgid=<n> to name a tgid. The tgid of a task is shown by tasks.

Write comm=<name> to focus each process with a task of that comm, as inject add and tasks exist read it. A comm names a busybox applet or a script, which a path cannot. A task renamed after exec is seen at the next fork, exec or exit.

Use add and remove to change the list. Use clear to empty it.

1
machine focus [-h] run command ...
Argument Does
run Run name

Examples:

1
2
3
machine focus run1                            what is focused
machine focus run1 add /test +syscalls +ints count its calls and interrupts
machine focus run1 clear focus nothing

Prints:

1
2
3
4
5
6
Focus     each program or cgroup named, and what is counted for it
Steering each tgid a focus entry matches now
nothing yet, until a program that is named starts
Marks how many tasks are held, yield or have a slice
Switches context switches of steered tasks since the start
on I/O counts those where the task left to wait for I/O

See also:

machine focus add

Steer another program. See Focus.

The entry is a program path, cgroup=<path>, comm=<name> or tgid=<n>. Add +syscalls or +ints to count its system calls or interrupts.

1
machine focus run add [-h] path|cgroup=path|comm=name|tgid=n [+syscalls|+ints ...]
Argument Does
path|cgroup=path|comm=name|tgid=n Program path, cgroup, comm, or tgid to match
[+syscalls|+ints ...] Count this program’s syscalls or interrupts

Examples:

1
2
3
machine focus run1 add /test                  focus the program /test
machine focus run1 add /test +syscalls +ints count its calls and interrupts
machine focus run1 add comm=cat processes whose comm is cat

See also:

machine focus remove

Stop steering a program. See Focus.

Name the entry as it was added, by program path, cgroup=<path>, comm=<name> or tgid=<n>.

1
machine focus run remove [-h] path|cgroup=path|comm=name|tgid=n
Argument Does
path|cgroup=path|comm=name|tgid=n Program path, cgroup, comm, or tgid to match

Examples:

1
machine focus run1 remove /test  stop focusing /test

See also:

machine focus clear

Empty the list of steered programs. See Focus.

1
machine focus run clear [-h]

Examples:

1
machine focus run1 clear  focus nothing

See also:

machine send

Send input to a device of a run.

The command writes bytes to a serial port, the virtio console or the keyboard of a run. A program in the guest reads them from the device, as it reads what a person types.

The text is sent as written, and a newline follows it, which is Enter. Use -n to send no newline. Use -e to read escapes in the text, so that Ctrl-C is \x03.

The text - sends standard input as it is. The command adds no newline to it and reads no escape in it.

At the control prompt the command is /send. The text is the rest of the line after the options, with its quotes. A semicolon inside quotes does not end the command. A pair of quotes around the whole text keeps the spaces at its ends.

The keyboard takes text and types it on a US keyboard. The machine presses and releases a key for each character. It holds Shift or Ctrl where the character needs one.

A device that begins with a slash is the path of a file in the machine. The bytes go at the end of the file. A pipe is such a file. A program that reads its input from a pipe is sent input this way.

A process of the machine writes the file, and the command prints its id. The exec proc subcommand says whether the bytes were written.

The command does not run the machine. It queues the bytes and returns. The device takes them the next time the run executes, and the run records them there. A replay reproduces them.

Use fork to send other input than a run recorded. The branch takes its own.

1
machine send [-h] [-n] [-e] [--device name] run text
Argument Does
run Run name
text Text to send, or - to send standard input as it is
Option Does
-n Send no newline after the text
-e Read the escapes \n, \r, \t, \b, \e, \0, \ and \xNN in the text
--device name Device to send to: console, ttyS0 to ttyS3, hvc0 or keyboard, or the path of a file in the machine (default: console)

Examples:

1
2
3
4
5
6
7
8
machine send run1 ls                        type ls and Enter on the console
machine send run1 ls --device ttyS1 the same on ttyS1
machine send -n run1 y send y with no newline
machine send -n -e run1 '\x03' send Ctrl-C
machine send run1 - send standard input as it is
machine send run1 ls --device keyboard type it on the keyboard
machine send run1 -- -la a text that begins with a dash
machine send run1 c --device /dev/gdb_pipe write a pipe

Prints:

1
2
3
text     Queued <n> bytes for <device> of <run> (<id>)
json queued device run_id name process
process the id of the write to a path, empty for a device

Refuses, and the way past:

1
2
a machine that is replaying  machine fork run1 --name try2
a device with no room machine run run1

See also:

machine exec

Execute a command in the guest.

A command is a program of the machine’s disk and its arguments, written after two dashes. No shell reads it. Use -c to give a line to the shell of the disk, with its pipes and redirections.

The run subcommand executes it and runs the machine until it exits. The start subcommand sends it and returns.

Each command is a process with an id. The run records what the process prints and how it ends, each line at its entry. A replay reproduces them, and a branch has the processes of its parent.

Without a subcommand the command lists the processes of the run. The proc subcommand shows one.

The id is the entry the command was sent at and the process number, 23090p1. In the machine, its output is in /.machine/output/exec/ID/stdout and stderr. Its events are in /.machine/output/exec/ID/status. The output source /output/exec/ID/stdout names the first in output and run --until.

The machine’s disk runs the agent that executes the commands, /sbin/machine-exec. A disk without it executes none.

1
machine exec [-h] run command ...
Argument Does
run Run name

Examples:

1
2
3
4
machine exec run1                   the processes of run1
machine exec run1 run -- ls -la / execute ls and print its output
machine exec run1 start -- sleep 5 send it and return its id
machine exec run1 proc 40787p1 one process

Prints:

1
2
3
4
text     ID PID STATE ENTRIES COMMAND
STATE pending, running, exited <code>, killed <signal>
ENTRIES <started>:<ended>, the entries of the two events
json id state command pid exit_code signal started_entry

See also:

machine exec run

Execute a command and run the machine until it exits.

The command prints what the command printed, its standard output and its standard error each on its own. The exit status is that of the command.

Use --container to execute it in a container. The command then has the environment of the container. Use --user to choose the user it runs as there.

Use --net to run a program of the machine on a container’s network, such as tcpdump on its eth0. The program has the machine’s files and the container’s interfaces.

The machine runs until the command ends. Other programs of the machine run with it.

A command gets 30 minutes of machine time to end. Use --within to give it another. A command that has not ended by then stays, and the refusal names its id.

1
2
3
machine exec run run [-h] [-c line] [--container name] [--user user] [--net name]
[--within duration]
[command ...]
Argument Does
[command ...] The command and its arguments, after –
Option Does
-c line A line for the shell of the machine’s disk, in place of a command
--container name Container to execute it in, by its id, its name or its Compose service (default: the machine itself)
--user user User the command runs as in the container, such as root or 1000:1000
--net name Container whose network a program of the machine runs on, by its id, its name or its Compose service
--within duration Machine time the command gets to end in, such as 10s (default: 30m)

Examples:

1
2
3
4
5
machine exec run1 run -- ls -la /           execute ls and print its output
machine exec run1 run -c 'echo $PATH' a line for the shell
machine exec run1 run --container db -- ls in a container
machine exec run1 run --net db -- ip addr db's interfaces
machine -o json exec run1 run -- ls the process and its events

Prints:

1
2
text  what the command printed, and its exit status
json the process, with stdout stderr events

Refuses, and the way past:

1
2
3
a machine that is replaying           machine fork run1 --name try2
a command that has not ended in time machine exec run1 proc 40787p1
a container that is not running machine exec run1 run -- docker ps

See also:

machine exec start

Send a command to execute and return.

The command prints the id of the process. The machine does not run. The process starts the next time the run executes.

Use the proc subcommand with the id to read the process.

Use --container to execute it in a container. Use --net to run a program of the machine on a container’s network.

1
2
machine exec run start [-h] [-c line] [--container name] [--user user] [--net name]
[command ...]
Argument Does
[command ...] The command and its arguments, after –
Option Does
-c line A line for the shell of the machine’s disk, in place of a command
--container name Container to execute it in, by its id, its name or its Compose service (default: the machine itself)
--user user User the command runs as in the container, such as root or 1000:1000
--net name Container whose network a program of the machine runs on, by its id, its name or its Compose service

Examples:

1
2
3
4
machine exec run1 start -- sleep 5               send it and return its id
machine exec run1 start -- /app/load --rows 9 options after --
machine exec run1 start -c 'gdb -p 36 < /dev/p' input on a pipe
machine exec run1 start --net db -- tcpdump on db's network

Prints:

1
2
text  Sent <id> to <run> (<id>)
json id state command

Refuses, and the way past:

1
a machine that is replaying  machine fork run1 --name try2

See also:

machine exec proc

Show a process: its state, its output and its events.

The summary has the command, the pid, the state and how the process ended. A process is pending until the machine runs and the guest takes the request.

The events view lists every line the process printed and every change of its state, each with its entry. The stdout and stderr views print one stream. The exits view prints how the process ended.

In the machine, the streams are /.machine/output/exec/ID/stdout and /.machine/output/exec/ID/stderr.

1
machine exec run proc [-h] id [{summary,events,stdout,stderr,exits}]
Argument Does
id Process id
[{summary,events,stdout,stderr,exits}] What to show: summary, events, stdout, stderr or exits (default: summary)

Examples:

1
2
3
4
5
machine exec run1 proc 40787p1         its state and how it ended
machine exec run1 proc 40787p1 events every line and its entry
machine exec run1 proc 40787p1 stdout its standard output
machine exec run1 proc 40787p1 stderr its standard error
machine exec run1 proc 40787p1 exits how it ended, and where

Prints:

1
2
3
4
summary  Process Run Command Pid State Entries Stdout Stderr
events ENTRY STREAM TEXT, the stream stdout, stderr or status
exits ENTRY and exited <code> or killed <signal>
json the process, or the events of the view

Refuses, and the way past:

1
a process the guest has not taken  machine run run1

See also:

machine put

Put a file into the guest or a container.

The command copies a file on this computer to a path in the running machine. Use --container to put it in a container instead. A file of any size is put. The directories on the way to the path are made.

The file has its name only once all of it is written, so a program never reads part of it. A file already at the path is replaced.

The machine runs while the file goes in. Other programs of the machine run with it. The bytes are a recorded input, so a replay puts the same file.

The file is written to the disk before the command returns. A checkpoint taken after it has the file, and fs reads it there.

1
2
3
machine put [-h] [--container name] [--mode mode] [--owner uid:gid] [--no-pipe]
[--within duration]
run file path
Argument Does
run Run name
file File on this computer to put, or - for standard input
path Path the file gets in the machine or in the container
Option Does
--container name Container to put the file in, by its id, its name or its Compose service (default: the machine itself)
--mode mode Mode of the file, as octal digits (default: 644)
--owner uid:gid Owner of the file, as numbers (default: root)
--no-pipe Send the file over the serial port instead of the faster data channel
--within duration Machine time allowed each time the command runs the machine, such as 10s (default: 30m)

Examples:

1
2
3
4
5
machine put run1 ./a.conf /etc/a.conf      into the machine
machine put run1 ./t.sh /t.sh --mode 755 with a mode
machine put run1 ./d /d --container db into a container
machine put run1 ./d /d --owner 1000:1000 with an owner
machine put run1 - /tmp/in standard input

Prints:

1
2
text  Put <n> bytes at <path> of <run> (<id>)
json size path container mode owner process run_id name entry_id

Refuses, and the way past:

1
2
a machine that is replaying      machine fork run1 --name try2
a container that is not running machine exec run1 run -- docker ps

See also:

machine get

Get a file from the guest or a container.

The command copies a file of the running machine to a file on this computer. Use --container to get it from a container instead. A file of any size is copied.

The file is read as it is now, with what programs wrote and the disk does not have yet. The fs command reads the disk at a checkpoint instead.

The machine runs while the file comes out. Other programs of the machine run with it, so a file a program is writing can change on the way.

1
machine get [-h] [--container name] [--within duration] run path file
Argument Does
run Run name
path Path of the file in the machine or in the container
file File on this computer to write, or - for standard output
Option Does
--container name Container to get the file from, by its id, its name or its Compose service (default: the machine itself)
--within duration Machine time allowed each time the command runs the machine, such as 10s (default: 30m)

Examples:

1
2
3
machine get run1 /etc/a.conf ./a.conf   from the machine
machine get run1 /d ./d --container db from a container
machine get run1 /var/log/app.log - to standard output

Prints:

1
2
text  Got <n> bytes from <path> of <run> (<id>)
json size path container file mode owner process run_id name entry_id

Refuses, and the way past:

1
2
a machine that is replaying      machine fork run1 --name try2
a container that is not running machine exec run1 run -- docker ps

See also:

machine containers

List the containers, or stop, kill, crash, pause or limit one.

The command prints each container with its Compose service, its state and its image. Stopped containers are listed too. A container that no service started has no service.

A verb puts a fault on one container while the others run: a node of a cluster that stops, dies, loses power, freezes or starves. Throttle limits its CPU, and memory its memory. Pids, files and ports run it out of processes, open files and local ports.

A limit holds when the container starts again, until it is lifted. A container that Compose recreates starts without it. A limit set with --for is set back after the duration, whichever start of the container runs then.

Every command that takes a container takes its id, its name or its Compose service. An id is the surer way to name one. The machine knows an id as soon as Docker creates the container. It knows a name shortly after the container starts.

A running container is found with the machine stopped. A name not known yet is refused, and its id works. A service stands for its one running container. The start verb asks Docker for a stopped container.

A name is matched first, then a service, then an id. An id takes 12 hex digits or more, as docker ps prints it.

The machine runs until Docker has answered. The list is of the machine as it is then. The command runs processes in the machine, and the run records them, so a replay acts at the same point.

1
machine containers [-h] [--within duration] run command ...
Argument Does
run Run name
Option Does
--within duration Machine time the command gets to end in, such as 10s (default: 30m)

Examples:

1
2
3
machine containers run1                       the containers of run1
machine -o json containers run1 every field of each
machine containers run1 kill db --restart 5s db dies, back in 5s

Prints:

1
2
3
text   SERVICE CONTAINER ID STATE IMAGE ADDRESS LINK
STATE running, exited, created, paused
json name service project state status image id

Refuses, and the way past:

1
a machine that is replaying  machine fork run1 --name try2

See also:

machine containers stop

Stop a container.

The container’s processes get SIGTERM, and SIGKILL after the grace period. A program that handles SIGTERM exits on its own first.

The container stays down. Use --restart 5s to start it again after 5 seconds of machine time. Use --restart 0 to start it at once. Other containers keep running while it is down.

A container started again keeps its writable layer and its volumes. Use --fresh to start it from its image instead, as Compose recreates a service. Its volumes are kept.

1
2
machine containers run stop [-h] [--grace duration] [--restart duration|never] [--fresh]
container
Argument Does
container Container, by its id, its name or its service’s
Option Does
--grace duration Machine time between SIGTERM and SIGKILL, such as 5s (default: 10s)
--restart duration|never Start it again after this machine time, 0 at once (default: never)
--fresh Start it from its image, its writable layer discarded

Examples:

1
2
3
machine containers run1 stop db              db stops
machine containers run1 stop db --grace 2s SIGKILL after 2s
machine containers run1 stop db --restart 0 and starts again

Refuses, and the way past:

1
a container that is not running or paused  machine containers run1 start db

See also:

machine containers kill

Kill a container.

Every process of the container gets SIGKILL and no cleanup runs. What the programs wrote stays, since the guest’s kernel writes it out.

The container stays down. Use --restart 5s to start it again after 5 seconds of machine time. Use --restart 0 to start it at once. Other containers keep running while it is down.

A container started again keeps its writable layer and its volumes. Use --fresh to start it from its image instead, as Compose recreates a service. Its volumes are kept.

1
machine containers run kill [-h] [--restart duration|never] [--fresh] container
Argument Does
container Container, by its id, its name or its service’s
Option Does
--restart duration|never Start it again after this machine time, 0 at once (default: never)
--fresh Start it from its image, its writable layer discarded

Examples:

1
2
3
4
machine containers run1 kill db                      db dies
machine containers run1 kill db --restart 5s back in 5s
machine containers run1 kill db --restart 0 back at once
machine containers run1 kill db --restart 0 --fresh back from its image

Refuses, and the way past:

1
a container that is not running or paused  machine containers run1 start db

See also:

machine containers crash

Cut a container as a power cut cuts a node.

The container’s volumes lose the writes they had not synced, and it dies. The others run on. A volume keeps what was synced, and loses the writes still in memory.

A crash cuts the volumes the project’s x-machine names, each a filesystem of its own. Name one by its path in the container to cut it alone. A volume replays its journal when the container starts again.

The container is frozen first, and the cut is when its volumes stop writing to the disk. A write the kernel finishes between the two is kept. A volume that another running container mounts is refused, since the cut would fail its writes too.

The container stays down. Use --restart 5s to start it again after 5 seconds of machine time. Use --restart 0 to start it at once. Other containers keep running while it is down.

A container started again keeps its writable layer and its volumes. Use --fresh to start it from its image instead, as Compose recreates a service. Its volumes are kept.

1
machine containers run crash [-h] [--restart duration|never] [--fresh] container [path]
Argument Does
container Container, by its id, its name or its service’s
[path] A volume’s path in the container, to cut it alone (default: every volume)
Option Does
--restart duration|never Start it again after this machine time, 0 at once (default: never)
--fresh Start it from its image, its writable layer discarded

Examples:

1
2
3
machine containers run1 crash db              a power cut of db
machine containers run1 crash db --restart 0 and its reboot
machine containers run1 crash db /data that volume alone

Refuses, and the way past:

1
2
3
a container with no volume of its own      machine containers run1
a volume another running container mounts machine containers run1 stop side
a container that is not running or paused machine containers run1 start db

See also:

machine containers start

Start a container that is down.

Use --fresh to start it from its image, as Compose recreates a service. Its writable layer is discarded and its volumes are kept.

1
machine containers run start [-h] [--fresh] container
Argument Does
container Container, by its id, its name or its service’s
Option Does
--fresh Start it from its image, its writable layer discarded

Examples:

1
2
machine containers run1 start db          db runs again
machine containers run1 start db --fresh from its image

Refuses, and the way past:

1
2
a container that is running  machine containers run1
--fresh on an older disk machine disks build --compose ./p app

See also:

machine containers pause

Freeze a container.

The container’s processes are frozen using the cgroup freezer. Other containers see it as unresponsive. Use --for 5s to thaw it after 5 seconds of machine time. Use resume to thaw it now.

1
machine containers run pause [-h] [--for duration] container
Argument Does
container Container, by its id, its name or its service’s
Option Does
--for duration Thaw it after this machine time, such as 5s

Examples:

1
2
machine containers run1 pause db           db freezes
machine containers run1 pause db --for 5s for 5s

Refuses, and the way past:

1
a container that is not running  machine containers run1

See also:

machine containers resume

Thaw a frozen container.

1
machine containers run resume [-h] container
Argument Does
container Container, by its id, its name or its service’s

Examples:

1
machine containers run1 resume db  db thaws

Refuses, and the way past:

1
a container that is not paused  machine containers run1

See also:

machine containers list

List the containers.

The command prints each container with its Compose service, its state and its image. Each running one has its addresses, its links and the limits of its cgroup. The listing is the one the command prints with no verb.

1
machine containers run list [-h]

Examples:

1
machine containers run1 list  each container and its limits

See also:

machine containers throttle

Limit the CPU of a container.

The container’s cgroup gets a CPU quota of the number of CPUs given. A number of 0 sets back the container’s own limit, its Compose file’s or none. The limit holds when the container starts again.

Use --for 30s to restore the previous limit after 30 seconds of machine time.

1
machine containers run throttle [-h] --cpus n [--for duration] container
Argument Does
container Container, by its id, its name or its service’s
Option Does
--cpus n CPUs it may use, such as 0.1; 0 for the container’s own limit
--for duration Restore the previous limit after this machine time, such as 30s

Examples:

1
2
3
machine containers run1 throttle db --cpus 0.1            a tenth of a CPU
machine containers run1 throttle db --cpus 0.1 --for 30s for 30s
machine containers run1 throttle db --cpus 0 no limit

Refuses, and the way past:

1
a container that is not running  machine containers run1

See also:

machine containers memory

Limit the memory of a container.

The container’s processes may use the memory given and no more. A size of 0 sets back the container’s own limit, its Compose file’s or none. The limit holds when the container starts again.

A limit below what they use makes the guest kernel reclaim memory. What it cannot reclaim, its out-of-memory killer takes: a process of the container is killed.

A soft limit, with --soft, is the memory past which the kernel reclaims and slows the processes. It kills none of them, and the hard limit stays as it was.

Use --for 30s to restore the previous limit after 30 seconds of machine time.

1
machine containers run memory [-h] [--soft] [--for duration] container size
Argument Does
container Container, by its id, its name or its service’s
size Memory it may use, such as 256m or 1g; 0 for the container’s own limit
Option Does
--soft Reclaim and slow its processes past the size, and kill none
--for duration Restore the previous limit after this machine time, such as 30s

Examples:

1
2
3
machine containers run1 memory db 256m           db may use 256MB
machine containers run1 memory db 64m --for 30s for 30s
machine containers run1 memory db 0 no limit

See also:

machine containers pids

Limit the processes of a container.

The container’s processes and threads may number the count given and no more. A count of 0 sets back the container’s own limit, its Compose file’s or none. The limit holds when the container starts again.

Past the limit, a fork or a new thread in the container fails with EAGAIN. The processes already running go on.

Use --for 30s to restore the previous limit after 30 seconds of machine time.

1
machine containers run pids [-h] [--for duration] container count
Argument Does
container Container, by its id, its name or its service’s
count Processes and threads it may have; 0 for the container’s own limit
Option Does
--for duration Restore the previous limit after this machine time, such as 30s

Examples:

1
2
3
machine containers run1 pids db 50            50 processes and threads
machine containers run1 pids db 20 --for 30s for 30s
machine containers run1 pids db 0 no limit

See also:

machine containers ports

Narrow the ephemeral ports of a container.

A connection the container opens takes its local port from the range given. A range of 0 sets back the container’s own range, its Compose file’s or the kernel’s 32768-60999. The range holds when the container starts again.

Once every port of the range is taken for one destination, a connect to it fails with EADDRNOTAVAIL. Two connections to different destinations can share a port. A program that binds port 0 before it connects gets EADDRINUSE. A port stays taken after its connection closes, in TIME_WAIT for 60 seconds.

The range applies to every process on the container’s network. The command refuses a container on the machine’s network. It also refuses a container that uses another container’s network.

Use --for 30s to restore the previous range after 30 seconds of machine time.

1
machine containers run ports [-h] [--for duration] container range
Argument Does
container Container, by its id, its name or its service’s
range Ports it may connect from, such as 32768-32770; 0 for the container’s own
Option Does
--for duration Restore the previous limit after this machine time, such as 30s

Examples:

1
2
3
machine containers run1 ports api 40000-40009      10 ports
machine containers run1 ports api 40000 --for 30s 1 port, 30s
machine containers run1 ports api 0 the kernel's range

Refuses, and the way past:

1
2
a container on the machine's network  machine containers run1
one on another's network machine containers run1 ports db 40000

See also:

machine containers files

Limit the open files of a container.

Each process of the container may hold open the count of files given and no more. A socket or a pipe counts as a file. A count of 0 sets back the container’s own limit.

Past the limit, an open, a socket or an accept fails with EMFILE. A file already open stays open.

The limit is set on the processes that run now, and what they start takes it. A process docker exec starts takes the container’s own limit. The limit holds when the container starts again.

Use --for 30s to restore the previous limit after 30 seconds of machine time.

1
machine containers run files [-h] [--for duration] container count
Argument Does
container Container, by its id, its name or its service’s
count Files each process may hold open; 0 for the container’s own limit
Option Does
--for duration Restore the previous limit after this machine time, such as 30s

Examples:

1
2
3
machine containers run1 files api 64            64 open files
machine containers run1 files api 32 --for 30s for 30s
machine containers run1 files api 0 the container's own

See also:

machine net

Show the faults of each container’s link, or put faults on one.

A link connects a container to a peer. A fault acts on the packets the container sends. The command delays them, loses some, sends some twice, changes them, sends them out of order or limits their rate. It holds them and releases them together.

The command cuts two containers apart, one way or both. It resets their TCP connections. It shows and changes a container’s neighbour table.

A fault is on every packet the container sends, or on the packets a match takes. A match names a peer, a protocol, a port or TCP SYNs. ARP is a protocol a match names. A fault on one match leaves the container’s other links as they are.

The faults are part of the machine. The run records each change, and a replay makes it at the same point. A checkpoint keeps a link’s faults.

Every command that takes a container takes its id, its name or its Compose service, as containers does. The machine runs until the link is set.

1
machine net [-h] [--within duration] run command ...
Argument Does
run Run name
Option Does
--within duration Machine time the command gets to end in, such as 10s (default: 30m)

Examples:

1
2
3
4
5
6
7
machine net run1                               each link and its faults
machine net run1 delay db 100ms --jitter 20ms 80 to 120ms each
machine net run1 partition db api db and api cut apart
machine net run1 loss db 100% --port 5432 port 5432 alone
machine net run1 hold db --for 5s a stall, then a burst
machine net run1 reset api --to db api's connections to db
machine net run1 heal every link healed

See also:

machine net delay

Delay the packets a container sends.

Each packet waits the delay before it leaves, give or take the jitter. A delay of 0 takes the delay away. The link keeps its other faults.

Use --to db to put the fault on what the container sends to db alone. Use --proto, --port and --syn to take some packets alone, such as port 5432. A packet a match takes gets only the match’s faults, not the whole link’s.

The link draws from a seed, so a replay acts on the same packets. Use --seed to give one. A link keeps its seed, and a link without one draws a new seed.

Use --for 30s to set back what the command set after 30 seconds of machine time. The link keeps a change made in between.

Use --log to write each decision of the fault to /output/drop/.machine/net, a file for each container. The run drops that output unless it keeps it.

1
2
3
4
machine net run delay [-h] [--jitter duration] [--to container]
[--proto {tcp,udp,icmp,arp}] [--port n] [--syn] [--seed n]
[--for duration] [--log | --no-log]
container delay
Argument Does
container Container, by its id, its name or its service’s
delay Delay of each packet, such as 100ms
Option Does
--jitter duration Each packet’s delay differs by up to this, such as 20ms
--to container The packets to this container alone (default: to every peer)
--proto {tcp,udp,icmp,arp} The packets of this protocol alone
--port n The packets with this source or destination port alone
--syn The TCP SYNs that open a connection alone
--seed n Seed of the draws (default: the link’s seed, or one drawn)
--for duration Set the link back after this machine time, such as 30s
--log, --no-log Write each decision of the fault to the container’s log (default: as the link has it)

Examples:

1
2
3
4
5
machine net run1 delay db 100ms                db's packets wait 100ms
machine net run1 delay db 100ms --jitter 20ms 80 to 120ms
machine net run1 delay db 300ms --to api only to api
machine net run1 delay db 1s --port 5432 port 5432 alone
machine net run1 delay db 0 no delay

See also:

machine net loss

Lose some of the packets a container sends.

Each packet is lost by chance, the share given. A share of 0% takes the fault away. The link keeps its other faults.

Use --to db to put the fault on what the container sends to db alone. Use --proto, --port and --syn to take some packets alone, such as port 5432. A packet a match takes gets only the match’s faults, not the whole link’s.

The link draws from a seed, so a replay acts on the same packets. Use --seed to give one. A link keeps its seed, and a link without one draws a new seed.

Use --for 30s to set back what the command set after 30 seconds of machine time. The link keeps a change made in between.

Use --log to write each decision of the fault to /output/drop/.machine/net, a file for each container. The run drops that output unless it keeps it.

1
2
3
machine net run loss [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] [--syn]
[--seed n] [--for duration] [--log | --no-log]
container share
Argument Does
container Container, by its id, its name or its service’s
share Share of the packets, such as 10%
Option Does
--to container The packets to this container alone (default: to every peer)
--proto {tcp,udp,icmp,arp} The packets of this protocol alone
--port n The packets with this source or destination port alone
--syn The TCP SYNs that open a connection alone
--seed n Seed of the draws (default: the link’s seed, or one drawn)
--for duration Set the link back after this machine time, such as 30s
--log, --no-log Write each decision of the fault to the container’s log (default: as the link has it)

Examples:

1
2
3
4
5
6
machine net run1 loss db 10%               a tenth of db's packets lost
machine net run1 loss db 10% --log each decision logged
machine net run1 loss db 100% --for 5s db's packets lost for 5s
machine net run1 loss api 100% --syn no new connections
machine net run1 loss db 100% --proto arp db's ARP lost
machine net run1 loss db 0% none lost

See also:

machine net duplicate

Send some of a container’s packets twice.

Each packet is sent twice by chance, the share given. A share of 0% takes the fault away. The link keeps its other faults.

Use --to db to put the fault on what the container sends to db alone. Use --proto, --port and --syn to take some packets alone, such as port 5432. A packet a match takes gets only the match’s faults, not the whole link’s.

The link draws from a seed, so a replay acts on the same packets. Use --seed to give one. A link keeps its seed, and a link without one draws a new seed.

Use --for 30s to set back what the command set after 30 seconds of machine time. The link keeps a change made in between.

Use --log to write each decision of the fault to /output/drop/.machine/net, a file for each container. The run drops that output unless it keeps it.

1
2
3
machine net run duplicate [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n]
[--syn] [--seed n] [--for duration] [--log | --no-log]
container share
Argument Does
container Container, by its id, its name or its service’s
share Share of the packets, such as 10%
Option Does
--to container The packets to this container alone (default: to every peer)
--proto {tcp,udp,icmp,arp} The packets of this protocol alone
--port n The packets with this source or destination port alone
--syn The TCP SYNs that open a connection alone
--seed n Seed of the draws (default: the link’s seed, or one drawn)
--for duration Set the link back after this machine time, such as 30s
--log, --no-log Write each decision of the fault to the container’s log (default: as the link has it)

Examples:

1
2
machine net run1 duplicate db 5%  one packet in 20 sent twice
machine net run1 duplicate db 0% none sent twice

See also:

machine net corrupt

Change a bit of some of the packets a container sends.

Each packet has one bit flipped by chance, the share given. A checksum finds most of them. A share of 0% takes the fault away. The link keeps its other faults.

Use --to db to put the fault on what the container sends to db alone. Use --proto, --port and --syn to take some packets alone, such as port 5432. A packet a match takes gets only the match’s faults, not the whole link’s.

The link draws from a seed, so a replay acts on the same packets. Use --seed to give one. A link keeps its seed, and a link without one draws a new seed.

Use --for 30s to set back what the command set after 30 seconds of machine time. The link keeps a change made in between.

Use --log to write each decision of the fault to /output/drop/.machine/net, a file for each container. The run drops that output unless it keeps it.

1
2
3
machine net run corrupt [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n]
[--syn] [--seed n] [--for duration] [--log | --no-log]
container share
Argument Does
container Container, by its id, its name or its service’s
share Share of the packets, such as 10%
Option Does
--to container The packets to this container alone (default: to every peer)
--proto {tcp,udp,icmp,arp} The packets of this protocol alone
--port n The packets with this source or destination port alone
--syn The TCP SYNs that open a connection alone
--seed n Seed of the draws (default: the link’s seed, or one drawn)
--for duration Set the link back after this machine time, such as 30s
--log, --no-log Write each decision of the fault to the container’s log (default: as the link has it)

Examples:

1
2
machine net run1 corrupt db 1%  one packet in 100 changed
machine net run1 corrupt db 0% none changed

See also:

machine net reorder

Send some of a container’s packets ahead of the others.

Each packet is sent at once by chance, the share given, while the rest wait the link’s delay. A link to reorder needs a delay. A share of 0% takes the fault away.

Use --to db to put the fault on what the container sends to db alone. Use --proto, --port and --syn to take some packets alone, such as port 5432. A packet a match takes gets only the match’s faults, not the whole link’s.

The link draws from a seed, so a replay acts on the same packets. Use --seed to give one. A link keeps its seed, and a link without one draws a new seed.

Use --for 30s to set back what the command set after 30 seconds of machine time. The link keeps a change made in between.

Use --log to write each decision of the fault to /output/drop/.machine/net, a file for each container. The run drops that output unless it keeps it.

1
2
3
machine net run reorder [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n]
[--syn] [--seed n] [--for duration] [--log | --no-log]
container share
Argument Does
container Container, by its id, its name or its service’s
share Share of the packets sent ahead, such as 25%
Option Does
--to container The packets to this container alone (default: to every peer)
--proto {tcp,udp,icmp,arp} The packets of this protocol alone
--port n The packets with this source or destination port alone
--syn The TCP SYNs that open a connection alone
--seed n Seed of the draws (default: the link’s seed, or one drawn)
--for duration Set the link back after this machine time, such as 30s
--log, --no-log Write each decision of the fault to the container’s log (default: as the link has it)

Examples:

1
2
machine net run1 reorder db 25%  a quarter ahead of the delay
machine net run1 reorder db 0% in order again

See also:

machine net rate

Limit how fast a container sends.

The link sends at most the rate given, and the packets past it wait. A rate of 0 takes the limit away. The link keeps its other faults.

Use --to db to put the fault on what the container sends to db alone. Use --proto, --port and --syn to take some packets alone, such as port 5432. A packet a match takes gets only the match’s faults, not the whole link’s.

Use --for 30s to set back what the command set after 30 seconds of machine time. The link keeps a change made in between.

Use --log to write each decision of the fault to /output/drop/.machine/net, a file for each container. The run drops that output unless it keeps it.

1
2
3
machine net run rate [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] [--syn]
[--for duration] [--log | --no-log]
container rate
Argument Does
container Container, by its id, its name or its service’s
rate Bits a second, such as 1mbit or 500kbit; 0 for no limit
Option Does
--to container The packets to this container alone (default: to every peer)
--proto {tcp,udp,icmp,arp} The packets of this protocol alone
--port n The packets with this source or destination port alone
--syn The TCP SYNs that open a connection alone
--for duration Set the link back after this machine time, such as 30s
--log, --no-log Write each decision of the fault to the container’s log (default: as the link has it)

Examples:

1
2
machine net run1 rate db 1mbit  db sends 1 megabit a second
machine net run1 rate db 0 no limit

See also:

machine net partition

Cut the link between two containers.

Everything each container sends the other is lost, and their other links stay as they are. Use --one-way to cut what the first sends the second alone. Use --port 5432 to cut the packets of port 5432 alone.

Use --for 30s to set both directions back together after 30 seconds of machine time.

1
2
3
machine net run partition [-h] [--one-way] [--proto {tcp,udp,icmp,arp}] [--port n] [--syn]
[--for duration]
container other
Argument Does
container One container of the two
other The other container
Option Does
--one-way Cut what the first sends the second, not the reverse
--proto {tcp,udp,icmp,arp} The packets of this protocol alone
--port n The packets with this source or destination port alone
--syn The TCP SYNs that open a connection alone
--for duration Set the link back after this machine time, such as 30s

Examples:

1
2
3
4
machine net run1 partition db api              nothing between db and api
machine net run1 partition db api --one-way db to api cut
machine net run1 partition db api --for 10s for 10s
machine net run1 partition db api --port 5432 port 5432 alone

See also:

machine net list

List each container’s links and their faults.

The listing is the one the command prints with no verb.

1
machine net run list [-h]

Examples:

1
machine net run1 list  each link and its faults

See also:

machine net hold

Hold the packets a container sends.

Every packet waits in the link until a release, and then they all leave together. The link holds 1000 full-size packets and drops the ones past them.

Without a container the command holds every link at the instruction the machine is stopped at. All networking between the containers stops there, and a replay holds at the same point. A release without a container lets every packet go.

Use --to db to put the fault on what the container sends to db alone. Use --proto, --port and --syn to take some packets alone, such as port 5432. A packet a match takes gets only the match’s faults, not the whole link’s.

Use --for 5s with a container to release its packets after 5 seconds of machine time. The hold of every link ends with a release.

1
2
3
machine net run hold [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] [--syn]
[--for duration]
[container]
Argument Does
[container] Container, by its id, its name or its service’s (default: every one)
Option Does
--to container The packets to this container alone (default: to every peer)
--proto {tcp,udp,icmp,arp} The packets of this protocol alone
--port n The packets with this source or destination port alone
--syn The TCP SYNs that open a connection alone
--for duration Release the packets after this machine time, such as 5s

Examples:

1
2
3
4
machine net run1 hold db                       until a release
machine net run1 hold db --for 5s released after 5s
machine net run1 hold every link, here
machine net run1 hold db --to api --port 5432 one flow alone

See also:

machine net release

Release the packets a container’s links hold.

The held packets leave together, in the order they were sent. The link keeps its other faults. Without a container the command releases every link.

1
2
3
machine net run release [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n]
[--syn]
[container]
Argument Does
[container] Container, by its id, its name or its service’s (default: every one)
Option Does
--to container The packets to this container alone (default: to every peer)
--proto {tcp,udp,icmp,arp} The packets of this protocol alone
--port n The packets with this source or destination port alone
--syn The TCP SYNs that open a connection alone

Examples:

1
2
machine net run1 release db  db's held packets leave
machine net run1 release every link released

See also:

machine net reset

Reset the TCP connections of a container.

The program on the container’s side sees its socket fail with ECONNABORTED, and the peer receives a TCP reset. A listening socket stays open. Use --to db and --port 5432 to reset some connections alone.

1
machine net run reset [-h] [--to container] [--port n] container
Argument Does
container Container, by its id, its name or its service’s
Option Does
--to container The connections to this container alone (default: every one)
--port n The connections with this local or remote port alone

Examples:

1
2
3
machine net run1 reset api             every connection of api
machine net run1 reset api --to db its connections to db
machine net run1 reset db --port 5432 those of port 5432

See also:

machine net arp

Show or change a container’s neighbour table.

The table maps each peer’s address to the MAC that ARP resolved. The show verb lists it, with the peer each address belongs to and the state of each entry.

The other verbs change the table. They set a wrong MAC, mark an entry stale or delete entries. The reachable verb sets how long an entry stays reachable. A container that starts again begins with an empty table.

1
machine net run arp [-h] command ...

Examples:

1
2
3
machine net run1 arp show api                      api's neighbour table
machine net run1 arp set api db 02:42:ac:12:00:99 a wrong MAC
machine net run1 arp flush api resolved again by ARP

See also:

machine net arp show

List a container’s neighbour table.

Each entry shows its interface, the address, the peer it belongs to, the MAC and its state.

1
machine net run arp show [-h] container
Argument Does
container Container, by its id, its name or its service’s

Examples:

1
machine net run1 arp show api  api's neighbour table

See also:

machine net arp set

Map a peer’s address to a MAC.

The entry is permanent, so ARP never corrects it. The container sends the peer’s frames to that MAC until the entry is deleted.

1
machine net run arp set [-h] container peer mac
Argument Does
container Container, by its id, its name or its service’s
peer The peer, by its id, its name or its service’s
mac The MAC, such as 02:42:ac:12:00:99

Examples:

1
machine net run1 arp set api db 02:42:ac:12:00:99  a wrong MAC

See also:

machine net arp stale

Mark a peer’s entry stale.

The kernel uses the entry, then confirms it by ARP at the next packet.

1
machine net run arp stale [-h] container peer
Argument Does
container Container, by its id, its name or its service’s
peer The peer, by its id, its name or its service’s

Examples:

1
machine net run1 arp stale api db  confirmed by ARP next

See also:

machine net arp delete

Delete a peer’s entry.

The next packet to the peer resolves its address by ARP again.

1
machine net run arp delete [-h] container peer
Argument Does
container Container, by its id, its name or its service’s
peer The peer, by its id, its name or its service’s

Examples:

1
machine net run1 arp delete api db  db resolved again

See also:

machine net arp flush

Delete every entry of the table.

Every peer is resolved by ARP again at its next packet.

1
machine net run arp flush [-h] container
Argument Does
container Container, by its id, its name or its service’s

Examples:

1
machine net run1 arp flush api  every entry deleted

See also:

machine net arp reachable

Set how long an entry stays reachable.

After ARP confirms an entry, the kernel keeps it reachable for a random time around this one. The entry is then stale.

1
machine net run arp reachable [-h] container duration
Argument Does
container Container, by its id, its name or its service’s
duration The base reachable time, such as 1s

Examples:

1
machine net run1 arp reachable api 1s  stale after about 1s

See also:

machine net heal

Take the faults off a container’s links.

Without a container the command heals every link. Use --to db to heal the faults on packets to db alone. The command releases the packets a hold keeps. A packet the link delays already still leaves at its time.

1
2
machine net run heal [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] [--syn]
[container]
Argument Does
[container] Container, by its id, its name or its service’s (default: every one)
Option Does
--to container The packets to this container alone (default: to every peer)
--proto {tcp,udp,icmp,arp} The packets of this protocol alone
--port n The packets with this source or destination port alone
--syn The TCP SYNs that open a connection alone

Examples:

1
2
3
4
machine net run1 heal db              db's links healed
machine net run1 heal db --to api db's link to api healed
machine net run1 heal db --port 5432 the faults on port 5432
machine net run1 heal every link healed

See also:

machine core

Get a core dump of a process.

The command tells a process to generate a core dump and copies it to your computer. A debugger reads the core with the program of the process. Use --container for a process of a container, by the pid it has there.

Taking a core ends the process. The process dies of the signal. Use fork first to keep the run as it is, and take the core on the branch.

A process that handles the signal dumps no core. Use --signal to name another. The first process of a container dumps none.

The machine runs until the core is written and copied. Other programs of the machine run with it.

1
machine core [-h] [--container name] [--signal signal] [--within duration] run pid file
Argument Does
run Run name
pid Pid of the process in the machine or in the container
file File on this computer to write the core to
Option Does
--container name Container the process is in, by its id, its name or its Compose service (default: the machine itself)
--signal signal Signal the process dies of, by name or number (default: ABRT)
--within duration Machine time allowed each time the command runs the machine, such as 10s (default: 30m)

Examples:

1
2
3
machine core run1 412 ./app.core              a core of pid 412
machine core run1 7 ./db.core --container db of a container
machine core run1 412 ./c --signal SEGV by another signal

Prints:

1
2
text  Core of <pid> of <run> (<id>): <n> bytes in <file>
json size pid container signal file process run_id name entry_id

Refuses, and the way past:

1
2
a machine that is replaying      machine fork run1 --name try2
a container that is not running machine containers run1

See also:

machine profile

Profile what the guest runs.

The machine samples each CPU at every timer interrupt, 250 times a second. Nothing runs in the guest for it. The command prints the functions that ran. Each has the share of samples it ran in and the share it was on the stack of.

Use --from to profile a stretch the run already recorded. The stretch is replayed on a branch, which is deleted after. The run is not changed and need not be running. A stretch profiled twice gives the same samples.

Use --pid or --container to show one process or one container. Use --folded to print every stack with its count, one a line. That text is what flame graph tools read. Use --flame to draw a flame graph in the terminal.

The names come from the program files on the machine’s disk when the stretch ends. No checkpoint is taken. A function in a file that is not on the disk is shown by the file’s name.

Time in the kernel is one frame for each system call, named like sys_read_[k].

Without --from the machine runs for the time sampled. A machine that is idle has no samples.

Use --in-guest to sample with a program in the guest, which the exec agent of the disk runs. It names the functions of the kernel, and programs not yet written to the disk. The run records that program like any other.

1
2
3
4
machine profile [-h] [--for duration | --from point [point ...]] [--to point [point ...]]
[--pid pid] [--container name] [--in-guest] [--folded | --flame] [--top n]
[--within duration]
run
Argument Does
run Run name
Option Does
--for duration Machine time to run and sample for, such as 500ms (default: 1s)
--from point [point ...] Point a recorded stretch begins at: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, parent [n] or root. The stretch is replayed on a branch
--to point [point ...] Point the recorded stretch ends at, as --from takes one (default: the run’s last entry)
--pid pid Process to show, by its pid in the machine or in the container (default: every process)
--container name Container to show, by its id, its name or its Compose service (default: the machine itself)
--in-guest Sample with the profiler of the exec agent, a process of the guest: kernel functions by name, at the cost of changing the run
--folded Print the stacks, one a line with its count, for another tool to draw
--flame Draw a flame graph in the terminal
--top n Functions the table shows (default: 20)
--within duration Machine time allowed to find the container, such as 10s (default: 30m)

Examples:

1
2
3
4
5
6
machine profile run1                          where a second of run1 went
machine profile run1 --for 5s --container db one container
machine profile run1 --from ^slow --to ^done a recorded stretch
machine profile run1 --flame a flame graph in the terminal
machine profile run1 --in-guest kernel functions by name
machine profile run1 --folded the stacks, for another tool

Prints:

1
2
3
4
5
text      SELF TOTAL FUNCTION
SELF the share of samples the function was running in
TOTAL the share it was on the stack of
--folded comm;outer;...;inner <count>
json samples hz stacks left run_id name

Refuses, and the way past:

1
2
a container the run does not have  machine containers run1
--in-guest with --from machine profile run1 --from ^slow

See also:

machine irq

Inject an interrupt into the vCPUs. See Interrupts.

The command takes a vector, or --nmi alone for an NMI. Use irqs to list the vector names.

The vector is a number, or a name from irq_vectors.h of the guest kernel. The name preempt is 0xf5, the vector Machine preempts a task with. The name reschedule is 0xfd, the reschedule vector of the guest kernel.

The vector is delivered as given. It reaches the handler that the kernel of the machine installed for it. A vector with no handler is a spurious interrupt. Vectors below 32 are CPU exceptions and are refused.

The interrupt is queued when the command returns. It is delivered and recorded when the machine next runs. A replay delivers it at the same instruction. Step first to choose that instruction.

1
machine irq [-h] [--vcpu n] [--nmi] run [vector]
Argument Does
run Run name
[vector] Vector: a name such as preempt, or a number 32-255
Option Does
--vcpu n Which vCPU takes it, or all (default: the last entry’s)
--nmi Deliver as an NMI rather than a fixed interrupt

Examples:

1
2
3
machine irq run1 0xec     inject vector 0xec
machine irq run1 preempt inject the vector Machine preempts with
machine irq run1 --nmi inject an NMI

See also:

machine irqs

List the interrupts of a machine. See Holding interrupts.

The command lists every interrupt the machine owes or holds now, then the vectors irq injects.

An owed interrupt was raised by a device or a timer and is injected at the next dispatch. A held interrupt waits for its due, which run --irq-delay drew. The state hash irq_pending is of the held ones, and dump --show irq keeps the first 32 of them.

The command changes nothing. Use ints for the interrupts the guest took, by vector.

1
machine irqs [-h] run
Argument Does
run Run name

Examples:

1
2
machine irqs run1          the pending interrupts and the vectors
machine -o json irqs run1 the same as JSON

See also:

machine inject

Inject faults into the guest’s system calls and kernel.

A fault makes a system call fail with an error, or wait. A fault at a kernel site fails a disk request or drops its completion. It also fails an allocation of the kernel, or a copy to or from a program’s memory. The guest kernel decides at each call, without stopping the machine.

A replay acts on the same calls the same way. The faults are part of the machine. A checkpoint keeps them, and a fork has its parent’s. The run records each change, and a replay makes it at the same point.

Use --in to say whose calls a fault acts on. It is required. A program is the absolute path of the file it runs, the EXE that tasks shows. It takes the tasks running that file, and the processes they fork without exec.

A comm is the name the kernel gives a task at exec, and comm=name takes the tasks of that comm. A script runs as its interpreter, so its comm names it. tid=n takes one task, tgid=n the tasks of one tgid, and all every task in the machine.

A fault lets a number of matching calls through first. It then fires on each call by a chance, up to a number of calls or without end. Each fault counts the calls it matches on its own. The first fault that fires on a call acts on it.

A program’s fault can be added before the program starts. Its calls match from its exec on, and the commands that start it match none. A fault on a tgid applies to whatever that tgid runs, exec’s setup of it first. Use tasks exist to check that what --in names is running.

Without a verb the command lists the faults.

1
machine inject [-h] run command ...
Argument Does
run Run name

Examples:

1
2
3
4
5
6
7
machine inject run1                                       the faults
machine inject run1 add syscall connect EIO --in /app every connect
machine inject run1 add disk --prob 0.01 --in all a request in 100
machine inject run1 add slab --times 1 --in tgid=412 one allocation
machine inject run1 add disk-corrupt --file /db --in all a file
machine inject run1 delete 2 the second fault
machine inject run1 clear every fault

See also:

machine inject add

Inject a fault into a subsystem of the guest.

The subsystem is syscall, or one of the kernel’s own fault sites: disk, disk-timeout, slab, pages, usercopy and futex-access. The subsystems disk-corrupt, disk-torn and disk-lost are faults of the disk itself. The function subsystem makes a function of a program return a value.

Every subsystem takes the same options, with the same meaning. The disk subsystems also take --file, --op and --sectors.

A fault fires on every matching call by default. Use --prob to fire on a call by chance. A --prob of 0 fires on none and counts the matches.

The fault draws from its seed which calls it fires on. A replay draws the same. Use --seed to draw as another run did.

Use --on and --off to make a fault cycle on the machine’s clock. With --on 2s --off 5s, the fault is active for 2 seconds from when it is added. It then rests for 5 seconds, and the cycle repeats. A resting fault matches no call and counts none.

1
machine inject run add [-h] subsystem ...

Examples:

1
2
3
4
5
6
machine inject run1 add syscall openat ENOENT --in /app      fail
machine inject run1 add syscall read --delay 2ms --in /app wait
machine inject run1 add disk --times 3 --in /app three requests
machine inject run1 add pages --after 99 --times 1 --in all the 100th
machine inject run1 add disk-lost --file /db/wal --in all lose
machine inject run1 add disk --on 2s --off 5s --in all 2s in 7

See also:

machine inject add syscall

Fail calls of system calls with errors, or hold them.

A failed call returns an error without running. Each system call gets its own fault. A failing call returns one error of the set, drawn at random.

The set holds names or numbers. Use valid for the errors the system call’s man page documents, and all for every error Linux names. A -ERRNO in the set takes that error out.

Use --delay to hold each call the fault fires on for a time, drawn from a span. A fault with a delay and no errors runs the call after the hold. A fault with both holds the call, then fails it. The hold is on the guest’s own clock, so a replay holds the same calls for the same time.

Use --pick to take a number of the set at random when the fault is added. Each system call picks its own. A run then fails with its own few errors, as swarm testing does.

A dynamic program’s loader opens and reads its libraries after the exec. A fault on openat or read added before the program starts fails the loader. Add such a fault once the program is up.

1
2
3
4
machine inject run add syscall [-h] [--delay span] [--pick n] --in who [--prob p]
[--after n] [--times n] [--seed n] [--on duration]
[--off duration]
syscall[,syscall...] [errors]
Argument Does
syscall[,syscall...] System calls, by name or number
[errors] Errors: ENOENT, a number, valid or all, comma-separated; -ERRNO takes one out. Without them, --delay is required and the call runs
Option Does
--delay span Hold each call the fault fires on for a time drawn from a span such as 0:20ms, at most 1s
--pick n Pick n of the errors at random when the fault is added (default: all of them)
--in who Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task
--prob p Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1)
--after n Let n matching calls through first (default: 0)
--times n Fire on at most n calls, then let the rest through (default: no end)
--seed n Seed the draws of the fault (default: a random seed)
--on duration Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active)
--off duration Machine time the fault rests in each cycle, such as 5s; needs --on

Examples:

1
2
3
4
5
6
7
8
machine inject run1 add syscall connect EIO --in /a          every connect
machine inject run1 add syscall open valid --in /a documented
machine inject run1 add syscall read all --pick 1 --in /a one of all
machine inject run1 add syscall read EIO --prob 0.1 --in /a one in ten
machine inject run1 add syscall read EIO --after 9 --in /a past the 9th
machine inject run1 add syscall open EIO --in comm=cat a cat applet
machine inject run1 add syscall write EIO --in tgid=412 one tgid
machine inject run1 add syscall read --delay 20ms --in /a slow reads

See also:

machine inject add function

Make a function of a program return a value without running.

A call the fault fires on returns the value of --return to its caller, and the function does not run. A malloc that returns 0 is an allocation that failed. The C library does not set errno, so a caller that reads errno reads what was there.

The function is found by its name in the files a running task maps, a library or the program itself. The task is one --in names, or the task of --at when what --in names does not run yet. The fault acts on every task that maps the same file and that --in names.

The guest kernel puts a breakpoint at the function in the tasks --in names. Each call there traps into the kernel, which decides as for every fault.

1
2
3
4
machine inject run add function [-h] [--return value] [--at tid] --in who [--prob p]
[--after n] [--times n] [--seed n] [--on duration]
[--off duration]
function
Argument Does
function Function, by its name
Option Does
--return value Value it returns, such as 0 or -1; 0 by default
--at tid Task whose files the function is found in, when --in names none running
--in who Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task
--prob p Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1)
--after n Let n matching calls through first (default: 0)
--times n Fire on at most n calls, then let the rest through (default: no end)
--seed n Seed the draws of the fault (default: a random seed)
--on duration Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active)
--off duration Machine time the fault rests in each cycle, such as 5s; needs --on

Examples:

1
2
3
machine inject run1 add function malloc --in /app        malloc fails
machine inject run1 add function f --prob 0.1 --in /app 1 in 10
machine inject run1 add function f --in comm=c --at 4 found in 4

See also:

machine inject add disk

Fail block requests of the guest’s disks with EIO.

A read or write request fails with EIO in the task that submits it. A program submits its reads, its O_DIRECT writes and the writes its fsync forces. The kernel’s flusher threads submit buffered writes later, so --in all or a kworker comm matches those. A request the kernel splits counts once for each part.

A buffered read is a readahead request, then one more read of the page when the readahead fails. With --times 1 the kernel’s second read succeeds and the program reads the file. Use --times 2 or no limit to return EIO to the program. An O_DIRECT read fails on its first request.

A disk fault matches a request by its operation, its sectors and its file. Use --file to name the file whose data the request carries. Buffered writes reach the disk from the kernel’s flusher, so use --in all with --file.

A file matches whether it is opened or mapped before or after the fault is added.

The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.

1
2
3
machine inject run add disk [-h] --in who [--prob p] [--after n] [--times n] [--seed n]
[--on duration] [--off duration] [--file pattern]
[--op {read,write}] [--sectors lo:hi]
Option Does
--in who Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task
--prob p Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1)
--after n Let n matching calls through first (default: 0)
--times n Fire on at most n calls, then let the rest through (default: no end)
--seed n Seed the draws of the fault (default: a random seed)
--on duration Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active)
--off duration Machine time the fault rests in each cycle, such as 5s; needs --on
--file pattern Requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file)
--op {read,write} Requests of one operation (default: both)
--sectors lo:hi Requests that touch 512-byte sectors lo to hi of the disk (default: the whole disk)

Examples:

1
2
3
4
machine inject run1 add disk --in all                    every block request
machine inject run1 add disk --prob 0.1 --in /app a request in ten
machine inject run1 add disk --file /db --in all one file
machine inject run1 add disk --on 1s --off 10s --in all 1s outages

See also:

machine inject add disk-timeout

Drop the completions of block requests.

The request is decided in the task that submits it, and its completion is dropped when the disk answers. The guest’s disk driver has no timeout handler, so the request never completes and the task waiting on it hangs. The tasks that submit a request are as for disk.

A disk fault matches a request by its operation, its sectors and its file. Use --file to name the file whose data the request carries. Buffered writes reach the disk from the kernel’s flusher, so use --in all with --file.

A file matches whether it is opened or mapped before or after the fault is added.

The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.

1
2
3
machine inject run add disk-timeout [-h] --in who [--prob p] [--after n] [--times n]
[--seed n] [--on duration] [--off duration]
[--file pattern] [--op {read,write}] [--sectors lo:hi]
Option Does
--in who Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task
--prob p Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1)
--after n Let n matching calls through first (default: 0)
--times n Fire on at most n calls, then let the rest through (default: no end)
--seed n Seed the draws of the fault (default: a random seed)
--on duration Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active)
--off duration Machine time the fault rests in each cycle, such as 5s; needs --on
--file pattern Requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file)
--op {read,write} Requests of one operation (default: both)
--sectors lo:hi Requests that touch 512-byte sectors lo to hi of the disk (default: the whole disk)

Examples:

1
machine inject run1 add disk-timeout --times 1 --in comm=dd  dd hangs

See also:

machine inject add slab

Fail slab allocations of the guest kernel.

The allocation returns NULL, and the call that made it fails with ENOMEM or as the kernel handles it. An allocation the kernel promised, with __GFP_NOFAIL, never fails.

The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.

1
2
machine inject run add slab [-h] --in who [--prob p] [--after n] [--times n] [--seed n]
[--on duration] [--off duration]
Option Does
--in who Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task
--prob p Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1)
--after n Let n matching calls through first (default: 0)
--times n Fire on at most n calls, then let the rest through (default: no end)
--seed n Seed the draws of the fault (default: a random seed)
--on duration Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active)
--off duration Machine time the fault rests in each cycle, such as 5s; needs --on

Examples:

1
machine inject run1 add slab --prob 0.001 --in comm=cat  rarely

See also:

machine inject add pages

Fail page allocations of the guest kernel.

The allocation returns no pages, and the call that made it fails with ENOMEM or as the kernel handles it. An allocation the kernel promised, with __GFP_NOFAIL, never fails.

The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.

1
2
machine inject run add pages [-h] --in who [--prob p] [--after n] [--times n] [--seed n]
[--on duration] [--off duration]
Option Does
--in who Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task
--prob p Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1)
--after n Let n matching calls through first (default: 0)
--times n Fire on at most n calls, then let the rest through (default: no end)
--seed n Seed the draws of the fault (default: a random seed)
--on duration Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active)
--off duration Machine time the fault rests in each cycle, such as 5s; needs --on

Examples:

1
machine inject run1 add pages --after 99 --times 1 --in all  the 100th

See also:

machine inject add usercopy

Fail copies between the kernel and a program’s memory.

The copy fails, and the system call that made it returns EFAULT or a short count. A system call that copies a buffer to or from the program copies through it. A call that reads or writes one value with get_user or put_user does not.

The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.

1
2
machine inject run add usercopy [-h] --in who [--prob p] [--after n] [--times n] [--seed n]
[--on duration] [--off duration]
Option Does
--in who Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task
--prob p Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1)
--after n Let n matching calls through first (default: 0)
--times n Fire on at most n calls, then let the rest through (default: no end)
--seed n Seed the draws of the fault (default: a random seed)
--on duration Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active)
--off duration Machine time the fault rests in each cycle, such as 5s; needs --on

Examples:

1
machine inject run1 add usercopy --times 1 --in tid=41  one copy

See also:

machine inject add futex-access

Fail a futex’s access to a program’s memory.

A futex operation’s access to the program’s memory fails. The futex call returns EFAULT, or EDEADLK where a priority-inheritance futex checks its owner.

The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.

1
2
machine inject run add futex-access [-h] --in who [--prob p] [--after n] [--times n]
[--seed n] [--on duration] [--off duration]
Option Does
--in who Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task
--prob p Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1)
--after n Let n matching calls through first (default: 0)
--times n Fire on at most n calls, then let the rest through (default: no end)
--seed n Seed the draws of the fault (default: a random seed)
--on duration Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active)
--off duration Machine time the fault rests in each cycle, such as 5s; needs --on

Examples:

1
machine inject run1 add futex-access --in tgid=412  every futex call

See also:

machine inject add disk-corrupt

Change bytes of reads and writes of the guest’s disks.

The disk stores a write with bytes changed, or returns a read with bytes changed. The request succeeds. The changed bytes are drawn from the fault’s seed. A program’s own reads of a written file come from the page cache until the guest drops it.

A disk fault matches a request by its operation, its sectors and its file. Use --file to name the file whose data the request carries. Buffered writes reach the disk from the kernel’s flusher, so use --in all with --file.

A file matches whether it is opened or mapped before or after the fault is added.

The guest kernel decides on each request in the task that submits it. The virtual disk then changes what it stores or returns. A replay makes the same decisions and changes the same bytes.

1
2
3
4
machine inject run add disk-corrupt [-h] --in who [--prob p] [--after n] [--times n]
[--seed n] [--on duration] [--off duration]
[--file pattern] [--op {read,write}] [--sectors lo:hi]
[--bytes n]
Option Does
--in who Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task
--prob p Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1)
--after n Let n matching calls through first (default: 0)
--times n Fire on at most n calls, then let the rest through (default: no end)
--seed n Seed the draws of the fault (default: a random seed)
--on duration Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active)
--off duration Machine time the fault rests in each cycle, such as 5s; needs --on
--file pattern Requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file)
--op {read,write} Requests of one operation (default: both)
--sectors lo:hi Requests that touch 512-byte sectors lo to hi of the disk (default: the whole disk)
--bytes n Bytes of the request to change, 1 to 4096 (default: 1)

Examples:

1
2
3
4
machine inject run1 add disk-corrupt --file '/db/*/c.bin' --in all  parts
machine inject run1 add disk-corrupt --op read --in /app reads
machine inject run1 add disk-corrupt --bytes 8 --times 1 --in all once
machine inject run1 add disk-corrupt --sectors 0:7 --in all sectors

See also:

machine inject add disk-torn

Store part of a write of the guest’s disks.

The disk stores the first sectors of a write and reports success. It stores one sector at least, and fewer than all. The rest keeps what the disk held. A sector is written whole, so a write of one sector does not tear.

A disk fault matches a request by its operation, its sectors and its file. Use --file to name the file whose data the request carries. Buffered writes reach the disk from the kernel’s flusher, so use --in all with --file.

A file matches whether it is opened or mapped before or after the fault is added.

The guest kernel decides on each request in the task that submits it. The virtual disk then changes what it stores or returns. A replay makes the same decisions and changes the same bytes.

1
2
3
machine inject run add disk-torn [-h] --in who [--prob p] [--after n] [--times n]
[--seed n] [--on duration] [--off duration]
[--file pattern] [--sectors lo:hi]
Option Does
--in who Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task
--prob p Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1)
--after n Let n matching calls through first (default: 0)
--times n Fire on at most n calls, then let the rest through (default: no end)
--seed n Seed the draws of the fault (default: a random seed)
--on duration Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active)
--off duration Machine time the fault rests in each cycle, such as 5s; needs --on
--file pattern Requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file)
--sectors lo:hi Requests that touch 512-byte sectors lo to hi of the disk (default: the whole disk)

Examples:

1
2
machine inject run1 add disk-torn --file /db/wal --in all  torn writes
machine inject run1 add disk-torn --times 1 --in all one write

See also:

machine inject add disk-lost

Lose writes of the guest’s disks.

The disk reports a write done and stores none of it. The disk keeps what it held, as a disk that lost its cache does.

A disk fault matches a request by its operation, its sectors and its file. Use --file to name the file whose data the request carries. Buffered writes reach the disk from the kernel’s flusher, so use --in all with --file.

A file matches whether it is opened or mapped before or after the fault is added.

The guest kernel decides on each request in the task that submits it. The virtual disk then changes what it stores or returns. A replay makes the same decisions and changes the same bytes.

1
2
3
machine inject run add disk-lost [-h] --in who [--prob p] [--after n] [--times n]
[--seed n] [--on duration] [--off duration]
[--file pattern] [--sectors lo:hi]
Option Does
--in who Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task
--prob p Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1)
--after n Let n matching calls through first (default: 0)
--times n Fire on at most n calls, then let the rest through (default: no end)
--seed n Seed the draws of the fault (default: a random seed)
--on duration Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active)
--off duration Machine time the fault rests in each cycle, such as 5s; needs --on
--file pattern Requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file)
--sectors lo:hi Requests that touch 512-byte sectors lo to hi of the disk (default: the whole disk)

Examples:

1
2
machine inject run1 add disk-lost --file /db/wal --in all        lost writes
machine inject run1 add disk-lost --after 10 --times 1 --in all the 11th

See also:

machine inject delete

Delete a fault.

The fault’s calls run again from here.

1
machine inject run delete [-h] n
Argument Does
n Fault number, as the listing shows it

Examples:

1
machine inject run1 delete 2  the second fault

See also:

machine inject list

List the faults.

The listing is the one the command prints with no verb.

1
machine inject run list [-h]

Examples:

1
machine inject run1 list  the faults, by number

See also:

machine inject clear

Delete every fault.

1
machine inject run clear [-h]

Examples:

1
machine inject run1 clear  every fault

See also:

machine axis

Set the defaults of the draw axes a run uses.

A default is a plan clause the run keeps. Every run, exec and plan draws from the defaults until they are removed. The axes are the ones a plan clause takes, such as mode, vtime-rate, task-sched and irq-delay.

A run option or a plan clause replaces a default for the axes it names. A default with a when condition replaces one with no condition where the condition holds. Of two conditions that hold, the default set last wins.

The defaults are part of the machine. A checkpoint keeps them, and a fork has its parent’s. The run records each change, and a replay makes it at the same point.

Without a verb the command lists the defaults.

1
machine axis [-h] run command ...
Argument Does
run Run name

Examples:

1
2
3
4
5
machine axis run1                             the defaults
machine axis run1 set task-sched hold every dispatch
machine axis run1 set when /app:on mode step in /app
machine axis run1 unset task-sched no condition
machine axis run1 clear every default

See also:

machine axis set

Set a default of the draw axes.

The default is a clause of a plan, written as a plan file writes it. It names one or more axes and their values. A clause that starts with when applies where its predicate holds.

The default replaces the values its axes had under the same condition. A clause that names several axes draws them together.

1
machine axis run set [-h] ...
Argument Does
... A plan clause, such as when /app:on task-sched hold

Examples:

1
2
3
machine axis run1 set vtime-rate 10:40                      a drawn rate
machine axis run1 set irq-delay blk=1ms disk interrupts
machine axis run1 set when /app:on mode step vtime-rate 40 together

See also:

machine axis unset

Remove defaults of the draw axes.

The axes are named as a clause names them, without values. A clause that starts with when removes the defaults under that predicate. Without when, the command removes the defaults that have no condition.

Use --all to remove the axes’ defaults under every condition.

1
machine axis run unset [-h] [--all] ...
Argument Does
... Axis names after an optional when predicate, such as when /app:on mode
Option Does
--all Remove the axes’ defaults under every condition

Examples:

1
2
3
machine axis run1 unset mode               no condition
machine axis run1 unset when /app:on mode in /app
machine axis run1 unset --all mode every condition

See also:

machine axis clear

Remove every default.

1
machine axis run clear [-h]

Examples:

1
machine axis run1 clear  every default

See also:

machine vtime

Show and change machine time. See Machine time.

Machine time is the guest’s clock. It is derived from what the machine executes, not from the host clock.

Use rate to change how fast machine time passes. Use cost to change what each exit costs on it. Use add to push the clock forward without running the machine. The clock only goes forward.

Use rewind to put the run back at an earlier point.

1
machine vtime [-h] run command ...
Argument Does
run Run name

Examples:

1
2
3
machine vtime run1           machine time and its rate
machine vtime run1 rate 5 set the rate to 5
machine vtime run1 add 1000 advance the clock 1000 ns

See also:

machine vtime rate

Set how fast machine time passes. See Machine time.

The rate is the nanoseconds of machine time for each conditional branch the guest retires. A run starts at 10 unless create was given --vtime-rate.

Give the rate as a number or as num/den. Add N=rate to give vCPU N its own rate, so its work costs more or less time than the others’. A rate for every vCPU clears each vCPU’s own.

The run records each change, and a replay makes it at the same point. A run’s --vtime-rate draws one rate for every vCPU for that run alone.

1
machine vtime run rate [-h] rate
Argument Does
rate Rate, num/den, and N=rate for vCPU N’s own, such as 10,1=40

Examples:

1
2
3
4
machine vtime run1 rate 20       20 ns for each conditional branch
machine vtime run1 rate 1/2 1 ns for every two
machine vtime run1 rate 10,1=40 vCPU 1 at 40, the rest at 10
machine vtime run1 rate 1=20 vCPU 1's own rate alone

See also:

machine vtime add

Push the clock forward without running the machine. See Machine time.

Give a time such as 1s or 250ms. A bare number is nanoseconds.

1
machine vtime run add [-h] time
Argument Does
time Time to add, such as 1s or 250ms; a bare number is nanoseconds

Examples:

1
machine vtime run1 add 10us  advance the clock 10 us

See also:

machine vtime cost

Set what each exit costs on the machine’s clock.

Each exit of the guest advances machine time by a cost. The exits priced are an I/O, MMIO or hypercall exit, an RDTSC and each PAUSE of a spin loop. The rate prices the guest’s computing, and the costs price its waiting. Their ratio decides whether a spin or an I/O completion wins a race.

Give io=, rdtsc= or pause= with a time from 1ns to 1s. A bare number is nanoseconds. A cost not given stays as it is. The RDTSC cost follows the I/O cost until it is given, and rdtsc=io makes it follow again.

The run records the change, and a replay makes it at the same point. A run’s --vtime-io-cost, --vtime-rdtsc-cost or --vtime-pause-cost draws its own costs for that run alone.

1
machine vtime run cost [-h] exit=time [exit=time ...]
Argument Does
exit=time [exit=time ...] An exit and its cost, such as io=2us, rdtsc=1us or pause=50; rdtsc=io follows the I/O cost

Examples:

1
2
3
4
machine vtime run1 cost io=20us    each I/O exit costs 20 us
machine vtime run1 cost pause=50 each PAUSE costs 50 ns
machine vtime run1 cost rdtsc=1us an RDTSC apart from I/O
machine vtime run1 cost rdtsc=io an RDTSC as an I/O again

See also:

machine clock

Show and change the guest’s date and each container’s.

The command steps the date and sets it. It makes a container’s date drift, or slews it to a new value. It steps the date to a drawn offset each period, as NTP does. It inserts or deletes a leap second.

The date is what date, time and gettimeofday return. A change moves it for every process in the guest. Elapsed time, the TSC and timers measured in elapsed time do not move. Use vtime to change how fast time runs.

The guest kernel applies a change when the run next executes. The command answers with the date as it stands and the change waiting. The reading after that run names a change the guest kernel refused.

A change is part of the machine. The run records it, and a replay makes it at the same point. A checkpoint keeps the date and a pending undo.

A container’s date is an offset from the guest’s date. Every process of the container reads it, those docker exec starts included. A timer armed after a change agrees with it. A timer armed before keeps its deadline.

Without a verb the command shows the date and each container’s. The command finds a container with the machine stopped and never runs it. It tries a name first, then a service, then an id. An id works from the container’s start, while a name is known shortly after.

A jump or a set takes a running container. A show or a reset also takes a stopped one with an offset. A service stands for its one container of those.

1
machine clock [-h] [--container name] run command ...
Argument Does
run Run name
Option Does
--container name A container by its id, its name or its Compose service

Examples:

1
2
3
4
5
6
7
8
9
10
machine clock run1                                   the guest's date
machine clock run1 jump +59m50s 10s short of an hour
machine clock run1 jump -2s a step back
machine clock run1 set 2038-01-19T03:13:30Z a date
machine clock run1 reset machine time
machine clock run1 --container db db's date
machine clock run1 jump +2s --for 1m --container db db alone
machine clock run1 drift +200ppm --container db db drifts
machine clock run1 jitter -200ms:200ms --every 10m NTP's steps
machine clock run1 leap insert 23:59:60 at midnight

See also:

machine clock jump

Step the guest’s date forward or back.

The step is the one an NTP step or date -s makes. A timer armed for an absolute date sees the step. A timerfd with TFD_TIMER_CANCEL_ON_SET is woken. Each step adds to the ones before it.

Use --for 30s to undo the change after 30 seconds of machine time. The undo takes back the command’s own change, and what a drift gave meanwhile stays. A later change cancels an undo still pending.

Use --slew 30s with --container to move a container’s date over 30 seconds of machine time. No read in it sees a jump. With --for, the date slews back over the same time.

Use --container db to change one container’s date alone. The container keeps its offset from the guest’s date across a restart.

1
machine clock run jump [-h] [--for duration] [--slew duration] [--container name] offset
Argument Does
offset The step, such as +5s, -2h or 59m50s; no sign is forward
Option Does
--for duration Undo the change after this much machine time, such as 30s
--slew duration Move a container’s date over this machine time, such as 30s
--container name A container by its id, its name or its Compose service

Examples:

1
2
3
4
5
machine clock run1 jump +5s                            forward
machine clock run1 jump -500ms --for 30s back, then undone
machine clock run1 jump +2s --container db db alone
machine clock run1 jump +1h --container 59636762b6f8 by id
machine clock run1 jump +4s --slew 30s --container db slewed

See also:

machine clock set

Set the guest’s date.

The date is written with its zone, such as 2027-03-28T00:59:50Z. The change is a step, as date -s makes one.

Use --for 30s to undo the change after 30 seconds of machine time. The undo takes back the command’s own change, and what a drift gave meanwhile stays. A later change cancels an undo still pending.

Use --slew 30s with --container to move a container’s date over 30 seconds of machine time. No read in it sees a jump. With --for, the date slews back over the same time.

Use --container db to change one container’s date alone. The container keeps its offset from the guest’s date across a restart.

1
machine clock run set [-h] [--for duration] [--slew duration] [--container name] date
Argument Does
date The date with its zone, such as 2027-03-28T00:59:50Z
Option Does
--for duration Undo the change after this much machine time, such as 30s
--slew duration Move a container’s date over this machine time, such as 30s
--container name A container by its id, its name or its Compose service

Examples:

1
2
3
machine clock run1 set 2026-12-31T23:59:50Z                 the year's end
machine clock run1 set 2027-03-28T00:59:50Z --for 1m for a minute
machine clock run1 set 2038-01-19T03:13:30Z --container db db alone

See also:

machine clock drift

Make a container’s date gain or lose time.

The container’s date gains the rate of each second of machine time, such as +200ppm, 0.72 seconds an hour. No read sees a jump. Its processes read it in the vDSO and in system calls. A timer armed after agrees with it.

Use drift 0 to stop it and keep the offset it reached. A jump, a set or a jitter moves the date under the drift.

The guest’s date has no drift. Use vtime rate to make the whole machine’s time run fast or slow.

Use --for 30s to undo the change after 30 seconds of machine time. The undo takes back the command’s own change, and what a drift gave meanwhile stays. A later change cancels an undo still pending.

1
machine clock run drift [-h] [--for duration] [--container name] rate
Argument Does
rate The rate, such as +200ppm, -1.5ppm or 300ppb; 0 stops it
Option Does
--for duration Undo the change after this much machine time, such as 30s
--container name A container by its id, its name or its Compose service

Examples:

1
2
3
machine clock run1 drift +200ppm --container db          0.72s an hour
machine clock run1 drift -50ppm --for 1h --container db for an hour
machine clock run1 drift 0 --container db stopped

See also:

machine clock jitter

Step the guest’s date to a drawn offset each period.

Each period of machine time the date is set to a value drawn from the choices. The value is around the date the jitter leaves alone. A choice is a span such as -200ms:200ms or one value such as 1s. A weight in brackets draws a choice more often, as -1s[1],0[8],1s[1] does.

The first step is taken when the guest takes the command, and the next a period later. Each step is a step of the date, as NTP makes one. A jump or a set moves the date under the jitter, which goes on around it. Use jitter off to stop it and return to the date without it.

The draws come from --seed, or from a seed drawn at random and shown. A replay steps by the same amounts at the same points. A fork has its parent’s jitter and draws on from where it was.

Use --for 30s to undo the change after 30 seconds of machine time. The undo takes back the command’s own change, and what a drift gave meanwhile stays. A later change cancels an undo still pending.

Use --container db to change one container’s date alone. The container keeps its offset from the guest’s date across a restart.

1
2
3
machine clock run jitter [-h] [--every duration] [--seed n] [--for duration]
[--container name]
choices
Argument Does
choices The choices, such as -200ms:200ms or -1s[1],0[8],1s[1]; off stops it
Option Does
--every duration Machine time between two steps, such as 10m
--seed n The seed the steps are drawn from (default: one drawn at random)
--for duration Undo the change after this much machine time, such as 30s
--container name A container by its id, its name or its Compose service

Examples:

1
2
3
4
5
machine clock run1 jitter -200ms:200ms --every 10m        NTP's steps
machine clock run1 jitter -1s[1],0[8],1s[1] --every 1m weighted
machine clock run1 jitter -50ms:50ms --every 1m --seed 7 a seed
machine clock run1 jitter 1s --every 5s --container db db alone
machine clock run1 jitter off stopped

See also:

machine clock leap

Insert or delete a second at the next UTC midnight.

The guest kernel inserts 23:59:60 or skips 23:59:59 at the next midnight of the guest’s date. It does so as an NTP daemon’s leap flag asks it to. Then it clears the flag. Use leap off to cancel a leap still pending.

A leap second is the guest kernel’s timekeeping, so it moves every container with the guest’s date. A reset cancels it too.

1
machine clock run leap [-h] [--container name] {insert,delete,off}
Argument Does
{insert,delete,off} Insert, delete, or off to cancel a leap pending
Option Does
--container name A container by its id, its name or its Compose service

Examples:

1
2
3
machine clock run1 leap insert  23:59:60 at midnight
machine clock run1 leap delete 23:59:59 skipped
machine clock run1 leap off cancelled

See also:

machine clock reset

Step the guest’s date and every container’s back to machine time.

Machine time is the date the guest kernel had at boot, moved by elapsed time. The reset undoes every step, those of clock and those a process made, and cancels every pending undo.

Use --container db to return one container to the guest’s date. The guest’s date and the other containers stay as they are.

1
machine clock run reset [-h] [--container name]
Option Does
--container name A container by its id, its name or its Compose service

Examples:

1
2
machine clock run1 reset                 the guest and every container
machine clock run1 reset --container db db alone

See also:

machine rng

Show and seed the RNG device of a machine. See Random numbers.

The RNG device gives the guest its random bytes from a seed. A run takes the seed from create --rng-seed.

Use seed to seed it again here. The run records the seed, and a replay seeds the device at the same point. A guest kernel that reads the seed reseeds its own generator at once. A kernel that does not read the seed picks it up the next time that kernel reseeds.

A run created with --no-rng has no RNG device, and the command refuses it.

1
machine rng [-h] run command ...
Argument Does
run Run name

Examples:

1
2
machine rng run1         the seed and whether the guest reseeds at once
machine rng run1 seed 7 seed the RNG device from 7

See also:

machine rng seed

Seed the RNG device again.

The device gives its bytes from the new seed, as from create --rng-seed. The run records the seed, so a replay seeds the device at the same point.

1
machine rng run seed [-h] n
Argument Does
n Seed, 0 to 2^64-1

Examples:

1
machine rng run1 seed 7  seed the RNG device from 7

See also:

machine regs

Show the registers of a task.

The registers are the user registers of a task of the guest, by its tid. A task on a vCPU has them in the vCPU. A task off every vCPU has them where it saved them. A task in the kernel has the ones it entered with.

Without names the general registers are shown. Use --all to show the x87, SSE and AVX registers as well. Name registers to show those alone.

NAME=VALUE writes a register. A value is a number or a location as mem takes one, such as main+4. The write is part of the run, and a replay makes it again. The selectors and the x87, SSE and AVX registers are not written.

1
machine regs [-h] [--all] run tid [register ...]
Argument Does
run Run name
tid Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process
[register ...] Register to show, such as rip or xmm0, or NAME=VALUE to write one (default: the general registers)
Option Does
--all Show the x87, SSE and AVX registers as well

Examples:

1
2
3
4
machine regs run1 412                   the general registers of tid 412
machine regs run1 412 rip rsp two of them
machine regs run1 412 --all with x87, SSE and AVX
machine regs run1 412 rax=0 rip=main+4 write two of them

Prints:

1
2
text  REGISTER VALUE NATURAL, then where they were read
json tid tgid start where cpu values, each value in hex

Refuses, and the way past:

1
a tid the machine does not have  machine tasks run1

See also:

machine mem

Show the memory of a tgid.

The memory is the tgid’s own, through its page tables. The task names its tgid. Guest kernel memory is not shown.

A location names an address in the task’s process. Every command that takes a location takes one of these:

1
2
3
4
5
6
7
0x401136            an address
*0x401136 an address, as GDB writes it
main a function or a variable, by its name
libc.so.6:malloc a name in one file, by its path or file name
init.c:106 a source line, at its first code
$rsp a register of the task, as it is now
main+4, $rsp-8 any of these moved by +N or -N

A format after the location shows the memory as GDB’s x command does. A format is a count, a letter and a unit, such as 8xg, 4i or s. The letters are x, d, u, o, t, a, c, f, s, i and z. The units are b, h, w and g.

Each line begins with its address and the symbol the address is in. A read that reaches an address that is not mapped ends there.

LOCATION=VALUE writes there, and the write is part of the run. A quoted string writes its bytes. A number takes the size of the variable it is written to, or of a unit after it. A page the tgid cannot write is not written.

LOCATION^=MASK flips the bits the mask sets in the value there, as ^= does in C. The value is as wide as the variable or the unit after it. The flip is a write and is part of the run.

1
2
3
machine mem [-h] [--find pattern] [--dump file] [--restore file] [--fill pattern]
[--length n] [--mapped]
run tid [location] [format]
Argument Does
run Run name
tid Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process
[location] Address to read at: a number, *ADDR, $register, NAME, PATH:NAME or FILE:LINE, moved by +N or -N. LOCATION=VALUE writes there, LOCATION^=MASK flips bits
[format] Count, format letter and unit, as GDB’s x takes them, such as 8xg, 4i or s (default: 8xg)
Option Does
--find pattern Find a pattern in --length bytes: text, or 0x and hex bytes
--dump file Write --length bytes to a file on this computer
--restore file Write the bytes of a file on this computer at the location
--fill pattern Write a pattern over --length bytes: text, or 0x and hex bytes
--length n Bytes a find, a dump or a fill reaches over
--mapped List the pages the tgid maps, with their access and file

Examples:

1
2
3
4
5
6
7
8
9
10
11
12
machine mem run1 412 '$sp'                  eight 8-byte words at the stack
machine mem run1 412 '$pc' 4i the next four instructions
machine mem run1 412 main 16xb sixteen bytes of main
machine mem run1 412 environ a what environ points at
machine mem run1 412 counter=42 write counter, at its size
machine mem run1 412 '$sp+8=-1' w write a word at the stack
machine mem run1 412 'counter^=0x10' flip bit 4 of counter
machine mem run1 412 --mapped the pages the tgid maps
machine mem run1 412 b --find x --length 9 find x
machine mem run1 412 b --dump f --length 8 into file f
machine mem run1 412 b --restore f file f back
machine mem run1 412 b --fill x --length 8 8 bytes of x

Prints:

1
2
text  <address> <symbol+offset>: units, as GDB's x
json tid address format unit lines unmapped

Refuses, and the way past:

1
2
a symbol no file names           machine symbols run1 412 --regex name
a tid the machine does not have machine tasks run1

See also:

machine backtrace

Show the stack of a task.

Each frame is a function the task is in, the innermost first. The frames are walked by the call frame information of the program files. Code with no call frame information is walked by its frame pointer.

The names come from the program files on the machine’s disk as it stands. A frame of a file the disk does not have is named by its file. A frame of a program built with debug information names its source line.

1
machine backtrace [-h] [--frame n] run tid
Argument Does
run Run name
tid Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process
Option Does
--frame n Describe one frame, as GDB’s info frame: where it is and what called it

Examples:

1
machine backtrace run1 412  the stack of tid 412

Prints:

1
2
text  # ADDRESS FUNCTION SOURCE FILE
json tid frames

Refuses, and the way past:

1
a tid the machine does not have  machine tasks run1

See also:

machine call

Call a function in a task.

The task runs the function from where it stands. The arguments go in the registers the x86-64 ABI passes them in. An argument is an integer or a pointer. The value returned is printed by the function’s return type.

The task’s registers are put back after the call. What the function wrote in memory stays written. The call is part of the run, and a replay makes it again.

A breakpoint the call reaches ends it there. The registers are put back then too, and the call is abandoned.

1
machine call [-h] run tid function [arguments ...]
Argument Does
run Run name
tid Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process
function Function to call: a name or a location
[arguments ...] Arguments: expressions as print takes them, integers or pointers

Examples:

1
2
machine call run1 412 square 3  square of 3, by its type
machine call run1 412 fflush 0 flush every stream

Prints:

1
2
text  FUNCTION(ARGUMENTS) = VALUE
json tid function returned abandoned hits

Refuses, and the way past:

1
a tid the machine does not have  machine tasks run1

See also:

machine return

Return from a task’s innermost function.

The task goes on at the instruction after the call, in its caller. The caller’s stack pointer and the registers the function saved are restored, as GDB’s return does. A value goes in rax.

The registers come from the call frame information of the program files on the machine’s disk. Machine refuses the operation if the function has no call frame information.

The write is part of the run, and a replay makes it again. Memory is not changed.

1
machine return [-h] run tid [value]
Argument Does
run Run name
tid Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process
[value] Value the function returns in rax: a number or a location

Examples:

1
2
machine return run1 412    leave the function, rax as it is
machine return run1 412 0 leave it, returning 0

Prints:

1
2
text  Returned from FUNCTION to PLACE, in tid TID
json tid from address function offset source line rax

Refuses, and the way past:

1
a tid the machine does not have  machine tasks run1

See also:

machine print

Print the value of an expression.

An expression is a variable of the frame, a global or a $register. A member, an arrow, an index, a star and an ampersand apply to it as in C. The value is printed by its type.

The types and the variables come from the program’s debug information on the machine’s disk. Use --frame to print in a frame of a caller, as backtrace numbers them.

A format after the expression prints each number in it, as print/x does. The letters are x, d, u, o, t, c and z. Use --type to print the type instead, as whatis and ptype do.

TARGET = VALUE writes VALUE at TARGET and prints TARGET after the write. The write is part of the run, and a replay makes it again.

1
machine print [-h] [--frame n] [--type] run tid expression [format]
Argument Does
run Run name
tid Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process
expression A variable or $register, with .m, ->m, [i], * and &
[format] Format letter for each number, as print/x takes one: x, d, u, o, t, c or z
Option Does
--frame n Frame of the stack, as backtrace numbers it (default: 0)
--type Print its type instead, as GDB’s whatis and ptype do

Examples:

1
2
3
4
5
6
7
8
machine print run1 412 counter          a global or a local
machine print run1 412 'shape->at.x' a member through a pointer
machine print run1 412 'squares[2]' an element of an array
machine print run1 412 '*(int *)arg' through a cast
machine print run1 412 index --frame 1 in the caller's frame
machine print run1 412 counter x in hexadecimal
machine print run1 412 shape --type its type
machine print run1 412 'counter = 3' write it

Prints:

1
2
text  EXPRESSION = VALUE
json tid frame expression type value address

Refuses, and the way past:

1
a name the frame and the globals do not have  machine locals run1 412

See also:

machine locals

List the local variables of a frame.

Each local of the function is printed with its value, by its type. Use --args to list the function’s arguments instead. Use --frame to list a frame of a caller.

The variables come from the program’s debug information on the machine’s disk. A variable the debug information places where it is not read is listed with why.

1
machine locals [-h] [--frame n] [--args] run tid
Argument Does
run Run name
tid Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process
Option Does
--frame n Frame of the stack, as backtrace numbers it (default: 0)
--args The function’s arguments instead

Examples:

1
2
3
machine locals run1 412            the locals of the innermost frame
machine locals run1 412 --args the arguments instead
machine locals run1 412 --frame 1 the caller's locals

Prints:

1
2
text  NAME = VALUE, a line each
json tid frame function variables

See also:

machine symbols

Find the symbols of a tgid.

A name finds the address of each symbol of that name. An address finds the symbol it is in. A symbol of one file is written PATH:NAME. A source line written FILE:LINE finds its code.

Use --regex to find each function whose name matches a pattern. The symbols come from the program files on the machine’s disk.

1
2
3
machine symbols [-h] [--regex pattern] [--variables] [--types] [--scope] [--sources]
[--file path]
run tid [name|address|file:line]
Argument Does
run Run name
tid Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process
[name|address|file:line] Symbol to find the address of, address to find the symbol of, or source line to find the code of
Option Does
--regex pattern Find each function whose name the regular expression matches
--variables Find variables with --regex, not functions
--types Find the types the debug information names with --regex, as info types
--scope List the variables in scope at a location, as info scope
--sources Find the source files the line tables name with --regex, as info sources
--file path File to look in with --regex, by its path or its file name (default: every file the tgid maps)

Examples:

1
2
3
4
5
6
machine symbols run1 412 main                         where main is
machine symbols run1 412 0x401ac6 what is at an address
machine symbols run1 412 data-race.c:56 a source line's code
machine symbols run1 412 --regex ^wait functions that match
machine symbols run1 412 --regex env --variables variables
machine symbols run1 412 --regex ^m --file libc.so.6 of one file

Prints:

1
2
text  ADDRESS SIZE KIND NAME SOURCE FILE
json tid symbols, or tid lines for a source line

See also:

machine libs

List the program and the libraries a tgid maps.

Each row is a file the tgid maps executable, at its range of addresses. The first row is the program. A file the machine’s disk does not have has no symbols.

1
machine libs [-h] run tid
Argument Does
run Run name
tid Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process

Examples:

1
machine libs run1 412  the program and libraries of 412

Prints:

1
2
text  START END OFFSET FILE
json tid libraries

See also:

machine step

Step a task.

A step is one instruction of the task. Use --line to step a source line. Use --over to step over a call, so the function and all it calls are one step. Use --out to run until the function returns.

The machine runs whichever task the guest schedules. A step of one task runs the machine until that task is in its own code, then steps it. Other tasks run meanwhile.

Each step runs the machine, and the run records it, so a replay makes it again. A breakpoint the step reaches ends it. A line step steps out of a function that has no line tables.

A C++ exception that leaves the function ends --over and --out at the catch that takes it, as GDB does. In a C++ program such a step takes 2 of the run’s 3 debug registers. With fewer free, the step is refused.

A longjmp out of the function ends the step after 30 minutes of machine time. Use cancel to end it sooner.

Use --skip NAME with --line to step out of a function or a source file of that name. It does as GDB’s skip does. Give it once for each name.

Use --back to go back over the task’s executions on any vCPU, as reverse-stepi does. An execution is one stretch the task ran, one instruction where the run stepped. The machine switches onto a new branch there, as go does.

With --line, --over or --out, --back goes back over the task’s own code. It does as reverse-step, reverse-next, reverse-nexti and reverse-finish do. It goes back over what the run recorded one instruction at a time. Past that it stops, and says the history ends there.

1
machine step [-h] [--line] [--over] [--out] [--back [n]] [--skip name] run tid [count]
Argument Does
run Run name
tid Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process
[count] Steps to make (default: 1)
Option Does
--line Step a source line, not an instruction
--over Step over a call, the function it calls and all it calls being one step
--out Run until the function returns to the one that called it
--back [n] Go back n steps over the task’s dispatches, onto a branch
--skip name Step out of a function or a source file of this name in a line step (repeatable)

Examples:

1
2
3
4
5
6
7
machine step run1 412                one instruction, as stepi
machine step run1 412 5 --over five, a call being one, as nexti
machine step run1 412 --line a source line, as step
machine step run1 412 --line --over a line over calls, as next
machine step run1 412 --out until the function returns, as finish
machine step run1 412 --back --line back a line, as reverse-step
machine step run1 412 --back --out back to the call, as reverse-finish

Prints:

1
2
3
4
text         Stepped N instructions to WHERE, in tid TID
json tid steps address symbol source hits
back Stepped back N lines to WHERE, in tid TID, on BRANCH
history end No more history before entry N

See also:

machine points

List the breakpoints, watchpoints and catchpoints.

The three share one table and its numbers. Each row has its number and its kind. Use break, watch or catch to change the row of its kind.

1
machine points [-h] run
Argument Does
run Run name

Examples:

1
machine points run1  every breakpoint, watchpoint and catchpoint

Prints:

1
2
text  # KIND ADDRESS SIZE WHERE TGID HITS WHEN
json breakpoints

See also:

machine break

List and manage breakpoints.

A breakpoint stops the run when a task of its tgid reaches its location. The listing has the run’s breakpoints, each by its number. Use points to list them with the watchpoints and catchpoints.

The run keeps them in its run log. A branch has the ones its parent had where it was forked. A run holds 64 breakpoints. Hardware breakpoints and watchpoints share the 3 debug registers.

A breakpoint stops the tasks of its tgid alone. Another process that maps the same file does not stop. A read of the task’s memory shows the program’s byte, not the breakpoint’s.

A run that reaches a breakpoint ends there and names it. The run arms its breakpoints while it records and not while it replays. So run --back finds only the hits the run recorded.

1
machine break [-h] run command ...
Argument Does
run Run name

Examples:

1
2
3
machine break run1               the breakpoints of run1
machine break run1 add 412 main stop the tgid of 412 at main
machine break run1 delete 2 delete breakpoint 2

Prints:

1
2
text  # KIND ADDRESS SIZE WHERE TGID HITS WHEN
json breakpoints

See also:

machine break list

List the breakpoints.

The listing is the one the command prints with no verb.

1
machine break run list [-h]

Examples:

1
machine break run1 list  the breakpoints, by number

See also:

machine break add

Add a breakpoint.

The breakpoint stops the run before the instruction at its location runs. It stops the tgid of the task alone. Another tgid that runs the same address goes on.

The location is resolved now, from the tgid’s program files. The run log keeps the breakpoint, and a branch has its parent’s.

Use --hardware to put it in one of the 3 debug registers, as GDB’s hbreak does. A breakpoint in code the program allocated needs one, as does one in code the program reads.

1
2
3
4
machine break run add [-h] [--disabled] [--ignore count] [--temporary]
[--condition condition] [--print format] [--regex pattern]
[--hardware]
tid [location]
Argument Does
tid Task whose tgid the stop is in
[location] Location, as mem takes one: an address, *ADDR, NAME, PATH:NAME, FILE:LINE or $reg, moved by +N or -N
Option Does
--disabled Add it and do not arm it
--ignore count Pass this many hits where the condition holds (default: 0)
--temporary Delete it at its first hit where the condition holds
--condition condition Count a hit only where this expression holds, as print takes one, such as ‘$rax == 0’
--print format Print this at a hit and go on. {EXPR} prints an expression as print takes one, {EXPR:x} in a print format letter: ‘n={counter} p={$rdi:x}’
--regex pattern Add one at each function whose name matches, in place of a location
--hardware Put it in a debug register, as GDB’s hbreak

Examples:

1
2
3
4
machine break run1 add 412 main           at the start of main
machine break run1 add 412 init.c:104 at a source line
machine break run1 add 412 '*0x401ca9' at an address
machine break run1 add 412 --regex '^tc' at each function that matches

Prints:

1
2
3
text          Breakpoint N at WHERE, in tgid TGID
json number kind address size tgid program location
json --regex breakpoints: each added, as break lists them

See also:

machine break delete

Delete breakpoints.

The run log keeps the delete, so a branch forked before it keeps the breakpoint.

1
machine break run delete [-h] number [number ...]
Argument Does
number [number ...] Number of a breakpoint

Examples:

1
machine break run1 delete 2  delete breakpoint 2

See also:

machine break clear

Delete the breakpoints at a location.

Each breakpoint at the address the location names in the tgid of the task is deleted.

1
machine break run clear [-h] tid location
Argument Does
tid Task whose tgid the breakpoints are in
location Location, as mem takes one

Examples:

1
machine break run1 clear 412 main  delete the stops at main

See also:

machine break save

Save the breakpoints as the commands that add them.

Each line is a machine command that adds one breakpoint again. The command keeps its condition, ignore count and flags. A location written by a register is saved as its address.

1
machine break run save [-h] file
Argument Does
file File to write, or - for the standard output

Examples:

1
2
machine break run1 save stops.sh  write the commands to a file
machine break run1 save - print them

See also:

machine break hits

List the hits of breakpoints.

A hit is a breakpoint that stopped the run. It has the entry it stopped at, its machine time and the number of the breakpoint. The run log keeps each one, so a branch lists its parent’s hits before it.

Use run --back to switch onto a branch at the newest hit.

1
machine break run hits [-h] [-n range]
Option Does
-n, --entries range Hits to list, by the entry each stopped at: N, A:B, A:, :B; or -N for the last N hits, -A:-B counted from the end (default: the last 20)

Examples:

1
2
machine break run1 hits       the last 20 hits
machine break run1 hits -n : every hit

Prints:

1
2
text  ENTRY VTIME # KIND EVENT
json hits

See also:

machine break enable

Enable a breakpoint.

The machine arms an enabled breakpoint and not a disabled one. The run log keeps the change.

1
machine break run enable [-h] [--once | --count n] number
Argument Does
number Number of the breakpoint
Option Does
--once Disable it again after its next hit, as GDB’s enable once
--count n Disable it again after N hits, as GDB’s enable count

Examples:

1
machine break run1 enable 2  arm stop 2 again

See also:

machine break disable

Disable a breakpoint.

The machine arms an enabled breakpoint and not a disabled one. The run log keeps the change.

1
machine break run disable [-h] number
Argument Does
number Number of the breakpoint

Examples:

1
machine break run1 disable 2  keep stop 2 and do not arm it

See also:

machine break condition

Set the condition of a breakpoint.

A hit where the condition does not hold does not end the run. Without a condition the breakpoint has none.

1
machine break run condition [-h] number [condition]
Argument Does
number Number of the breakpoint
[condition] Condition, such as ‘$rax == 0’ or ‘counter > 3’ (default: none)

Examples:

1
2
3
machine break run1 condition 2 '$rax == 0'  where rax is 0
machine break run1 condition 2 'n > 3' where n is above 3
machine break run1 condition 2 no condition

See also:

machine break ignore

Pass the next hits of a breakpoint.

Each hit where the condition holds is counted. A counted hit is passed while the ignore count lasts. A count of 0 stops at the next.

1
machine break run ignore [-h] number count
Argument Does
number Number of the breakpoint
count Hits to pass

Examples:

1
machine break run1 ignore 2 5  pass the next 5 hits

See also:

machine watch

List and manage watchpoints.

A watchpoint stops the run after a task of its tgid writes its bytes, or reads them with --on access. The listing has the run’s watchpoints, each by its number. Use points to list them with the breakpoints and catchpoints.

The run keeps them in its run log, as it keeps its breakpoints. A run holds 3 watchpoints and hardware breakpoints together, one in each debug register. A step over a call takes one of the 3 while it runs. So do step --out and run --until at= or write=.

1
machine watch [-h] run command ...
Argument Does
run Run name

Examples:

1
2
machine watch run1 add 412 counter  a write of counter
machine watch run1 the watchpoints of run1

Prints:

1
2
text  # KIND ADDRESS SIZE WHERE TGID HITS WHEN
json breakpoints

See also:

machine watch list

List the watchpoints.

The listing is the one the command prints with no verb.

1
machine watch run list [-h]

Examples:

1
machine watch run1 list  the watchpoints, by number

See also:

machine watch add

Add a watchpoint.

The watchpoint stops the run after an instruction writes its bytes. Use --on change to stop only where the bytes change, as GDB’s watch does. Use --on access to stop at a read as well. It watches the tgid of the task alone.

A variable is watched at its size. So is an expression, as a local, a member or an element, at the bytes it names when added. A local’s watch stays after its function returns. Any other location takes --size, of 1, 2, 4 or 8 bytes.

The run log keeps the watchpoint, and a branch has its parent’s.

1
2
3
4
machine watch run add [-h] [--disabled] [--ignore count] [--temporary]
[--condition condition] [--print format] [--size bytes]
[--on {write,change,read,access}]
tid location
Argument Does
tid Task whose tgid the stop is in
location Location, as mem takes one: an address, *ADDR, NAME, PATH:NAME, FILE:LINE or $reg, moved by +N or -N
Option Does
--disabled Add it and do not arm it
--ignore count Pass this many hits where the condition holds (default: 0)
--temporary Delete it at its first hit where the condition holds
--condition condition Count a hit only where this expression holds, as print takes one, such as ‘$rax == 0’
--print format Print this at a hit and go on. {EXPR} prints an expression as print takes one, {EXPR:x} in a print format letter: ‘n={counter} p={$rdi:x}’
--size bytes Bytes to watch: 1, 2, 4 or 8 (default: the variable’s size)
--on {write,change,read,access} What stops the run: a write, a write that changes the bytes, a read, or any access (default: write)

Examples:

1
2
3
machine watch run1 add 412 counter               a write of the variable
machine watch run1 add 412 counter --on access any access
machine watch run1 add 412 '*0x4c8b50' --size 4 a write of 4 bytes

Prints:

1
2
text  Watchpoint N at WHERE, in tgid TGID
json number kind address size tgid program location

See also:

machine watch delete

Delete watchpoints.

The run log keeps the delete, so a branch forked before it keeps the watchpoint.

1
machine watch run delete [-h] number [number ...]
Argument Does
number [number ...] Number of a watchpoint

Examples:

1
machine watch run1 delete 3  delete watchpoint 3

See also:

machine watch save

Save the watchpoints as the commands that add them.

Each line is a machine command that adds one watchpoint again. The command keeps its condition, ignore count and flags. A location written by a register is saved as its address.

1
machine watch run save [-h] file
Argument Does
file File to write, or - for the standard output

Examples:

1
2
machine watch run1 save stops.sh  write the commands to a file
machine watch run1 save - print them

See also:

machine watch hits

List the hits of watchpoints.

A hit is a watchpoint that stopped the run. It has the entry it stopped at, its machine time and the number of the watchpoint. The run log keeps each one, so a branch lists its parent’s hits before it.

Use run --back to switch onto a branch at the newest hit.

1
machine watch run hits [-h] [-n range]
Option Does
-n, --entries range Hits to list, by the entry each stopped at: N, A:B, A:, :B; or -N for the last N hits, -A:-B counted from the end (default: the last 20)

Examples:

1
2
machine watch run1 hits       the last 20 hits
machine watch run1 hits -n : every hit

Prints:

1
2
text  ENTRY VTIME # KIND EVENT
json hits

See also:

machine watch enable

Enable a watchpoint.

The machine arms an enabled watchpoint and not a disabled one. The run log keeps the change.

1
machine watch run enable [-h] [--once | --count n] number
Argument Does
number Number of the watchpoint
Option Does
--once Disable it again after its next hit, as GDB’s enable once
--count n Disable it again after N hits, as GDB’s enable count

Examples:

1
machine watch run1 enable 2  arm stop 2 again

See also:

machine watch disable

Disable a watchpoint.

The machine arms an enabled watchpoint and not a disabled one. The run log keeps the change.

1
machine watch run disable [-h] number
Argument Does
number Number of the watchpoint

Examples:

1
machine watch run1 disable 2  keep stop 2 and do not arm it

See also:

machine watch condition

Set the condition of a watchpoint.

A hit where the condition does not hold does not end the run. Without a condition the watchpoint has none.

1
machine watch run condition [-h] number [condition]
Argument Does
number Number of the watchpoint
[condition] Condition, such as ‘$rax == 0’ or ‘counter > 3’ (default: none)

Examples:

1
2
3
machine watch run1 condition 2 '$rax == 0'  where rax is 0
machine watch run1 condition 2 'n > 3' where n is above 3
machine watch run1 condition 2 no condition

See also:

machine watch ignore

Pass the next hits of a watchpoint.

Each hit where the condition holds is counted. A counted hit is passed while the ignore count lasts. A count of 0 stops at the next.

1
machine watch run ignore [-h] number count
Argument Does
number Number of the watchpoint
count Hits to pass

Examples:

1
machine watch run1 ignore 2 5  pass the next 5 hits

See also:

machine catch

List and manage catchpoints.

A catchpoint stops the run at an event of a task rather than at an address. The events are exec, exit, fork, system call, signal, fault, disk request, file mapped and C++ exception. The listing has the run’s catchpoints, each by its number. Use points to list them with the breakpoints and watchpoints.

The run keeps them in its run log, as it keeps its breakpoints. A run holds 8 catchpoints at most.

1
machine catch [-h] run command ...
Argument Does
run Run name

Examples:

1
2
machine catch run1                       the catchpoints of run1
machine catch run1 add fork --in tgid=1 stop where tgid 1 forks

Prints:

1
2
text  # KIND ADDRESS SIZE WHERE TGID HITS WHEN
json breakpoints

See also:

machine catch list

List the catchpoints.

The listing is the one the command prints with no verb.

1
machine catch run list [-h]

Examples:

1
machine catch run1 list  the catchpoints, by number

See also:

machine catch add

Add a catchpoint.

The catchpoint stops the run at an event of a task. The events are exec, exit, fork, vfork, syscall, signal, load, unload, fault and io. The C++ events throw, catch and rethrow are breakpoints in one tgid.

A fault catchpoint stops where an injected fault fires. A syscall fault stops before its delay and its error. The hit names the call and the error, or delayed for a fault with no error. A site fault stops at the site, and the hit names the site.

An io catchpoint stops at a disk request as its task submits it. No disk fault has decided on the request yet, so a fault added at the stop decides on it. The hit names the request’s operation, sectors and tid in the form inject add takes them.

Use --file to stop at the requests of the files a pattern matches, and --op for reads or writes alone. A file matches whether it is opened or mapped before or after the catchpoint is added.

A syscall catchpoint names its system calls, and a signal catchpoint its signals, four at most. A fault catchpoint names system calls or the sites of inject add, such as disk. A load, unload or exec catchpoint names a path or a file’s name. Without one, every event of its kind stops the run.

Use --in to stop at the events of one task or one tgid alone. The run log keeps the catchpoint, and a branch has its parent’s.

Use --in comm=NAME to stop at the events of the tasks of that comm. The comm is read as tasks exist reads it. An exec catchpoint sees the new program’s comm. A task renamed after exec is seen by its new comm at the next fork, exec or exit.

An exit catchpoint sees the comm of the tgid’s leader, its last task to go. A worker of that comm in a tgid whose leader has another does not stop it. A syscall catchpoint with a comm makes the guest report every task’s calls while it is armed.

An exit is the end of a tgid, so an exit catchpoint takes --in tgid=N and not a tid. A throw, catch or rethrow catchpoint is a breakpoint at the C++ runtime’s function. It needs --in tgid=N and takes no values. It is one of the run’s 64 breakpoints.

1
2
3
4
machine catch run add [-h] [--in tid=N|tgid=N|comm=NAME] [--file pattern]
[--op {read,write}] [--disabled] [--ignore count] [--temporary]
[--condition condition] [--print format]
event [value ...]
Argument Does
event Event: exec, exit, fork, vfork, syscall, signal, load, unload, fault, io, throw, catch, rethrow
[value ...] System call of a syscall or fault catchpoint, signal, by name or number, or the path of a load, unload or exec
Option Does
--in tid=N|tgid=N|comm=NAME Stop at the events of this task, tgid or comm alone (default: any task)
--file pattern An io catchpoint’s requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file)
--op {read,write} An io catchpoint’s requests of one operation (default: both)
--disabled Add it and do not arm it
--ignore count Pass this many hits where the condition holds (default: 0)
--temporary Delete it at its first hit where the condition holds
--condition condition Count a hit only where this expression holds, as print takes one, such as ‘$rax == 0’
--print format Print this at a hit and go on. {EXPR} prints an expression as print takes one, {EXPR:x} in a print format letter: ‘n={counter} p={$rdi:x}’

Examples:

1
2
3
4
5
6
7
8
9
machine catch run1 add exec                        any task execs
machine catch run1 add syscall openat write either call
machine catch run1 add signal SIGSEGV --in tid=57 a signal of 57
machine catch run1 add load libssl.so.3 a library mapped
machine catch run1 add throw --in tgid=57 a C++ throw
machine catch run1 add exit --in comm=cat a cat ends
machine catch run1 add fault openat a fault fires
machine catch run1 add io --file '/db/*/c.bin' a disk request
machine catch run1 add io --op read --in tgid=57 a read of 57

Prints:

1
2
text  Catchpoint N: EVENT, in tgid TGID, in tid TID or in comm NAME
json number kind tgid tid comm names

See also:

machine catch delete

Delete catchpoints.

The run log keeps the delete, so a branch forked before it keeps the catchpoint.

1
machine catch run delete [-h] number [number ...]
Argument Does
number [number ...] Number of a catchpoint

Examples:

1
machine catch run1 delete 4  delete catchpoint 4

See also:

machine catch save

Save the catchpoints as the commands that add them.

Each line is a machine command that adds one catchpoint again. The command keeps its condition, ignore count and flags.

1
machine catch run save [-h] file
Argument Does
file File to write, or - for the standard output

Examples:

1
machine catch run1 save stops.sh  write the commands to a file

See also:

machine catch hits

List the hits of catchpoints.

A hit is a catchpoint that stopped the run. It has the entry it stopped at, its machine time and the number of the catchpoint. A catchpoint’s hit also has its event. The run log keeps each one, so a branch lists its parent’s hits before it.

Use run --back to switch onto a branch at the newest hit.

1
machine catch run hits [-h] [-n range]
Option Does
-n, --entries range Hits to list, by the entry each stopped at: N, A:B, A:, :B; or -N for the last N hits, -A:-B counted from the end (default: the last 20)

Examples:

1
2
machine catch run1 hits       the last 20 hits
machine catch run1 hits -n : every hit

Prints:

1
2
text  ENTRY VTIME # KIND EVENT
json hits

See also:

machine catch enable

Enable a catchpoint.

The machine arms an enabled catchpoint and not a disabled one. The run log keeps the change.

1
machine catch run enable [-h] [--once | --count n] number
Argument Does
number Number of the catchpoint
Option Does
--once Disable it again after its next hit, as GDB’s enable once
--count n Disable it again after N hits, as GDB’s enable count

Examples:

1
machine catch run1 enable 4  arm catchpoint 4 again

See also:

machine catch disable

Disable a catchpoint.

The machine arms an enabled catchpoint and not a disabled one. The run log keeps the change.

1
machine catch run disable [-h] number
Argument Does
number Number of the catchpoint

Examples:

1
machine catch run1 disable 4  keep catchpoint 4 and do not arm it

See also:

machine catch condition

Set the condition of a catchpoint.

A hit where the condition does not hold does not end the run. Without a condition the catchpoint has none.

1
machine catch run condition [-h] number [condition]
Argument Does
number Number of the catchpoint
[condition] Condition, such as ‘$rax == 0’ or ‘counter > 3’ (default: none)

Examples:

1
machine catch run1 condition 4 '$rax == 0'  where rax is 0

See also:

machine catch ignore

Pass the next hits of a catchpoint.

Each hit where the condition holds is counted. A counted hit is passed while the ignore count lasts. A count of 0 stops at the next.

1
machine catch run ignore [-h] number count
Argument Does
number Number of the catchpoint
count Hits to pass

Examples:

1
machine catch run1 ignore 4 5  pass the next 5 hits

See also:

Python SDK

The SDK does what the client does, from Python. If you can type it in a terminal, you can call it from a program.

Installing

The SDK runs on the SDK Core, a proprietary compiled module licensed under the Client EULA. The SDK Core is not on PyPI. The client’s installer installs it with the client, so install the client first:

1
curl https://testflows.com/machine/install -fsS | bash

Then install the SDK from PyPI:

1
pip3 install testflows.machine

It needs Python 3.11 or later, on the systems the client runs on: Linux x86_64 and arm64, and Macs with Apple Silicon. If you need another platform, contact us.

Without the SDK Core, import testflows.machine.v1 raises ImportError with the installer’s command. Running the installer again updates the client and the SDK Core together.

Hello Machine

1
2
3
4
5
6
7
8
9
10
import testflows.machine.v1 as machine

client = machine.Client()

with client.sessions.create(cpus=1) as session:
client.sessions.use(session)
with client.create(disk="hello") as run:
run.run(until="tasks", timeout=300)
for line in run.output(lines=-20):
print(line)

This creates a session, boots a run from the disk hello and prints the last twenty lines of its console. A run waits for commands, so it moves only when you call run.run(...); pass daemon=False to have it run to the end by itself. Leaving a with block deletes what it created, whether the block ends normally or with an error. That is how you make sure a session never outlives your program.

The same names

Every client command has a matching SDK call, and the names follow a simple rule.

In the client In the SDK
a top-level command, machine create --disk app a method on the client, client.create(disk="app")
a group, machine sessions use ci-1 an attribute of the client, client.sessions.use("ci-1")
a command that takes a run, machine fork app --at ^boot a method on the run, run.fork(at="^boot")
a group that takes a run, machine tasks app hold 56 an attribute of the run, run.tasks.hold(56)
a positional argument a positional argument
a flag like --no-wait, -w or -f a keyword argument like wait=False, watch=True or follow=True

So once you know the client, you know most of the SDK. Commands lists them, and the client’s --help says what each call takes.

Signing in

The SDK signs in with an API key from TESTFLOWS_MACHINE_API_KEY if it is set. If it isn’t, it uses the login that machine login stored, which it shares with the client.

A program can sign in on its own too. The one-time code arrives by email.

1
2
machine.login.start("me@example.com")
client = machine.login.verify("me@example.com", code="123456")

machine.login.logout() signs out.

Sessions

A client has a current session, like the CLI does. You can set it when you create the client, or later.

1
2
client = machine.Client(session="ci-1")      # like -s ci-1
client.sessions.use("ci-2") # for this client only

sessions.use only changes the session for this client. It never changes it for the machine command. Without a session you can still read your account’s runs: client.runs(), client.branches() and a run’s describe(), lineage() and checkpoints() all work.

client.sessions does what machine sessions does. A session is a context manager, so leaving the with block deletes it:

1
2
with client.sessions.create("ci-1", cpus=4, mem=8192) as session:
print(session.name, session.state)
Call Does
client.sessions.create(name, cpus=..., mem=..., storage=..., class_=...) creates a session and waits until it runs; mem is in MB and storage in GB
client.sessions(), client.sessions.show(name) lists them, or reads one
client.sessions.keep(name, minutes) keeps a session running idle for that long
client.sessions.rename(name, new_name), client.sessions.delete(name) renames or deletes one

Disks

client.disks builds and manages disks, as machine disks does. A build takes the same sources: binary, from_, image or compose.

1
2
3
disk = client.disks.build("hello-py", from_="python:3.12-slim",
add=["hello.py"], command=["python", "hello.py"])
run = client.create("hello", disk=disk.name)

image and compose read your local Docker, as --image and --compose do. Disks says what each source needs.

Call Does
client.disks(), client.disks.show(name) lists your disks, or reads one: name, size_bytes, state, labels
client.disks.rename(name, new_name) renames one
client.disks.delete(name) deletes one
client.preload.disk(name), client.offload.disk(name) copies a disk into the session ahead of time, or drops the session’s copy; preload_and_offload.py shows both

Runs

Here is a typical run, from creating it to branching and replaying.

1
2
3
4
5
6
7
8
run = client.create("app", disk="app", rng=True)
run.run(until="tasks")
run.focus.add("/app")
run.run(until="/app:exit")

booted = run.checkpoint("booted")
branch = run.fork(at="^booted", name="retry")
branch.replay()

And the calls you will reach for most often:

Call Does
client.create(name, disk=..., cpus=..., ...) creates a run and boots it; takes every option machine create does
client.runs(), client.runs["app"] lists runs, or finds one by name
run.run(until=..., iters=...), run.plan(file) drives it
run.checkpoint(name), run.checkpoints() saves a checkpoint, lists them
run.fork(...), run.go(point), run.switch(branch), run.detach(...) branches
run.replay(), run.play(...), run.diff(...) replays and compares
run.hash() the hashes of the machine as it stands, memory and the disk included, as machine hash
run.dump(show="regs"), run.vcpu() the machine’s state, and where one vCPU stopped
run.rng(), run.rng.seed(7), run.fork(at="^booted", rng_seed=7) the RNG device’s seed, seeding it again, and a branch whose guest draws other random numbers
run.run(irq_delay="blk=1ms", timer_slack="0:20ms"), run.irq.pending() hold interrupts a drawn time, draw how late a timer may be, and list the interrupts owed and held
run.describe(), run.state(), run.now(), run.entries(...) reads it
run.output(...) reads what it printed
run.output.search(pattern, from_=..., to=...), run.output.find(pattern), run.output.entry(offset) finds text in what it printed, between two points if you like; each match has the entry that printed it and a place to go to
run.output.sources(), run.output(source="ttyS1"), run.output.lines(source="all") lists the sources that printed, reads one, and reads them all with the source of each line
run.fs.ls(path, at=...), .cat(...), .cp(...) reads its disk at a checkpoint, which at names
run.send(text, device=...) sends input to a device or to a file in the machine, as machine send does; returns a Sent with queued, and process for a file
run.exec(command), run.exec(command, wait=False), run.process(id), run.processes() executes a command in the machine, a list for a program and a str for a shell line, and returns its Process: pid, exit_code, stdout, stderr, events, exits
run.exec(command, container="db", user="root") executes it in a container, named by its Compose service or by its own name, as a user of it
run.exec(command, net="db") runs a program of the machine on db’s network, such as tcpdump on its eth0
run.core(pid, dest, container=...) gets a core dump of a process into a file; the process dies of it
run.profile("2s", container=..., pid=...) samples what runs for a time and returns a Profile: samples, functions, stacks, and folded text for flame graph tools
run.profile(from_="^request", to="^reply") profiles a stretch the run already recorded, without changing the run
run.profile("2s", in_guest=True) samples with a process in the run, for the functions of the kernel by name
run.containers(), run.containers.kill("db", restart="5s"), run.containers.crash("db", restart="0") lists the containers, the Compose service of each, and its addresses and links; stop, kill, crash, start, pause, resume, throttle, memory, pids, ports and files put a fault on one, as machine containers does
run.put(source, path, container=..., mode=..., owner=...) puts a file of any size into the machine or into a container of it; source is bytes or the path of a file
run.get(path, dest, container=...) gets a file out, as it is now; returns a Got with size, mode, uid, gid, and data when no dest is given
run.tasks(), run.focus.add(...), run.irq(...), run.vtime.rate(...) steers the machine
run.inject.add("syscall", "read", "valid", in_="/usr/bin/app", pick=1, prob=0.1), run.inject.add("syscall", "read", delay="0:20ms", in_="/usr/bin/app"), run.inject.add("disk", in_="all", prob=0.01), run.inject.add("function", "malloc", ret=0, prob=0.01, in_="/usr/bin/app"), run.inject.add("disk-corrupt", in_="all", file="/var/lib/db/*/data.bin", op="write", bytes=8, times=1), run.inject(), run.inject.delete(2), run.inject.clear() injects faults as machine inject does: system calls fail with chosen errors or wait first, a program’s function returns ret without running, and the kernel’s disk, allocation, usercopy and futex-access sites fail, and the disk corrupts, tears or loses data; lists the faults with their counts, and deletes them; in_ takes the words --in takes, delay the span --delay takes, at the tid whose files hold the function, as --at takes it, and file, op, sectors and bytes the disk options
run.tasks.exist("/usr/bin/app") returns the tasks running that program, comm, tid or tgid now, or raises NotFound; it changes nothing
run.task(tid) a guest task to debug, by its tid: registers(), read(address, size), address("main+4"), symbol(address), symbols(regex="^wait"), backtrace(), libraries()
task.set(rax=0, rip="main+4"), task.write(address, data) writes its registers and its tgid’s memory; a replay makes the writes again
task.print("shape->at.x", frame=1), task.print("counter = 3"), task.locals(args=True) an expression in a frame, a write with TARGET = VALUE, the locals or the arguments of a frame
task.step(), task.step(line=True, over=True), task.finish() steps an instruction or a line, over calls, or out of the function
task.back(), task.back(line=True, over=True) goes back over the task’s code onto a branch, as reverse-stepi and reverse-next
task.call("strlen", "buf"), task.return_(0) calls a function in the task, or returns from its innermost one
task.breakpoint("main"), task.watch("counter", on="access") stops its tgid at a location, or at a write or an access of its bytes; run.run(...).hits names the one the run ended at
run.breakpoints.catch("fork", in_="tgid=1"), run.breakpoints.catch("io", file="/db/*/data.bin", op="write") stops at an event of a task: exec, exit, fork, vfork, syscall, signal, load, unload, fault, io, throw, catch, rethrow; in_ takes the words --in takes, tid=N, tgid=N or comm=NAME; file and op are an io catchpoint’s, as --file and --op
run.breakpoints.list(), .enable(2, False), .condition(2, "n > 3"), .ignore(2, 5), .delete(2), .hits() the run’s stops, changed one at a time, and the hits the run log keeps
run.run(iters=0), run.run(until="tid=412:at=main"), run.back() runs to the next hit, or until a task reaches a location; goes back onto a branch at the newest hit
run.tasks.signal(412, "SIGTERM") sends a task a signal; a replay sends it at the same instruction
run.start(), run.stop(), run.pause(), run.kill(), run.delete() lifecycle
run.start(at="^booted"), run.start(at="^booted", read_only=True), run.start(recover=False) replays from a checkpoint to the tip, opens a checkpoint to read, or refuses to replay the tail, as machine start does
run.fork(from_="^booted", name="b"), run.rewind("^booted") a branch forked at the checkpoint before a point and replayed to it; the run put back at a checkpoint
run.sync(), run.cancel(), run.cancel(soft=True) publishes the run to your storage; cancels the drive or checkpoint in progress, at once or, with soft=True, at the end of the dispatch in progress so the run stays repeatable
client.create(disk="app", instr_margin=100000, dispatch_bound="2s"), run.replay(instr_margin=100000), run.describe().repeats_to sets where a recording stops, as Recording that repeats describes; replays with that margin; the first entry where recording the run again stops repeating, None when it repeats to its tip
run.keep() keeps a run created in a with block when the block exits, such as one that just failed, with its tree
run.marks.add("good"), run.marks.list(), run.marks.remove("good"), client.marks.list() names a branch, as Marks describes; the last lists every mark
run.tasks.hold(56), .release(56), .yield_(56), .slice(56, "10us"), .pin(56, "2") steers one thread, as Holding threads describes
run.syscalls.counts("/usr/bin/app"), run.ints.counts(...), run.vtime.add("5s") counts a task’s system calls or interrupts; pushes the clock forward
run.debug.log(lines=-20), run.debug.err(), run.control.log() the run’s debug output, its error log, and the commands issued to it

Every call returns a record with named fields, such as a RunInfo from describe() or a list of TaskInfo from tasks(). Call as_json() on one to get a plain dict.

run.run(...) and run.plan(...) return a RunResult, which says how far the drive went. An iteration has no fixed length, so iterations alone does not tell you:

Field Is
iterations the iterations that ran
vtime_elapsed how far the machine’s clock moved, in nanoseconds
reached whether until held; False when iters ran out first
clauses each clause the drive drew from, with the iterations it applied to
entry_id the entry the drive ended at
entries where each vCPU stopped
hits the breakpoints, watchpoints and catchpoints the drive ended at; empty when none ended it
printed the lines printed by stops that print, in the order the stops hit
1
2
3
done = run.run(iters=300, when="/app:on", mode="step")
for clause in done.clauses:
print(clause.clause, clause.matched) # when /app:on mode step×246 246

until takes the conditions machine run --until does, including a machine time: run.run(until="vtime=+10ms").

Bounding a drive

within bounds a drive by the machine’s clock, as --within does. With until, the one that happens first ends the drive, and reached says which. The machine checks the bound itself, so a slow host never ends the drive early:

1
2
3
done = run.run(until="console~ready", within="30s")
if not done.reached:
print("not ready after 30s of machine time")

timeout is different: it is how many seconds of your time the call waits, and it raises Timeout when they run out. To wait for a run that something else drives, such as another program or a run stopping in the background, use run.wait:

1
2
run.stop(wait=False)
run.wait(for_="stopped", timeout=60)

for_ takes the predicates machine wait does: ready, stopped, halted, entry>=N, vtime>=T, console~REGEX, or a task event such as /app:exit.

Scheduling from Python

Every option of machine run is a keyword. The draws described in Schedules look like this, on a run whose program was focused with run.focus.add("/app") before booted, so that its branches have the focus too:

1
2
3
4
for seed in range(10):
branch = run.fork(at="^booted", name=f"seed-{seed}")
done = branch.run(until="/app:exit", within="60s", mode="step,fast",
step="1:200", task_sched="keep:hold:yield", seed=seed)

The explore_seeds.py example keeps the branches that fail and marks them.

Faults

run.inject adds the faults Fault injection describes, with the same subsystems and options as keywords. Check the target first with run.tasks.exist, which raises NotFound when nothing runs it:

1
2
3
4
5
6
run.tasks.exist(f"tgid={pid}")
run.inject.add("syscall", ["open", "openat"], "ENOENT", in_=f"tgid={pid}", times=3)
run.run(within="5s")
for fault in run.inject():
print(fault.number, fault.syscall, fault.matched, fault.fired)
run.inject.clear()

A C library opens a file with open or openat, depending on the library, so the fault takes both. glibc’s open() makes no open call, so the add issues a UserWarning that names openat; the fault on open is added all the same. matched counts the calls the fault saw and fired the ones it failed. A disk fault takes the file and the operation. The kernel’s flusher sends an ordinary write, so the example syncs it, and the request is dd’s own:

1
2
run.inject.add("disk", in_="comm=dd", file="/var/data.bin", op="write")
run.exec("dd if=/dev/zero of=/var/data.bin bs=4096 count=1 conv=fsync", within="10s")

The fault_injection.py example runs both and prints what the program saw.

on and off make a fault cycle: run.inject.add("disk", in_="all", on="2s", off="5s") fails every disk request for 2 seconds in every 7 of machine time. Each Fault says where it is: active, and left_ns, the time left in that window.

Container faults

run.containers puts a fault on one service of a Compose project, as Container faults describes, while the others keep running. A crash loses what the service had not synced to the volumes its project names under x-machine:

1
2
3
4
5
6
7
8
run.exec("echo synced > /data/kept && sync", container="app")
run.exec("echo in memory > /data/lost", container="app")

crashed = run.containers.crash("app", restart="0")
print(crashed.volumes) # the volumes the crash cut
run.run(within="5s")

print(run.exec("ls /data", container="app").stdout) # kept is there, lost is not

stop, kill, start, pause, resume, throttle, memory, pids, ports and files work the same way. stop and kill take restart= as crash does, stop takes grace=, fresh=True starts a new container instead of the old one, pause and throttle take for_=, throttle takes cpus=, and memory("db", "64m", for_="30s") limits a container’s memory. pids("db", 20), ports("api", 40000, 40009) and files("api", 64) run a container out of processes, local ports or open files, and each takes for_= too. run.containers() lists the containers with the service, name, state and limits of each. The container_crash.py example builds a project, crashes one service and reads its volume after the restart.

Clock faults

run.clock changes the guest’s date or one container’s, as Clock faults describes. Each method returns the date as it stands, with the change waiting for the run to execute:

1
2
3
4
5
6
7
8
9
run.clock.jump("+59m50s")
run.run(within="30s", until="console~token refresh")
print(run.clock().offset) # ns from machine time

run.clock.drift("+200ppm", container="db")
run.clock.jump("+4s", for_="2m", slew="30s", container="db")
run.clock.jitter("-200ms:200ms", every="10m", seed=42, container="api")
run.clock.leap("insert")
run.clock.reset()

run.clock(container="db") reads one container. ClockInfo.containers lists each container off the guest’s date with its offset, drift, slew and jitter, and ClockInfo.jitter and ClockInfo.leap are the guest’s.

The guest kernel takes a change when the run executes, so read the clock after that run to see whether it was refused. ClockInfo.refused names each change of the last batch the guest refused, with its errno and the reason:

1
2
3
4
run.clock.set("2038-01-19T03:13:30Z")
run.run(within="1s")
for refused in run.clock().refused:
print(refused.command, refused.error, refused.reason)

To find how much skew a system tolerates, fork one checkpoint into branches, one skew each:

1
2
3
4
5
for skew_ms in (50, 100, 200, 400, 800):
branch = run.fork(at="^steady")
branch.clock.jump(f"+{skew_ms}ms", container="node2")
result = branch.run(within="2m", until="console~two leaders")
print(skew_ms, result.reached)

Network faults

run.net puts faults on the links between a run’s containers, as Network faults describes. A fault acts on every packet a container sends, or on those a match takes. A match is any of to, a peer; proto, tcp, udp, icmp or arp; port, a source or destination port; and syn=True, the TCP SYNs that open a connection:

1
2
3
4
5
6
7
8
9
10
run.net.delay("db", "100ms", jitter="20ms", seed=7)
run.net.loss("db", "10%", to="api")
run.net.loss("api", "100%", to="db", port=5432, for_="5s")
run.net.partition("db", "nest", for_="10s")
run.run(within="30s")

for link in run.net():
print(link.container, link.peer or "-", link.port, link.delay, link.loss, link.seed)

run.net.heal()

delay, loss, duplicate, corrupt, reorder, rate and hold return a LinkChange, the link before and after; partition returns one for each link it cuts, and heal and release return the links as they were before. duplicate, corrupt and reorder take a share as loss does, and rate takes bits a second, such as "1mbit". A Link gives its delay and jitter in nanoseconds and its shares in parts per billion, and what the fault did since it was set: sent, dropped, and queued, the packets it holds now. A replay delays and drops the same packets.

log=True on a fault writes each of its decisions to the container’s file under /output/drop/.machine/net, and log=False stops it, as --log and --no-log do:

1
2
3
4
run.net.loss("db", "30%", to="nest", seed=7, log=True)
run.run(within="5s")
for line in run.output(source="/output/drop/.machine/net/compose-db-1.log"):
print(line)

hold keeps a link’s packets until release, or for for_, and they then leave together. With no name, hold holds every link where the machine stopped, until release with no name. reset aborts TCP connections and returns how many:

1
2
3
4
run.net.hold("node1", for_="10s")         # node1's messages arrive late, in a burst
run.net.hold() # every link, at the machine's stop
run.net.release() # every packet held leaves
run.net.reset("api", to="db", port=5432) # api's connections to db get a reset

run.net.arp shows and changes a container’s neighbour table, and returns the table as Neighbor entries:

1
2
3
4
5
run.net.loss("db", "100%", proto="arp", to="api")   # db's ARP to api lost
run.net.arp.set("api", "db", "02:42:ac:12:00:99") # a wrong MAC for db
for entry in run.net.arp.show("api"):
print(entry.address, entry.peer, entry.mac, entry.state)
run.net.arp.flush("api")

Stopping at an event

A catchpoint stops the run at an event of a task, as machine catch does. The drive that reaches it returns the hit in hits:

1
2
3
4
5
6
stop = run.breakpoints.catch("syscall", "open", "openat", in_="comm=cat")
run.exec(["cat", "/etc/hostname"], wait=False)
done = run.run(within="10s")
for hit in done.hits:
print(hit.number, done.entry_id, hit.event)
run.breakpoints.delete(stop.number)

The kinds are exec, exit, fork, vfork, syscall, signal, load, unload, fault, io, throw, catch and rethrow. An io catchpoint takes file= and op=, and stops at a file’s disk request before any disk fault decides on it:

1
run.breakpoints.catch("io", file="/var/lib/db/*/data.bin", op="write")

Debugging a task

run.task(tid) is a task to debug while the machine is stopped, by its tid, as Debugging describes for the command line:

1
2
3
4
5
6
task = run.task(412)
regs = task.registers()
print(regs["rip"], regs["rsp"])
print(task.read(regs["rsp"], 16))
for frame in task.backtrace():
print(frame.level, frame.address, frame.function, frame.path)

The debug_task.py example stops at a catchpoint, reads the registers of the task that hit it, reads the path it was opening from its memory, and prints its stack.

Points in a run’s history

Calls that take a point, such as fork(at=...), go, rewind, replay(to=...) and profile(from_=..., to=...), take it as text, "^booted", "#300", "1.194s" or "-10ms", or as one of three records:

Record Is Works on
Point a time: vtime, the machine’s clock in nanoseconds every run; each finds its own entry at that time
Place one entry: the run_id that recorded it, entry_id, point and vtime_offset the run that recorded it, and branches forked after it
Checkpoint a saved state: its name, and the place it was saved at wherever its place works

start(at=...) and the fs readers take only a checkpoint, by its name or as a Checkpoint: they open a saved state, and a time or an entry is not one.

A point moves by a duration: an int of nanoseconds, a string such as "10ms", or a timedelta. Subtract one point from another for the nanoseconds between them. run.now() returns a Place, and run.place(point) finds the entry a point names.

1
2
3
4
booted = run.checkpoint("booted")
branch = run.fork(at=booted.place.point + "10ms")
here = branch.now()
run.place(here.point) # the same time, on the parent

A place is one run’s entry, so once the parent and the branch have both run past the fork, run.place(here) is refused: the parent’s entry with that id is another state. Pass here.point to take the same time instead.

Watching and following

A listing takes watch=True and then gives you a new list every time something changes. A log takes follow=True and gives you new lines as they come, like tail -f.

1
2
3
4
5
for runs in client.runs(states=["running"], watch=True, timeout=60):
print([r.name for r in runs])

for line in run.output(follow=True, timeout=60):
print(line)

Choosing entries

Anything that reads entries takes the same few spellings.

You write Means
200 or (100, 200) that entry, or that range, both ends included
"100:200", "100:", ":200" a range, from 100 on, or up to 200
"-20" or "-20:" the last 20 entries
"-20:-2" the 20th entry from the end through the 2nd
slice(100, 200) a Python slice, so 100 through 199
slice(-20, -1) a Python slice from the end, so the same as "-20:-2"

Be careful with the slice rows. A pair and a string include both ends, the way Machine’s own messages do. A slice is Python’s, so it stops before the end. That makes (100, 200) one entry longer than slice(100, 200).

Both ends of a range count from the same end: "-20:5" and slice(-20, 5) are refused.

lines= selects lines of output in the same grammar: 20 is line 20, -20 the last 20, "-10:-2" the tenth line from the end through the second, and 0 none, which starts a follow at the end.

Cleaning up

A session costs from the moment you create it, and a run keeps its log and checkpoints until you delete it. Both can be used as context managers, as in the first example, and deleting a run takes its branches with it.

If you want to keep a run that found something, don’t wrap it in a with block. Nothing is deleted until you call delete().

Errors

Every exception the SDK raises is a machine.Error, and each one says what went wrong.

Exception Raised when
NotSignedIn there are no credentials
Unauthorized a key or a sign-in code was rejected
Forbidden your account is not allowed to do that
NotFound nothing matches the name or id
AlreadyExists something with that name exists
InvalidArgument an argument is wrong, or a name matches more than one thing
NotRunning it exists, but it is not running
Conflict the request does not fit the thing’s current state
NoCapacity no host of the session’s class has room for it now; the same request can succeed later, or a smaller session or another class may fit
Diverged play() could not follow the recording; replay() returns a divergence in its result instead
Timeout a wait ran out of time; the work goes on in the service
ServiceUnavailable the service could not be reached
ApiError the service refused the request

A replay that does not reproduce the recording is a result, not an exception, so check it:

1
2
3
result = branch.replay()
if not result.converged:
print(branch.diff(entry=result.entry_id).diverged)

The exceptions you will usually handle are the ones with a way out. A full class can be retried or sized down, and a run that stopped can be started again:

1
2
3
4
5
6
7
8
9
try:
session = client.sessions.create("ci-1", cpus=8)
except machine.NoCapacity:
session = client.sessions.create("ci-1", cpus=4)

try:
run.run(until="console~ready", within="60s")
except machine.NotRunning:
run.start()

The error_handling.py example goes through every exception.

Examples

Small example scripts are in sdk/python/examples of the TestFlows Machine repository. Each one creates its own session and deletes it at the end. Most run as python <script> --disk <name>. disks.py builds the disk first and takes --binary and --name. container_crash.py builds its own disk too and takes --name, and deletes the disk at the end.

Script Shows
hello.py creates a run, boots it, reads its console
disks.py builds a disk from a static executable, boots a run from it, renames the disk and deletes it
checkpoint_and_fork.py a checkpoint after boot, and branches from it
replay_verify.py replays a recording and names what diverged
explore_seeds.py one branch per seed, with the failing ones marked and kept
force_interleaving.py holds one thread while another runs past it
plan_file.py drives a run with when and else clauses
watch_and_follow.py watches a listing and follows a console
syscall_profile.py counts a program’s system calls and interrupts
read_fs.py reads and copies files from a checkpoint’s disk
machine_time.py changes the machine’s clock (vtime) rate and pushes it forward
inject_interrupt.py injects an interrupt and replays it
stop_and_resume.py what stop, kill, start and sync keep
resume_elsewhere.py stops a run in one session and starts it in another
preload_and_offload.py preloads a disk onto the session, then drops the session’s copy
ci_job.py a CI job with an API key, a session per job and an exit code
error_handling.py every error and how to recover from it
fault_injection.py fails a program’s opens with a syscall fault and its synced write with a disk fault, and prints what the program saw and the fault counts
container_crash.py builds a Compose project with an x-machine volume, crashes one service, and reads what the volume kept; takes --name for the disk it builds
debug_task.py stops at a catchpoint on open, reads the task’s registers, memory and stack

License

The SDK is licensed under Apache 2.0, and its source is in sdk/python of the TestFlows Machine repository. The SDK Core it runs on is proprietary and is not part of the SDK. It comes with the client, under the Client EULA.

ClickHouse is a registered trademark of ClickHouse, Inc. See clickhouse.com.