What is it?
TestFlows Machine is a deterministic execution environment where you run, explore, debug and test your software. It is built on our deterministic hypervisor, which runs your programs, Docker images and Docker Compose environments unchanged.
Nothing in the machine is left to chance. It controls time, interrupts, random numbers and input from devices, and it records all of them while your program runs. Replay the recording and the program does exactly what it did the first time, instruction for instruction.
That gives you three things to work with:
- Record. Every run keeps a log of everything that could have turned out differently.
- Replay. Play the log back and the run repeats itself. If it failed once, it fails every time.
- Branch. Stop at any point, branch from it, and try something else. The original run stays as it was.
If you have ever chased a test that fails one run in fifty, you know why this is handy. Record the failing run once, then replay it as often as you need.
You work with Machine through a client called machine, or from Python using
the SDK. To begin, install the client. It is
one command.
Questions, feature requests and bug reports are welcome in the TestFlows Machine repository on GitHub. See Getting help.
Concepts
There are seven of them, and the rest builds on them.
Disk. What a machine boots. It is your program (or a whole application)
packaged with a small Linux system. You build one with machine disks build.
Session. The vCPUs and RAM you allocate, which one or more runs execute in. It is what you are billed for, from the moment you create it until you delete it, so delete it when you are done.
Run. A deterministic machine booting a disk inside a session. Every run has a name. Nearly every action is something you do to a run.
Entry. A run writes its history as a log. Each item in the log is an entry. Entry #42 is always the same point in that run.
Checkpoint. A saved copy of the whole machine at one point in the log. Every run has checkpoint 1, taken at its first entry, and you can add your own and name them.
Branch. A run that starts from another run at some point. It shares history
with its parent up to that point and has its own after that. You create one
with machine fork or machine go. Runs that grow out of one another form a
tree.
Mark. A second name for a branch, given after the fact. The generated name
says where a branch started. A mark says what it turned out to be,
like crashed.
Points
A lot of commands take a point, which is a place in a run’s history. You can name one in several ways.
| Point | Means |
|---|---|
^booted |
the checkpoint named booted |
#42 |
entry 42 |
1.5s |
a time on the machine’s clock |
-10ms |
ten milliseconds of machine time back from where the run is now |
now |
where the run is now |
parent |
the point this run branched from |
root |
the start of the tree |
%crashed |
the branch with the crashed mark |
console~login |
where the console first printed a match of login |
The same points work everywhere a point is taken: --at on fork, detach
and go, rewind ... to, and replay --to. --from takes them too, except
now and a mark. A bare word is
read as a checkpoint’s name first, then as a time, then as an entry, so write
^name or #42 when you mean one of them for sure.
A branch can only start from a checkpoint.
machine go and machine fork
take any of these points. If the one you name isn’t a checkpoint, they start
the branch at the nearest checkpoint before it and replay the rest of the way
for you. fork --no-start refuses such a point, because it leaves no machine
to replay on, and so does detach. A machine
time such as 1.5s or -10ms names the first entry at the last machine time
at or before the one you asked for.
Getting started
The client
You work with Machine through a client called machine. Install it with one
command:
1 | curl https://testflows.com/machine/install -fsS | bash |
The client is given to a TestFlows account, so the installer first signs you
in, or creates your account, at the terminal: you give your email and the code
sent to it. An account that has not accepted the current Terms of Service and
Client EULA is shown both and accepts them before anything is downloaded. The
installer signs out when it ends, so sign in with machine login afterwards.
Without a terminal, as in CI, the installer takes an API key from
TESTFLOWS_MACHINE_API_KEY instead, from an account that has accepted them.
The installer then downloads the client over HTTPS, checks it against the
SHA-256 checksum published next to it, and puts machine and machine-env in
~/.local/bin. If the checksum doesn’t match, nothing is installed. Where
another program named machine is already on your PATH, as
/usr/bin/machine is on a Mac, machine goes to ~/.local/share/machine/bin
instead, which is not on your PATH, and ~/.local/bin holds machine-env
alone. If ~/.local/bin isn’t on your PATH yet, the installer says so and
shows the line to add for your shell. The installer also puts the SDK Core,
which the Python SDK runs on, in
~/.testflows/machine/sdk, checked the same way. Run the same command again
whenever you want to update.
Supported systems. The client runs on Linux, x86_64 and arm64, and on Macs
with Apple Silicon. It needs only the OpenSSH client, which macOS and most
Linux systems already have, and the installer never asks for root. If ssh is
missing, install it with your package manager, for example
sudo apt install openssh-client on Debian and Ubuntu. It doesn’t run on Intel Macs or natively on Windows; on Windows
you can use it inside WSL. On an ARM machine, a Mac with Apple Silicon
or arm64 Linux, the disks you build must still hold x86_64 programs; see
building disks on an ARM machine. If you
need another platform,
contact us and tell us which one. The
download page has the full list, and a way to check the
download yourself.
Behind a proxy that inspects TLS. The client trusts only Let’s Encrypt,
which issues the certificate of the Machine API, so a certificate from any other
CA is refused, including a proxy’s. The error names
TESTFLOWS_MACHINE_API_CA_FILE, an environment variable the client reads: set
it to a PEM file of the CAs to trust instead, such as your proxy’s.
1 | export TESTFLOWS_MACHINE_API_CA_FILE=/etc/ssl/certs/corporate-proxy-ca.pem |
The file replaces the Let’s Encrypt roots rather than adding to them. If the client must also connect without the proxy, add the Let’s Encrypt roots to the same file. The variable applies to the connection to the API alone: sessions are reached over SSH with certificates TestFlows issues, and image pulls use your system’s CAs. A file that is missing or holds no certificate is refused, and the error names the file. The Python SDK reads the same variable.
On a Mac, use machine-env. macOS ships its own /usr/bin/machine, which
prints the processor type, so typing machine may not run the client. Putting
the client first on your PATH would hide the system command for every other
program. machine-env, which the installer writes on every system, starts a
shell in which machine is the client and changes nothing else. Its prompt
starts with (machine), your usual shell setup is loaded, and exit leaves. It
supports zsh and bash.
1 | $ machine-env |
The (machine) at the start of the prompt says you are in it: there machine is this client. exit leaves it, back to the shell you were in. To run one command there without starting a shell:
1 | machine-env -- machine --version |
That form runs one command in that environment and starts no shell, which
suits scripts and CI. To see which machine you get, run which machine:
/usr/bin/machine is the system’s, and the client is
~/.local/share/machine/bin/machine on a Mac or ~/.local/bin/machine on
Linux. If
machine --version prints a processor type instead of a version, you are
running the system command. On Linux, if another command on your system is
also named machine, use machine-env the same way.
To read the installer first, install a specific version or into another folder, check the download by hand, or uninstall, see the download page.
Run machine with no arguments, or with --help, to see every command. Each
command has its own --help too, and that is the place to look when a flag here
is not enough.
1 | machine --help |
machine --version prints the client’s version and its license. Include the
version when you report a problem.
1 | machine --version |
1 | ---- o o o ---- |
If something does not work, machine ping checks that you can reach the
service and your sessions, and how long the round trip takes.
In machine-env, Tab already completes commands, options and their values,
and the names of your runs. Where machine is the client in every shell, as on
most Linux systems, add the line for your shell to its startup file to have
the same there. On a Mac, use machine-env instead: outside it, machine is
the system’s command, and the line would run that one.
1 | eval "$(machine completion bash)" # ~/.bashrc |
The client answers each Tab itself, from its own commands, so a newer client needs no newer line.
To try it in one terminal first, run the same line there instead. Tab completes in that shell until it exits, and no other shell changes.
Signing up
The installer creates your account if you have none. You can also create it on
the sign-up page, or from the terminal with machine account signup. Then sign in from the terminal.
1 | machine login |
Machine emails you a one-time code and asks for it. You stay signed in until
you sign out with machine logout. machine account show tells you who you
are signed in as.
For scripts and CI there is no need to sign in at all. Create an API key and
put it in the TESTFLOWS_MACHINE_API_KEY environment variable.
1 | machine account api-keys create ci --expiry 90 |
When you create your first session, Machine sets up your account’s storage
first, so that session takes longer to create. To do that step on its own
beforehand, run machine account provision.
Your first run
If you have not yet signed up and signed in, do that first: see Signing up.
This example creates a session, builds a disk, runs a program, creates a checkpoint and starts a branch from it. The program is one line of Python, and you need nothing installed but the client: no Docker and no compiler.
First, create a session. Machine makes it the one your commands use.
1 | machine sessions create first --cpus 2 |
1 | ✔ Session first is running. |
Next, save this as hello.py.
1 | print("hello, world") |
Build a disk named hello that runs it. --from names the public Python
image, --add puts your file in it, and what follows -- is the command the
machine runs.
1 | machine disks build --from python:3.12-slim --add hello.py hello -- python hello.py |
1 | ➤ Packing hello (1 service) |
Only your file is uploaded. The build pulls the image itself and writes it onto the disk, which takes about two minutes for an image of this size. You build a disk once and create as many runs from it as you like.
A program you compile, an image of your own, or several services together build the same way from other sources. See Disks.
Create a run from the disk. The machine waits for your commands. With
--no-daemon it would boot and run to the end by itself.
1 | machine create hello-run --disk hello |
1 | ✔ Started hello-run (whUxdR33Zad9Qxc0ASfig) |
Now the run is waiting. It only moves when you drive it, and machine run is how
you drive it. The --until tasks part means “keep going until Linux is up.”
1 | machine run hello-run --until tasks |
1 | iterations: 15646 run ic: 271676660 vtime: +0.692741050s |
Read the last five lines the machine printed. --plain prints the lines
alone. Without it, a terminal also shows the entries, the source and the
number of each line.
1 | machine output hello-run -n -5 --plain |
1 | futex hash table entries: 256 (order: 2, 16384 bytes, linear) |
To see the full console log, use -n :. Ranges use the same colon as a Python slice. Lines are numbered from 1 and both ends count, so -n 1: is the first line through the last.
A negative number counts from the end, and -1 is the last line. -n -5 and -n -5: are the last five lines, and -n -10:-2 is the tenth line from the end through the second. Both ends of a range count from the same end.
Create a checkpoint of the machine’s state so you can come back to it, and start a branch from there.
1 | machine checkpoint hello-run booted |
1 | ✔ Created checkpoint booted at entry 15792 |
The checkpoint is in your session. machine sync publishes the run, so the
checkpoint outlasts the session and another session can start from it.
Stopping the machine publishes it too.
1 | machine fork hello-run --at ^booted --name try-1 |
1 | ✔ Forked hello-run/try-1 (whUxfLYSFJMDHzKaiBOvH) |
The branch starts exactly where booted was taken. Notice its full name,
hello-run/try-1. A branch is named after the run its tree starts from, a
slash, and the name you gave it. Look at the tree.
1 | machine branches hello-run |
1 | ● hello-run (whUxdR33Zad9Qxc0ASfig) [0, ∞) ← this run |
Linux is up on the branch and your program has not started yet. Drive the branch until the machine shuts down, which it does when the program exits.
1 | machine run hello-run/try-1 --until halted |
1 | iterations: 17148 run ic: 2095789032 checkpoints: 1 |
The program’s own lines start with app-1. Find them in the branch’s console.
1 | machine output hello-run/try-1 -n : | grep app-1 |
1 | app-1 | hello, world |
The parent hello-run is still at booted, with the program yet to run. You
can fork it again and get the same start every time.
When you are done, delete the runs and then the session. A run keeps its log and checkpoints until you delete it, and a session costs money until you do.
1 | machine delete hello-run --recursive --stop --yes |
That’s the whole loop. You built a disk, created a checkpoint in a run, branched from it and ran a program to its end. Everything else in this page is a variation on it.
Getting help
TestFlows Machine is in public beta, and reports from users directly influence what gets fixed and built next. Report bugs, request features and ask questions in the TestFlows Machine repository on GitHub.
To report a bug, search the issues first, then
open an issue with the commands you ran, what you
expected, what happened, and the output of machine --version. To request a
feature, open a feature request that describes what
you’re trying to do. To ask a question or share an idea, start a discussion in
Discussions. You can also send feedback through the
contact form.
✋ Issues and discussions are public. Never include an API key, a token, a password or other secrets in them.
For questions about your account or billing, email support@testflows.com.
For privacy-related questions, email privacy@testflows.com. Report
abuse to abuse@testflows.com. Report security vulnerabilities to
security@testflows.com, or through GitHub’s
private vulnerability reporting.
Disks
A disk is what a machine boots, and machine disks build makes one. You give
it exactly one source.
| Source | What it takes |
|---|---|
--binary path |
a static x86_64 executable, wrapped in a scratch image |
--from ref |
a public image, by name, with the files you add to it |
--image ref |
a Docker image of your own, as a save tar or a name in your local Docker |
--compose dir |
a Compose project directory and the images its services name |
Everything that runs in a machine runs under Docker Compose, inside the machine. A
Compose project is one container for each of its services. An image or a binary
is a project of one service, and --entrypoint and the arguments after -- are
that service’s entrypoint and command. A binary is no exception: --binary
wraps the executable in a scratch image, one layer holding that file and
nothing else, with no base image, no loader and no libraries, and the machine
runs that image as its one service. machine writes this image itself, so
--binary needs no Docker on your computer.
Docker and Compose
An image of your own needs Docker. A Compose project
also needs Docker Compose. Two sources need
neither. machine disks build --binary
wraps a static executable and nothing else, so a program you compile needs no
Docker. The executable has to be self-contained. The image has no loader and no
libraries, so a dynamically linked program cannot run. --from names a public
image, and the build pulls it for you.
1 | machine disks build --image myapp:latest app |
The name defaults to whatever the source is called. Use --entrypoint to
choose the executable the service runs. The service’s own arguments go after
a --. This one runs data-race with the
argument 3:
1 | machine disks build --binary ./data-race race -- 3 |
From Python, client.disks.build takes the same sources as keywords:
binary=, from_=, image= or compose=, with command= for the arguments
after --. The SDK’s Disks has the rest.
1 | disk = client.disks.build("race", binary="./data-race", command=["3"]) |
Programs that need a runtime
A program that is not a static executable needs an image that already contains
what it runs: an interpreter, a runtime, its libraries. Public images have
those, so name one with --from and put your files in it with --add. It
reads like a Dockerfile, FROM then ADD then the command, and you need no
Docker for it. The build pulls the image itself, so only your files are
uploaded.
An image you built yourself, or one from a private registry, goes in with
--image from your local Docker.
A complex application, several services together, is a Compose project.
Python program
Save this as hello.py.
1 | print("hello, world") |
1 | machine disks build --from python:3.12-slim --add hello.py hello-py -- python hello.py |
It runs in a machine with the default 256MB of memory, and takes about ten seconds from power-on to the program’s exit.
Node.js program
Save this as hello.js.
1 | console.log("hello, world") |
1 | machine disks build --from node:22 --add hello.js hello-node -- node hello.js |
ClickHouse queries
The ClickHouse® image has clickhouse local, which runs SQL with no server.
Save this as queries.sql.
1 | CREATE TABLE orders (id UInt32, customer String, amount UInt32) ENGINE = Memory; |
1 | machine disks build --from clickhouse/clickhouse-server:24.8 --add queries.sql \ |
The machine’s console shows the result, and the machine ends when the queries do.
1 | app-1 | ada 55 |
PostgreSQL queries
The PostgreSQL image starts a server, and it needs a password before it will.
--env sets a variable in the service’s environment. The image runs every
.sql file in /docker-entrypoint-initdb.d when the server first starts, so
add the queries there. Save them as queries.sql.
1 | CREATE TABLE orders (id int, customer text, amount numeric); |
1 | machine disks build --from postgres:17 --env POSTGRES_PASSWORD=secret \ |
1 | app-1 | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/queries.sql |
The server keeps running after the queries, so this machine does not end by
itself. To run queries and have the machine end, use the
PostgreSQL project. The disk holds each --env
value as written, and a value is not changed on the way: a $ in it arrives
as a $.
Both ran in a machine with 1024MB of memory. A server that other services connect to is a Compose project.
Where added files go
--add puts a file or a directory in the image, as one more layer on top of
the public one. You can repeat it.
| You write | It lands at |
|---|---|
--add hello.py |
the image’s working directory, as hello.py |
--add conf |
the working directory, as the directory conf |
--add hello.py:/app/ |
/app/hello.py |
--add hello.py:/app/main.py |
/app/main.py |
--add conf:/etc/app/ |
/etc/app/conf, the directory inside /etc/app |
--add conf:/etc/app |
/etc/app, the directory’s contents |
The rule is the one cp -r follows: a destination that ends in / is a
directory the path goes into, and any other destination is what the path
becomes. A Dockerfile ADD of a directory copies its contents; the last row is
how you write that here.
A script that was executable still is. --add works with --binary too, for
a program that reads a file beside it.
--add takes files, directories and links. A device, a FIFO or a socket in
the directory is refused. Every added file belongs to root in the image, so
the set-user-id and set-group-id bits are not kept: a bit that meant “run as
this file’s owner” on your computer would mean “run as root” there. An image
that needs one is built with Docker and named with --image.
The image is the one the name pointed at when you ran the build. The build
records its digest and pulls exactly that, so a tag that moves later does not
change your disk. You can name a digest yourself, as python@sha256:… or the
python:3.12@sha256:… that docker pull prints.
The disk’s size defaults to what the image needs plus a gigabyte. Use
--dry-run to see the numbers, and --size for more room.
--from takes public images of up to 5GB as they download. For a private
image, a larger one, or one you build yourself, use --image:
1 | docker build --platform linux/amd64 -t myapp:latest . |
An image that needs more than one service, such as a database server with a client, is a Compose project.
Compose projects
The examples have Compose
environments under compose/. Clone them, pull each image for linux/amd64
into the local Docker, then build the disk from the project directory.
machine disks build --compose packs
those images and the project. The machine runs the project and ends it when a
service exits on its own, stopping the others, or when the project is taken
down with docker compose down. A service that Machine stops, kills or crashes
with machine containers ends nothing.
1 | git clone https://github.com/testflows/Machine.git |
The same project can be run on this computer first, from its own directory,
which is how to fix it without waiting for a boot. The build commands below
are from the examples directory.
A two-service project
compose/two-services is a busybox server and a client that fetches a file
from it by the service name server. Services in a project reach each other
by name, on the network Compose gives the project.
depends_on only orders the start. The client retries, because nothing waits
for the server to accept connections. A service that takes time to become
ready should use a healthcheck, and depends_on with
condition: service_healthy.
1 | docker pull --platform linux/amd64 busybox:1.36 |
To run it on this computer first:
1 | cd compose/two-services |
ClickHouse project
compose/clickhouse is a ClickHouse server and a client that queries it.
Both services use one image, so the disk carries that image once. The client
waits until the server is healthy, prints its result, and exits, which ends
the project.
The server takes time to start before it accepts a query, so the project uses a healthcheck rather than a sleep. config.xml in the
project replaces the image’s config. It listens on IPv4 only, because the
machine’s kernel has no IPv6, and it caps the background pools, because the
image’s defaults hang startup in a small machine. CLICKHOUSE_SKIP_USER_SETUP
stops the entrypoint from restricting the default user to loopback, which
would refuse the client.
The config is mounted from ./config.xml, a path inside the project. A bind
mount from outside the project directory is refused. A named volume or a
tmpfs is created empty when the machine boots.
1 | docker pull --platform linux/amd64 clickhouse/clickhouse-server:24.8.14.39-alpine |
PostgreSQL project
compose/postgres is a PostgreSQL server and a client that runs
queries.sql against it. Both services use one image. The client prints the
results and exits, which ends the project, so this is how to run queries
against PostgreSQL and have the machine end.
The client waits until the server is healthy. The healthcheck asks over TCP,
because the image first starts a temporary server on its Unix socket alone to
set the database up, and a check on the socket would pass before the real
server is listening. ON_ERROR_STOP makes a failed query the client’s exit
code. queries.sql is mounted from the project directory.
1 | docker pull --platform linux/amd64 postgres:17 |
1 | client-1 | CREATE TABLE |
Carrying images
A service can start containers of its own, through the machine’s Docker. The images it starts are named by no service this project runs, and a machine reaches no registry, so an image that is not named here is not on the disk.
scale: 0 names an image and starts nothing. The build packs every image the
project names, including these, and Compose starts none of them.
compose/clickhouse-regression carries the images its suite starts this way.
Another image, such as ZooKeeper or MinIO, is another service.
1 | services: |
The socket is /var/run/docker.sock. /run/docker.sock is not the machine’s
socket, and the build refuses it.
./ is mounted at /opt/compose, the path the machine runs the project from.
Compose resolves a bind on the machine’s filesystem, not inside the container
that asked for it, so the two paths have to be the same. The images still have
to be in the local Docker, under those names, before the build.
Project rules
Every service names an image with a tag or a digest. An image with no tag
is a different image tomorrow, and a service that only has build has nothing
to pack. Build the image, then set image.
The images have to be linux/amd64 and already in the local Docker. A
logging driver, if the project sets one, is json-file, local, or none.
Volumes a crash cuts
machine containers crash cuts a service’s
data the way a power cut would. It cuts the volumes the project names in an
x-machine block, each with the most it may hold:
1 | services: |
Each becomes an ext4 filesystem of its own on the disk, which costs only what
is written to it. The disk is made big enough for every volume to fill, and
--dry-run shows the room they take:
1 | size |
The capacity is what the disk holds, plus the volumes full, plus a gigabyte, rounded up to a whole GB.
A volume in x-machine is a plain Compose volume: no driver_opts, not
external. A project names 8 at most. A volume two services mount can’t be
crashed while both run. Such a volume has a journal, which is what recovers it
after a crash, and an fsync on it takes about four times as long as on a
volume the block doesn’t name. Name only the volumes whose crashes you want to
test.
Building disks on an ARM machine
A machine is an x86_64 computer, so a disk holds linux/amd64 programs, whatever
machine you build it on. On a Mac with Apple Silicon or on arm64 Linux, Docker
builds and pulls arm64 images by default, and those cannot run in a machine.
machine disks build checks every image and refuses one that is not
linux/amd64 before uploading anything. Ask Docker for linux/amd64:
1 | docker pull --platform linux/amd64 myapp:latest |
The last one builds a Compose project’s images for linux/amd64; setting
platform: linux/amd64 on each service does the same. Docker builds them under
emulation, which is slower than a native build but works. For --binary,
compile for x86_64 Linux, for example GOOS=linux GOARCH=amd64 go build.
Images at boot
A disk boots with its images already loaded, so the project starts as soon as Docker does.
A disk built with --load-at-boot keeps the images as docker save tars
instead, and Docker loads them each time the disk boots.
1 | machine disks build --load-at-boot --image myapp:latest app |
That boot is slower by the whole load, every time, and the disk holds each
image twice: the tar, and what it unpacks to. Use it when a service runs
docker save or docker push on one of the disk’s own images from inside the
machine. Starting containers does not need it.
Layers compressed with zstd are not read; save the image with gzip or uncompressed layers.
Disk size
By default Machine measures what the disk holds and adds a gigabyte, rounded
up to a whole GB. Use --size to set the size you want. Not sure how big it
will be? --dry-run reports what the disk would hold without building it.
1 | machine disks build --dry-run --from python:3.12-slim --add hello.py hello -- python hello.py |
1 | disk |
rootfs is the platform every disk carries. images is the images unpacked,
with a layer two images share counted once. used is what a smaller --size
is refused against, and the rest of capacity is what the machine can write.
With --load-at-boot there is one more row, images (tar).
To manage your disks:
1 | machine disks list |
machine disks list and machine disks show give two sizes. SIZE is the disk
as the machine sees it. DATA is what the disk takes in your account’s storage,
after deduplication and compression.
1 | NAME ID SIZE DATA STATE CREATED LABELS |
Disks share what they have in common: every disk carries the same platform,
and two disks built from one image share its layers. Storage keeps one copy,
and each disk’s DATA counts it, so the column adds up to more than your disks
take together. machine storage has that figure, and it is the one your quota
counts. A disk that shows — for DATA gets a figure from
machine disks verify.
If you ever doubt the copy of a disk in your account’s storage, check it:
1 | machine disks verify app |
1 | ✔ Verified app (4d13d6d911ee) |
machine disks verify reads the disk back and computes its ID again. It
takes about as long as reading the disk, and a disk that does not read as it
was stored is reported as damaged, in which case build it again.
Deleting a disk is permanent. Machine refuses to delete one that a run still
boots. --force deletes it anyway. Those runs stay, and can no longer start.
machine disks transfers shows the transfers in progress, and
machine disks cancel stops one.
Sessions
Runs live in a session, and you create one with machine sessions create.
Pick its size when you create it.
1 | machine sessions create ci-1 --cpus 2 --mem 4G |
--cpus sets the number of vCPUs, --mem the memory and --storage the
disk space. Your plan caps each session: 2 vCPUs, 4G of memory and 32G of
disk on Free, 4 vCPUs, 8G and 128G on Starter and on Free with usage
credits, 6 vCPUs, 12G and 192G on Plus, and 8 vCPUs, 16G and 256G on Pro. A hardware class can give a
session less than the plan allows, and the smaller of the two applies. Your plan
also limits the vCPUs your sessions hold together: 4 on Free, 8 on Starter and on
Free with usage credits, 12 on Plus and 24 on Pro. A session counts the vCPUs it
is charged for: 2 vCPUs with 8G of memory count 4. machine account show shows
how many are in use. --class picks the hardware class, and the default comes from your
plan. machine sessions classes lists
the classes, with the region and the CPU architecture of each. A session runs
in its class’s region, and its runs are stored there. standard-ip, on intel-1, is
the class to prefer. standard-ie, on intel-2, costs less. A run starts only
in a session of the architecture it was recorded on.
1 | machine sessions classes |
The name is optional. Leave it off and Machine makes one up. Machine waits
until the session is running and tells you when it is. Add --no-wait to
return once the request is accepted.
Then Machine uses the new session in this terminal, so your next command goes
to it. Add --no-use to keep using the session you are on. A session created
with --no-wait is not running yet, so it is not used.
Your first session also sets up your account’s storage, so it takes longer to
create, and Machine shows this as a step of its own. To set the storage up
before that, run machine account provision.
A machine takes its memory plus 128MB of the session’s memory, and 256MB of a
session is not available to machines. A machine with the default 256MB needs
384MB, and one created
with --mem 2048 needs 2176MB, so a 4G session holds one and an 8G session
holds three. A machine holds its memory until you stop it, whether or not its
program has finished.
A branch you switch onto with machine go uses the machine that is
already running, so it needs no more memory. A branch you start with
machine fork is another machine and needs its own.
machine runs --size adds two columns for what a run takes of the
session’s resources. RAM is the session memory its machine holds right now. It
can be above the machine’s own MEMORY. DATA is what the run holds on the
session’s disk. A run
with no machine shows — for RAM.
The MEM column of machine sessions list shows the memory taken over the
session’s memory, for example 4.5G/8G with two machines of 2048MB. A new
machine starts when what it needs fits in the rest, and is refused when it
does not.
To switch to another session, tell Machine which one to use.
1 | machine sessions use ci-1 |
machine sessions use applies to the terminal you ran it in.
If you only have one session, it needs no name at all.
After that, every command that needs a machine goes to ci-1. You can
send one command somewhere else with -s, for example
machine -s ci-2 runs. -s goes before the command.
In a script, name the session once with TESTFLOWS_MACHINE_SESSION, and
every command after it goes to that session.
1 | export TESTFLOWS_MACHINE_SESSION=ci-1 |
The variable names the session. machine sessions use connects this
computer to it, and is needed once. A tool that starts a new shell for each
command, as an AI agent does, loses the export between commands. There, put
-s ci-1 on each command. Commands that need no session, such as
machine disks list, ignore it.
You manage the rest of your sessions like this:
1 | machine sessions list |
✋ A session costs from the moment you create it, whether or not anything is running in it. Delete sessions you are not using.
Idle sessions stop
A session is idle while it serves no command. Your plan sets its idle time:
10 minutes on Free, 30 on Starter and on Free with usage credits, 45 on Plus,
and 60 on Pro.
machine account show shows yours. When a session reaches its idle time, it
stops, and so do its runs. The runs are kept. Start one again in any session
with machine start <run>.
Every command sent to the session resets the idle time, from any terminal, the SDK or the control prompt. To step away and keep the session, keep it:
1 | machine sessions keep ci-1 30m |
A keep holds the session idle for up to 30 minutes on Free, and an hour on
Starter, Free with usage credits, Plus and Pro. machine sessions list shows under
IDLE how long each session has gone without a command, and
machine sessions show when it stops. machine sessions list --all also lists
the sessions that stopped. One that stopped idle is listed as stopped idle,
with the idle time it reached under IDLE.
In the control prompt, /keep 30m keeps the session you are driving, and the
line under the clock shows the time left.
From Python, client.sessions does the same: create, use, show, keep
and delete. A session is a context manager, so a with block deletes it at
the end. The SDK’s Sessions has the calls.
When the service does not answer
A command that cannot reach the service, or that the service refuses as unavailable, ends with a hint to the status page, status.testflows.com. The page shows whether the website, the API, sign-in and sessions on each class work now. It also lists open incidents, the incidents of the last 14 days and any planned maintenance. If every component is operational, check your network. If it works, contact support.
Running a machine
Creating a run
machine create creates a run from a disk and starts it. The run’s name goes
first and is optional. Without one, Machine generates a name.
1 | machine create app --disk app --cpus 2 --mem 512 |
The defaults are 1 CPU and 256 MB of memory. Other options of create:
| Option | What it does |
|---|---|
--no-daemon |
boot and run to the end by itself, instead of waiting for your commands |
--pin-cpu n |
run on host CPU n, for this start only (the default picks the least-loaded) |
--like run |
start from another run’s options, then apply the ones you give |
--label name=value |
attach a label to the run, as many times as you like |
--rng, --rng-seed n |
an RNG device, on by default, seeded with n; the default seed is 0, and --no-rng leaves the device out |
--fs-sync |
write to the disk as the program writes, instead of later |
--mem-hash |
hash the machine’s memory at every entry, so a replay compares it too (off by default) |
--disk-hash |
hash the machine’s disk at every entry, so a replay compares it too (off by default) |
--timer-slack d |
how late a timer may be in code that takes no exit, 20ms by default; 0 keeps every timer on time |
--limit-margin n |
how many conditional branches short of an exact stop the CPU’s counter interrupts; the default is the host’s |
--repeatable |
record so that recording the run again gives the same entries; the default is the session’s, which machine describe shows, and recording is slower |
--vtime-base n |
the date the machine’s clock starts at, as a Unix timestamp; the default is 2026-03-01, see Machine time |
--no-wait |
return as soon as the request is accepted |
Random numbers
A machine has an RNG device, on unless the run was created with --no-rng.
The device gives the guest its random bytes from a seed. The default seed is
0, so a program sees the same random numbers every time. --rng-seed chooses
another, from 0 to 2^64-1.
1 | machine create app --disk app --rng-seed 42 |
machine rng prints the seed, and whether the guest
reseeds its own generator at once.
To seed the device again, where the run is now:
1 | machine rng app seed 7 |
The run records that seed, so a replay seeds the device at the same point. A
guest kernel that reads the seed reseeds at once. A kernel that does not
read the seed picks it up the next time that kernel reseeds. A run created with --no-rng has no
device, and the command refuses it.
A branch can draw other random numbers than its parent. machine fork,
machine detach and machine go
take --rng-seed, and the new run’s device is seeded from it once the run is
up:
1 | machine fork app --at ^booted --name other --rng-seed 7 |
At the control prompt the same commands are /rng and
/rng seed 7, and a branch takes rng-seed 7.
The seed on machine run --seed is a different
one. It draws how long each stretch of execution lasts.
Schedules covers it.
From Python, client.create("app", disk="app", rng_seed=42) seeds a new run,
run.rng() reads the seed, run.rng.seed(7) seeds the device again, and
run.fork(at="^booted", rng_seed=7) makes a branch with its own.
Multiple vCPUs
A machine can have more than one vCPU. Ask for them with --cpus and the guest
sees every one, so its kernel spreads threads across them.
Build a disk from Ubuntu 26.04 whose container runs sleep infinity, so it
stays up and waits for your commands. Then
create a run with four vCPUs and ask the guest what it has:
1 | machine disks build --from ubuntu:26.04 ubuntu-26.04 --entrypoint /bin/sleep -- infinity |
1 | 4 |
Nothing about recording changes with more vCPUs. The machine records each of them, along with every interrupt one vCPU sends to another, so a run on four vCPUs replays exactly like a run on one. A race between two threads on different vCPUs plays out the same way every time you replay it.
Each scheduling cycle gives every vCPU a turn. To drive just one of them, use
machine run --vcpu N. And to explore how threads on several vCPUs could
interleave, see Schedules.
Driving it
A run executes only while you drive it. machine run is the way, and --until
says when to stop. (A run created with --no-daemon answers
✗ Machine not in control mode, because the machine isn’t waiting for you.)
1 | machine run app --until tasks |
A plain machine run can stop before the point you are waiting
for. With --until, Machine keeps going until what you asked for is true. If it never gets
there, the command exits with status 3.
These are the conditions you can wait for. machine run --until and
machine wait --for take the same ones.
| Condition | True when |
|---|---|
running, paused, stopped, halted |
the machine is in that state |
ready |
the machine accepts commands |
tasks |
Linux is up and its task table can be read, which is before it starts anything in userland |
console~REGEX |
the console output matches the pattern |
ttyS1~REGEX |
the output of another source matches it, here the serial port ttyS1 |
entry>=N |
the log has reached entry N |
entry=+N |
N more entries have been recorded |
vtime>=T |
the machine’s clock has reached T, a duration such as 500ms |
vtime=+T |
the machine’s clock has moved T from where it was |
/app:exit |
the program at /app has exited |
The last row is one example of a task condition. A condition on a program needs
the program focused first, with machine focus app add /app, and the command
refuses it otherwise. There are more task conditions under
Steering programs.
The two commands read a text condition differently. run --until waits for
text printed during the run, so a line already on the console does not end
it, and the same --until can stop at the next of several lines alike.
wait --for asks whether the text has been printed at all, before the run
or during it:
1 | machine run app --until 'console~ready' # runs until the next ready line |
A source name with a ~ in it is quoted inside the pattern, as in
'"/output/keep/app/app.log~"~ERROR'.
--iters limits how many iterations a drive may spend, and the global
--timeout limits how long you are willing to wait. Whichever hits first wins.
An iteration has no fixed length. It is one dispatch of a vCPU, and it ends
where the guest next exits to the machine: after five instructions or after a
million. So no count is right in advance, and nobody can know how long a
program will run. Stop on a condition, and use --iters or a vtime
condition as a limit. Every run ends by saying how far it went:
1 | iterations: 2000 run ic: 238893097 vtime: +0.383620410s |
vtime is how far the machine’s clock moved. A vtime condition is checked
between iterations, so the run stops at the end of the iteration that reaches
the time, a little late. A run that went too far is not lost:
machine rewind and machine go take it back to an earlier
point.
A time bound
--within ends a run when the machine’s clock has moved that far:
1 | machine run app --within 10s # run for 10 s of machine time |
Alone, it is the same run as --until vtime=+10s. With another --until it is
a bound on it: if the clock gets there first, the run ends with exit status 3
and says Machine time ran out. The machine checks the bound itself, so a slow
host does not end the run sooner, which the global --timeout in seconds would.
The clock moves as the programs in the machine work and while the machine is idle.
Now and then the host interrupts an entry while it is being recorded. Machine
records that entry again, and you see nothing except a short pause. --retry
sets how many more times one entry is tried, 2 by default. If they are all
interrupted the run stops with Host interrupted the run at entry N. It is not recorded. and the machine rewind command that puts the run back at the
entry before.
Schedules
A program that uses threads can behave differently depending on how they
interleave. Machine can draw how long each stretch of execution lasts, so every
run explores a different interleaving, and --seed makes those draws
repeatable.
1 | machine focus app add /usr/bin/app |
--mode is how the machine runs: fast, step, step-into, free or
fence, or a comma list to draw from. --step and --fast give the length of
a stretch in each mode, as a number or a range like 1:200. --strategy sets
how the draws are made, random (the default), golden or sweep, and
--seed is the seed for the random and golden ones.
A fence stretch ends in the same place every time the same state is
recorded, where a fast stretch ends close to its length but not always at the
same instruction. That exactness makes a fence stretch somewhat slower than a
fast one. Give its length with --fence or --limit.
Recording that repeats shows how to make a whole run
record the same way every time.
What a task does
Drawing the length of a stretch varies where threads are interrupted. To vary what the guest’s scheduler does with them, draw a mark for the task on the vCPU at each dispatch:
1 | machine run app --task-sched keep,hold,yield --task-slice 10us:600us --seed 3 --iters 500 |
--task-sched draws among keep, hold, yield, drain and drain-all:
hold takes the task off the run queue, yield puts it behind its peers,
drain undoes the oldest of those marks, drain-all undoes every one, and
keep changes nothing. --task-slice draws how long the task runs before the
scheduler decides again, from a span such as 10us:600us, and default puts
the kernel’s own back. --task-int draws whether to preempt it. Each takes a
comma list to draw from, and --seed makes the draws repeatable.
--when applies the draws only while a predicate holds, such as
/usr/bin/app:on for the program running its own code. A predicate on a
program needs the program focused first, as the second example does with
machine focus, and the run refuses it otherwise. To draw different
things at different moments, write a plan, where
each clause has its own when.
Combine that with branches. Fork several from one checkpoint, give each its
own seed, and keep the ones that crash. The explore_seeds.py example in the
SDK does exactly that.
To hold an interrupt a drawn time before it is delivered, see Holding interrupts.
A run takes checkpoints for you too. --auto-checkpoints N takes one every N
entries, and 0 turns them off.
Cancelling a drive
If a drive or a checkpoint is taking too long, machine cancel app cancels
whichever one is in progress. A run stops at once, where the host’s clock put
the stop, so a repeatable run no longer repeats from that entry. A replay stops
before its next entry.
--soft stops the run at the end of the dispatch in progress instead. That
stop is an ordinary entry, so the run stays repeatable. It takes at most the
run’s dispatch bound, 1s by default.
1 | machine cancel app # stop now |
Ctrl-C on a drive cancels softly, and so does the global --timeout on run
and exec run. Ctrl-C prints Stopping at the end of the current dispatch. Press Ctrl-C again to stop now, and a second Ctrl-C stops the run at once. A
run still busy 10 seconds after a Ctrl-C or a timeout is stopped at once as
well. At the
control prompt, /cancel stops the run at once and
/cancel soft at the end of its dispatch.
Recording that repeats
A replay always repeats a recording exactly. Recording the same run a second time is different: by default two recordings of one run, from the same disk and the same seeds, start out the same and then drift apart a little.
When you need the second recording to match the first, create the run with
--repeatable. Every recording of it then records the same entries, whether the
run has one vCPU or several, and a branch you fork from one of its checkpoints
records exactly what its parent recorded after that checkpoint.
1 | machine create app --disk app --repeatable --cpus 2 |
Repeatable recording is slower, and how much slower depends on the program.
For example, booting PostgreSQL 17 on one vCPU takes 1 minute 10 seconds to
record with --repeatable and 14 seconds without it. Loading its test data
with pgbench -i, which computes with hardly a pause, takes 1 minute 20
seconds against 4 seconds. A program that spends its time waiting on the disk, the
network or a timer pays much less. Replaying a repeatable run is as fast as
replaying any other.
--timer-slack is how late a timer may be while a program runs code that
takes no exit. The default is 20ms, and it applies to every fast run. A timer
that comes due in code that exits to the machine is delivered at that exit. 0 keeps
every timer on time, and recording is slowest that way. The slack takes ns,
us, ms or s, as in --timer-slack 1s. A drive can draw the
slack for that drive:
1 | machine run app --timer-slack 0:20ms |
--limit-margin is how many conditional branches short of an exact stop the
CPU’s counter interrupts. The default is the host’s, and
machine describe shows that default as
host's. A smaller margin records faster. One too small misses stops, and a
later recording may not repeat.
machine replay --limit-margin sets it for that
replay.
Four more options decide where a recording stops. Each takes 0 for its
default, and machine describe shows them as
Instr Margin, Dispatch Bound, Fast Backstop and IRQ Wait:
| Option | What it does |
|---|---|
--instr-margin n |
how many instructions short of a stop placed by an instruction count the CPU’s counter interrupts; the default is the host’s, 50,000 on amd-1, the standard class, and 3,630 on Intel, which describe shows as host's |
--dispatch-bound d |
how much host time a fast dispatch runs before it stops, 1s by default; it takes a duration such as 500ms or 2s |
--fast-backstop n |
how many instructions a fast dispatch runs with no other stop, 100,000,000 by default |
--irq-wait n |
how many conditional branches a dispatch runs while an interrupt waits for the guest to take it, 200 by default |
A stop placed by --dispatch-bound falls where host time put it, so another
recording of the same state does not stop there. A margin the run gives must
be smaller than its backstop, and the run is refused otherwise.
1 | machine create app --disk app --repeatable --instr-margin 100000 --dispatch-bound 2s |
A branch keeps its parent’s --repeatable, --timer-slack, --limit-margin,
--instr-margin, --dispatch-bound, --fast-backstop and --irq-wait
unless you give it others. machine fork,
machine detach and machine go
take all seven. machine start keeps what the run
was created with.
Where a run stops repeating
Some stops are placed by host time, and no other recording of the same state
stops at the same instruction: a stop at the dispatch bound, an immediate
machine cancel and a signal the host sends the
machine. A
repeatable run repeats up to the first such entry. For a repeatable run,
machine describe shows that entry as
Repeats To, and tip when the whole run repeats:
1 | Repeatable: true |
machine checkpoints has a REPEATS TO
column for a repeatable run: for each of its own checkpoints, the first entry
after it where recording again from that checkpoint stops repeating:
1 | NAME ID VTIME DURABLE REPEATS TO |
A
machine cancel --soft and a Ctrl-C end the run at the end of the dispatch in
progress, so they keep the run repeatable. See
Cancelling a drive.
What you send a run is part of what a recording repeats. Commands given with
machine exec, put and get in the same order
at the same points name their processes alike in every recording, such as
40787p1: the entry the run stood at, and the command’s place among those
sent there.
Time compression
Idle time takes almost no time.
When every vCPU is halted and waiting for a timer, the machine doesn’t sit through the wait. It moves its clock straight to the earliest timer the guest armed, and the guest wakes up at exactly the time it asked for, as if it had waited. No instruction runs in between, so there is nothing for the host to do. Time the guest spends working is never skipped.
So a program that sleeps for ten minutes sleeps ten minutes of machine time, and only a few seconds of yours. Try it on a one-vCPU run of the Ubuntu disk from Multiple vCPUs:
1 | machine create app --disk ubuntu-26.04 |
1 | Sun Mar 1 00:00:05 UTC 2026 |
Ten minutes went by in the guest, and 2.8 seconds went by for you.
Every vCPU idle
Time compression works the same way with several vCPUs. The machine only moves the clock once all of them are idle, so a vCPU that still has work finishes it first. Here is the same sleep on the four-vCPU run:
1 | time machine exec app4 run --container app -- sh -c 'date; sleep 600; date' |
1 | Sun Mar 1 00:00:07 UTC 2026 |
On replay
Every move of the clock is recorded in the run’s log, so a replay makes the same moves at the same points and ends up in the same state. Fork a branch from the start of the run and replay it, ten minutes of sleep included:
1 | machine stop app4 |
Then machine diff compares the branch with the
original:
1 | machine diff app4/check |
1 | ✔ No differences |
How much a run compresses depends on how idle the guest really is. A database that keeps running background tasks while your test sleeps uses machine time for them, and that time is never skipped.
Turning time compression off
--no-idle-skip keeps the clock moving at its own rate while every vCPU is
idle, instead of jumping to the next timer. Use it to see what a program does
while it waits in real time, at the cost of the speed:
1 | machine run app --no-idle-skip --within 10s |
Waiting
machine wait doesn’t drive anything. It watches a run, or several, and
returns when a condition holds.
1 | machine wait --for stopped app |
1 | ✔ Reached stopped: app (whUxfLYSFJMDHzKaiBOvH) 2.1s |
It exits with 0 when the condition is true, 2 on an error and 3 on a timeout,
so scripts can tell them apart. Remember that it only watches. Waiting for a
condition on a run nobody is driving never finishes unless you set --timeout.
With --timeout 0 it checks once, and the message tells you why it failed: The machine was waiting for commands.
At the control prompt the two are /run until and
/wait for, and /run takes within too:
1 | /run until 'console~ready' within 30s |
/wait for answers at once, because nothing drives the machine while the
prompt waits. timeout 5 makes it wait 5 seconds, for a run another terminal
drives.
From Python, run.run(until="console~ready", within="30s") drives until a
condition, and run.wait(for_="stopped", timeout=60) waits for one without
driving. Bounding a drive shows both.
Inspecting a run
| Command | Shows |
|---|---|
machine runs |
your runs, newest first (-w keeps the list live) |
machine describe app |
the full configuration of a run |
machine state app |
its state and where it stands; a run with no machine reads stopped |
machine now app |
the last entry it recorded, and the time on the machine’s clock |
machine output app |
what the machine printed (-f follows, -n -20 shows the last 20 lines, --source reads another source than the console) |
machine entries app |
the entries in its log |
machine ops |
operations on runs and how each ended, newest first: checkpoint, branch, switch, rewind, start and delete; one you interrupted with Ctrl-C is still listed |
machine debug log app |
the debug output of a run (machine debug err shows its error and crash log) |
machine dump app |
the machine’s registers, code and other state (--show picks the sections, --entry reads an earlier one); its hash section is what the newest entry recorded |
machine hash app |
the hashes of the machine as it stands, to compare with another stopped at the same point; see Comparing two machines |
machine vcpu app |
where one vCPU stopped, and how its last dispatch ended (--vcpu picks another) |
machine irqs app |
the interrupts the machine owes or holds; see Holding interrupts |
machine output also takes --entries, if you only want what a few
entries printed.
The run’s log is a list of entries, one for each time the machine stopped,
with why it stopped and where. machine entries
reads it, and -f follows new entries as the run records them:
1 | machine entries app -n -20 # the last 20 entries |
With many runs, machine runs --roots shows one row for each tree, with how
many of its runs are running, and --not-synced shows the runs nothing has
been published from yet.
Finding a line
Often you want the opposite: you know what the machine printed, and you want the entry that printed it, so you can go there. A line is printed over a range of entries, but a byte is printed by exactly one, and Machine can tell you which.
machine output app search finds a pattern and gives the entry of
each match:
1 | machine output app search 'Linux version' |
Each match is a row with the columns ENTRY, TIME, SOURCE, LINE:COL and
TEXT. ENTRY is the entry that printed the first byte of the match, and
TIME is that entry’s time, a point such as 1.194000000s+2. TEXT is the
whole line, with the match in bold.
A match has to cover at least one character, because it is the entry that
printed it that you get back. A pattern that can match nothing at all, such as
6*, is refused, in a search and in a console~REGEX condition or point alike;
6+ is the one to write.
The search runs where the console is kept, so a large console is not downloaded.
-f keeps going with what the machine prints next, and --max 1 stops at the
first match. The entry is a point, so you can go straight to it:
1 | machine go app '#1276' |
console~REGEX is the same thing written as a point, where the console first
printed a match. machine go, machine fork --at and
machine rewind all take it:
1 | machine go app 'console~Linux version' |
console:N is a line by its number, as machine output -n numbers it: the
point where the console finished printing line N. Any source takes it, such as
ttyS1:20. A line counts from 1, so the point names the same place each time
it is read:
1 | machine go app console:120 |
If you would rather search with your own tools, pipe the console to them.
grep -b prints the byte offset of a match, and
machine output app entry turns an offset into its entry:
1 | machine output app -n : | grep -bo 'Linux version' # 438:Linux version |
On a terminal, every line shows the range of entries that printed it, the
source it came from and its number. Into a pipe or a file the lines are printed
alone, as the machine printed them. --wide and --plain choose either way:
1 | machine output app -n -5 # with their entries |
On a terminal, a control character the guest printed shows as text, such as
␛[?25l, and your terminal does not act on it. Colors stay colors. A pipe or a
file gets every byte as the guest printed it.
From Python, run.output.find returns the first match, with the entry that
printed it and a place that go takes, and run.output.search yields the
matches one at a time:
1 | hit = run.output.find("Linux version", from_="^booted") |
Sources
The console is not the only place a machine prints. It has four serial ports,
ttyS0 to ttyS3, and the console is the first. A program in the guest that
writes to /dev/ttyS1 prints on ttyS1, apart from the boot log and from
everything else on the console:
1 | # inside the guest |
Everything a run prints is kept in one log, in the order it was printed, and
each part of it has a source. machine output app sources
lists the sources that printed:
1 | machine output app sources |
1 | SOURCE BYTES LINES ENTRIES |
--source reads one of them. Without it you get the console, as before:
1 | machine output app --source ttyS1 |
--source all reads every source together, in the order printed, with the
source in front of each line:
1 | machine output app --source all -n -4 |
1 | ttyS0 starting the app |
A line belongs to one source. If one source prints while another has a line unfinished, that line ends there, so what you read is always what one source printed, in order.
The name of a source also takes the place of console in a condition and in a
point. A run that waits for text on ttyS1 stops at the entry that printed it,
and what the other sources print in the meantime does not interrupt it:
1 | machine run app --until 'ttyS1~ready' |
| Source | What it is |
|---|---|
console |
the console of the run, which is the default |
ttyS0 to ttyS3 |
the four serial ports; ttyS0 is the console |
hvc0 |
the virtio console |
debugcon |
the firmware’s debug port |
a path under /output |
an output file, or every file under a directory |
all |
every source, in the order printed |
Output files
A service on a disk has no serial port to write to, and three ports are not many. Every service is given a directory instead, and each file it writes there is a source of its own. The directory is named by a variable:
1 | # inside the service |
The file is stored with the run as it is written, each write with its entry.
Its name as a source is its path: /output/keep, the name of the service,
and the path the service wrote. A disk built from a binary or one image has
one service, app.
1 | machine output app sources |
1 | SOURCE BYTES LINES ENTRIES |
Everything that takes a source takes the path, and the path of a directory is every file under it:
1 | machine output app --source /output/keep/app/events.jsonl |
A file does not have to be text. machine output app bytes
writes what a source printed exactly as it was written:
1 | machine output app bytes --source /output/keep/app/core.bin > core.bin |
Dropped output
Debug logs can be gigabytes, and a run that stores them is slow to copy and to read. A service writes those under a second directory, and what is written there is not stored:
1 | # inside the service |
The file is a source under /output/drop. It still has its name, and
machine output app sources lists it with 0 bytes, so you can see what a run
dropped. A run can still wait for a line of it. The wait stops the run at the
entry that wrote the line and stores nothing:
1 | machine run app --until '/output/drop/app/debug.log~connection refused' |
Because a run replays exactly, the file is not lost. Fork a branch that keeps the file and starts from a point before the part you care about:
1 | machine fork app --from ^before --name look --keep-output /output/drop/app/debug.log |
--from replays the branch from that point to where the run is now, and the
branch stores the file on the way. Use --at to stop the branch at an earlier
point. machine go takes --from too, and moves the machine onto the branch.
machine detach takes it for a new root that holds the file and shares nothing
with the run it came from.
--keep-output and --drop-output say what a run stores besides what the
two directories say. They are options of create, fork, detach and go,
the commands that make a run, and a run keeps them for its whole life.
machine describe shows the two directories and the names the run gives.
- A name is a file or a directory under
/output, a port such asttyS1, or a pattern such as'/output/drop/*/*.log'. A pattern is in the syntax of Python’sglobmodule:*,?,[abc],[a-z]and[!abc]stay within one directory name, and**matches any number of directories, as in'/output/**/*.log'. A name that begins with a dot is matched like any other. - When a kept and a dropped name cover the same file, a name spelled out
decides over a pattern, and of two names spelled out the longer one
decides:
--keep-output /output/drop/app --drop-output /output/drop/app/huge.logkeeps the directory except that file. Otherwise the file is stored. - A branch has its parent’s two lists. An option on the fork replaces the
list of its own kind and leaves the other as it was. The name
none, given alone, is an empty list:--keep-output nonekeeps nothing beyond what the directories say.
What is stored never changes what the machine executes, so a branch that keeps more still reproduces the recording.
Sending input
A program that reads what a person types can be typed to.
machine send writes input to a device of the run:
1 | machine send app ls # type ls and Enter on the console |
It works like echo: the text is sent as written and a newline follows, which
is Enter. -n leaves the newline out, and -e reads escapes such as \t and
\x03.
send does not run the machine. It queues the input and returns, and the
device takes it the next time the run executes:
1 | machine send app 'echo hello' |
The run records the input where it arrived, so a replay reproduces it without being sent anything. To try other input, fork a branch and send to the branch.
At the control prompt the same command is /send. The
options come first and the text is the rest of the line, as you typed it,
quotes included:
1 | /send ls -la |
The prompt reads quotes as a shell does. A ; separates commands, except
inside quotes, so /send echo "a;b" sends echo "a;b". A quote that is not
closed refuses the whole line, so write /send "don't". Put the text in
quotes to keep spaces at its ends, as in /send " y ", and put -- before a
text that begins with a dash.
| Device | Is |
|---|---|
console |
the run’s console, which is the default |
ttyS0 to ttyS3 |
the serial ports |
hvc0 |
the virtio console, on a run created with --console virtio |
keyboard |
the PS/2 keyboard |
A serial port and the virtio console take bytes. The keyboard takes text and types it on a US keyboard, a key pressed and released for each character, with Shift or Ctrl held where the character needs one. Use it for a program that reads the screen’s terminal and not a serial line.
Commands in the guest
machine exec executes a command in the machine,
beside the programs already running there:
1 | machine exec app run -- ls -la / # execute it and print its output |
run runs the machine until the command exits, prints what it printed, and
exits with its status. A command gets 30 minutes of machine time to end, and
--within gives it another: machine exec app run --within 10s -- make check.
start sends the command and returns an id. The process starts the next time
the run executes, so follow it with machine run.
A command is a program and its arguments, written after --. No shell reads
it. Use -c to give a line to the shell of the disk instead, with its pipes
and redirections: machine exec app run -c 'ls / | wc -l'.
Processes
Each command is a process. The run records every line the process prints and how it ends, each at its entry, so a replay reproduces them and a branch has the processes of its parent.
machine exec app with no verb lists every command executed in the run, with
the entries it started and ended at:
1 | ID PID STATE ENTRIES COMMAND |
A process’s id is the entry it was sent at and its number. proc shows one
process:
1 | machine exec app proc 23090p1 # pid, state, exit status |
An event’s entry is a point. machine go app '#23224' forks a branch at the
entry that printed a line.
The output of a process is a source like any other, under /output/exec/<id>.
machine output reads it, and
machine run app --until '/output/exec/23090p1/stdout~done' stops the run at a
line the process prints. In the machine, its standard output and standard error
are the files /.machine/output/exec/23090p1/stdout and
/.machine/output/exec/23090p1/stderr, and its events are in
/.machine/output/exec/23090p1/status.
Inside a container
Use --container to run a command in a container of your Compose project. It
takes the container’s id, its name or the name of its service:
1 | machine exec app run --container db -- psql -c 'select 1' |
--net runs a program of the machine on a container’s network instead: the
program comes from the machine’s disk, so it needs nothing in the container’s
image, and it sees the container’s interfaces and addresses. A capture of db’s
packets with the machine’s tcpdump, in Network faults:
1 | machine exec app start --net db -- tcpdump --immediate-mode -U -i eth0 -w /.machine/output/keep/db.pcap |
machine containers lists the containers
with the columns SERVICE, CONTAINER, ID, STATE, IMAGE, ADDRESS
and LINK: each container’s address on each of its networks, and the
machine’s interface at the other end of its link, which a capture names in
Network faults. The service, the container name and the
id are what --container takes:
1 | machine containers app |
Each service’s output, every line it prints across every start of its container, is a source of its own. The console shows all of them together, each line after its service’s name:
1 | machine output app --source /output/keep/db/console |
The command has the container’s environment and working directory. --user
picks the user it runs as, by name or as uid:gid; without it the command runs
as the container’s own user. A service that runs several containers is refused,
and the message names them, so you pick one. machine put
and machine get take the same --container.
A command without --container runs in the machine, beside Docker, and can
call Docker itself. The project is named compose:
1 | machine exec app run -- docker ps --format '{{.Names}}' # the containers |
docker compose exec needs -T, since the command has no terminal. A shell
line after the service is run by the image’s own shell, with the image’s
tools.
/exec at the control prompt
At the control prompt the command is /exec, with the
same verbs and flags. It needs no --: the arguments are the words after the
verb, and a line for the shell is the rest of the line after -c, its quotes
kept for the shell.
1 | /exec run ls -la / |
A ; separates commands at the prompt, except inside quotes, so a line with
one is written in quotes, as in a terminal. A pair of quotes around the whole
-c line is taken off. A $ line, below, needs none of this.
The machine’s shell: $
$ puts you in the machine’s shell. Type a line after it as you would in a
terminal. The prompt runs the machine until the line finishes and prints what
it wrote. A line that exits other than 0 says so after its output.
Nothing in the line is the prompt’s. Everything after the $ goes to the shell
as you typed it, so pipes, redirections, ;, &&, variables and quotes all
work without escaping:
1 | $ uname -a |
A $ line is /exec run -c with that line, so the run records it like any
other command. /exec lists it, a checkpoint keeps what it changed, and a fork
or a replay repeats it. Look around between two steps of a run, change a file,
and rewind to a checkpoint to try something else.
A container’s shell: db$
Put a container before the $ to run the line in that container’s shell.
Name the container by its Compose service or by its container name, as
--container takes it. Put a user and @ before the container to run the
line as that user, by name or by uid. Without a user the line runs as the
container’s own user.
1 | db$ psql -c 'select 1' # in the db service |
db$ LINE is /exec run --container db -c with that line, and
root@db$ LINE adds --user root. Everything after the $ goes to the
container’s shell as you typed it, and the run records it as it records a $
line.
Commands over several lines
A line goes on in the next one while a quote is open, after a trailing
backslash, and through a here-document, as in a shell. The prompt shows > for
each line after the first, and runs the whole command once it is complete. A
pasted query of several lines is one command:
1 | clickhouse@app$ clickhouse-client -q ' |
The lines of a command are edited together: Backspace at the start of a line goes on into the line above, and the arrows move between the lines. Ctrl+C drops the command.
Input for a program you started
A program that reads commands from its standard input, such as a debugger, is driven through a named pipe. Make the pipe, keep it open, start the program with its input on the pipe, and send to the pipe:
1 | machine exec app run -c 'mkfifo /dev/gdb_pipe' |
A --device that begins with / is the path of a file in the machine, and
machine send puts the bytes at its end. A process
in the machine writes them, and send prints its id: machine exec app proc
with that id says whether they were written. One send to a path is one
request of at most 4096 bytes. A send too large for it is refused, and the
message gives the most bytes that fit.
machine send writes the machine’s own files. For a program in a container,
make the pipe in the container and write it from there:
1 | machine exec app run --container db -- mkfifo /tmp/p |
<id> is the id exec start printed for GDB.
GDB inside the machine walks through a whole session this way, on ClickHouse.
machine exec uses the serial port ttyS3 for its requests. A program in the
machine that reads ttyS3 takes them.
From Python, run.exec(["cat", "/etc/hostname"]) runs a program and returns a
Process with exit_code, stdout and stderr. A string is a shell line,
container="db" and user="root" pick where it runs, and wait=False
returns at once with the process’s id:
1 | proc = run.exec("ls /data", container="app") |
Copying files
machine put copies a file of your machine into the
running machine, or into one of its containers:
1 | machine put app ./app.conf /etc/app.conf # into the machine |
- A file of any size is put.
- The directories on the way to the path are made, and a file already there is replaced.
- The file has its name only once all of it is written, so a program never reads part of it.
- The machine runs while the file goes in, and the other programs in it run too.
- The bytes are a recorded input, so a replay puts the same file and a branch has it.
- The file is on the disk when
putreturns. A checkpoint taken after it has the file, andmachine fsreads it there.
--container takes a container’s id or name, as
machine containers app lists them, or the
name of its Compose service: --container db is the container of the service
db. A service that runs several containers is refused, and the message names
them, so you pick one. The container must be running. A path is resolved inside
the container, as a program there would resolve it.
machine send --device <path> is for a line into a
pipe, one request of at most 4096 bytes. put is for a file.
machine get copies a file the other way, from the
machine or from a container to your machine:
1 | machine get app /var/log/app.log ./app.log |
get reads the file as it is now, with what programs wrote and the disk does
not have yet. machine fs reads the disk at a
checkpoint instead. The machine runs while the file comes out, so a file a
program is writing can change on the way.
At the control prompt the commands are /put and /get,
with the same operands and options: /put ./app.conf /etc/app.conf.
A machine whose disk was built before put used this path still takes a file,
far slower. Rebuild the disk to get the speed.
From Python, run.put("./app.conf", "/etc/app.conf") puts a file in, and
source can be bytes. run.get("/var/log/app.log", "./app.log") gets one out,
and without a destination the Got it returns carries the bytes in data.
Both take container=.
Core dumps
machine core tells a process to generate a core
dump and copies it to your computer, where a debugger reads it with the program:
1 | machine core app 412 ./app.core # pid 412 of the machine |
- Taking a core ends the process: it dies of a signal,
ABRTunless you name another with--signal. - To keep your run as it is, fork first and take the core on the branch.
- With
--container, the pid is the one the process has inside the container, asmachine exec app run --container db -- psshows it. - A process that handles the signal dumps no core. Name another, such as
--signal SEGV. - The first process of a container dumps no core. Run your program under an
init,
init: truein the Compose file, so it is not the first.
From Python, run.core(412, "./app.core") gets the core of process 412, and
container="db" names the process by its pid in that container.
Profiling
machine profile shows where a run spent its
time, by function and by call stack:
1 | machine profile app # a second of everything |
Here is one second of memstress,
one of the example programs, which maps, touches and unmaps memory from four
threads:
1 | 463 samples at 250 a second |
SELF is the share of samples a function was running in, and TOTAL the share
it was on the stack of. So worker itself ran in 19% of the samples and was on
the stack of 94% of them: the rest of its time went to the calls under it.
A name that ends in _[k] is time in the kernel. sys_munmap_[k] is the
munmap system call. kernel_[k] is kernel time that is no system call, here
the page faults of touching fresh memory.
--flamedraws a flame graph in the terminal.--foldedprints every stack with its count, one a line. That text is what flame graph tools read, so you can save it and draw it with the tool you like, or hand it to an agent:1
2
3
4
5memstress;clone3;start_thread;worker;kernel_[k] 56
memstress;clone3;start_thread;worker;munmap;sys_munmap_[k] 32
memstress;clone3;start_thread;worker 31
memstress;clone3;start_thread;worker;mmap64;sys_mmap_[k] 12
memstress;clone3;start_thread;worker;madvise;sys_madvise_[k] 10-o jsongives the same stacks as data.
Machine takes the samples from outside the guest, at every timer interrupt of each CPU. Nothing is installed in your image and nothing runs in the guest for it, so a run is the same with a profile and without. Your programs do not need frame pointers: a stack is walked by the unwind information every program carries for exceptions.
A recorded stretch
Because a run is recorded, you can profile a stretch of it after the fact, even if no profiler was on at the time:
1 | machine profile app --from ^request --to ^reply |
The stretch is replayed on a branch, which is deleted afterwards. Your run is not touched, and it does not need to be running. Profile the same stretch twice and you get the same samples.
Where names come from
- Names come from the program files on the machine’s disk as it is when the stretch ends. You do not need a checkpoint, and a profile does not take one.
- A file the guest has written but not synced to disk yet shows as
[file]. Runsyncin the guest first. - A stripped program with no unwind information, such as Alpine’s
busybox, is one frame named after its file. - Time in the kernel is one frame. The functions of the kernel itself are not shown.
--in-guestsamples with a small process inside the guest instead. Use it when you need the kernel’s own functions by name. It is part of the run, and it cannot profile a recorded stretch.- Samples are taken 250 times a second on each CPU.
- C++ names are printed as the program has them. Pipe
--foldedthroughc++filtto read them.
From Python, run.profile("5s", container="db") returns a Profile with
samples, functions, stacks and the folded text, and
run.profile(from_="^slow", to="^done") profiles a stretch already recorded.
Interactive mode
When you are working on one run for a while, you can open a prompt for it, so
you no longer type machine and the run’s name before every command.
1 | machine control app |
You issue commands to the run as slash commands, like /checkpoint,
/branches or /rewind. Most of the commands on this page have one, written
the same way without the run’s name. Commands that run the machine, take
checkpoints or make and switch branches take their options as plain words, so
--at ^booted becomes at ^booted. Commands that execute something in the
guest or read its state, such as /exec, /send, /profile or /regs, take
the same flags as on the command line: /exec run --container app nproc. Type
/help to see them all, and /q or Ctrl+D to leave.
/h and a command shows that command with examples, such as /h exec. A word
in capitals in the help is a type: /h RANGE says how to write a range, and
/h POINT a point.
Above the prompt, a line shows where the run is: its time, its entry, the vCPU,
the RIP, what the entry is, such as IO_OUT, and its state hash. Back at an
entry, the same hash means the same state.
1 | ⌖ vtime=4.648247180s entry=32103 vcpu=0 rip=0xffffffff8104880d IO_OUT hash=0x91c170d2d6da8e05 |
The prompt is the easiest way to try something and come back. The session below
starts from a new run made from the ubuntu-26.04 disk, the one built in
Multiple vCPUs from Ubuntu 26.04 with a container that waits
for your commands. Disks has more on building one.
1 | machine disks build --from ubuntu:26.04 ubuntu-26.04 --entrypoint /bin/sleep -- infinity |
1 | ✔ Started app (whyvcWEKxc7c5b2UwO7r6) |
1 | machine control app |
At the prompt, it boots the machine until Docker attaches to its container, keeps that point as a checkpoint, and removes a file:
1 | /f until console~Attaching within 60s |
/f runs the machine until Attaching appears on the console, or for 60
seconds of the machine’s own time at most. Here it takes 3.7 seconds of the
machine’s time, and the table shows where each vCPU stopped. To see the
machine as it was before the file was removed, fork a branch at booted and
switch onto it:
1 | /go to ^booted |
The branch is named after the entry it was forked at, and the file is there
again, while app keeps its own history without it. /switch app moves the
machine back. The prompt also remembers the branches you have visited:
/go back returns to where the last switch started, /go up and /go down
walk the visits in order, and /history lists them. /mark good gives the
branch you are on a name, and /switch %good returns to it.
A branch made at the prompt takes the same options as one made with
machine fork, written as words. This one records
a branch that repeats, with a smaller timer slack than the default, as
Recording that repeats describes:
1 | /fork at ^booted name exact repeatable timer-slack 5ms |
/fork starts the branch beside the one you are on and leaves you where you
are. /switch app/exact moves onto it, and /describe:app/exact shows
Repeatable: true and Timer Slack: 5ms.
-c runs a single slash command and exits, as in
machine control app -c '/state', and machine control log app lists the
commands that changed the run.
Keys
The prompt edits a line the way your shell does.
| Key | Does |
|---|---|
| Tab | completes a command, and after /h a command or a type |
| Up, Down | walk the commands typed at this prompt, the last 500; they are not saved when it closes |
| Ctrl+C | cancels the run, checkpoint or hash in progress, and keeps the prompt; a run stops at the end of its dispatch, and a second Ctrl+C stops it at once |
Ctrl+D, /q |
leaves the prompt; the run carries on as it was |
; |
runs the next command after this one, and stops at the first that fails: /s;/d |
;+ |
runs the next command whatever this one did: /replay;+/diff |
Quotes work as in a shell, so a ; inside quotes is part of the word:
/send "a;b" types a;b. A line with a quote left open continues on the next
one, at a > prompt.
Running at the prompt
/run drives the machine as machine run does, and
every option of machine run is a word: --until is until, --within is
within, --irq-delay is irq-delay. Two shorter forms pick the mode: /f
runs fast and /s steps. A suffix picks the vCPU and the number of dispatches:
=N drives vCPU N alone, and @N runs N dispatches.
1 | /f until console~ready within 30s |
The first runs fast until ready appears on the console, or for 30 seconds of
the machine’s time. /s@20 steps 20 dispatches, and /f=1@10 runs vCPU 1 fast
for 10 dispatches. The last two draw something per dispatch from a seed: the
mode, then what the focused tasks do and how long each runs, as
Schedules describes. /h run lists every word.
Reading another run
A command that reads a run takes :RUN to read another one without switching
to it: /describe:app/exact, /checkpoints:app/try-1, /entries:app -20,
/lineage:parent, /d:app/try-1 regs. Where you stand stays the same.
Words that differ from the command line
Most slash commands are machine commands with the run’s name left out and the
options of commands that act written as words. These differ:
| At the prompt | On the command line |
|---|---|
/go to ^42 |
machine go app ^42 |
/go up, /go down, /go back, /history |
none: the prompt’s own record of the branches you visited |
/delete try-1 recursive force |
machine delete app/try-1 -r -f; the prompt refuses the branch you are on |
/mark good, /mark -good, /marks |
machine marks add, remove, list |
/vcpu=1 |
machine vcpu app --vcpu 1 |
/d regs |
machine dump app --show regs |
/checkpoint booted |
machine checkpoint app booted; /checkpoint name booted also works |
/auto-checkpoints 10000 |
machine run app --auto-checkpoints 10000, at the prompt a setting that stays |
/now |
machine now app |
create, start, stop, pause, resume, kill, runs, sessions and
disks have no slash command. Use them from the command line beside the prompt.
Scripts and history
-c sends one line and exits, so a script can use the prompt’s syntax. A line
can chain commands:
1 | machine control app -c '/checkpoint name before' |
machine control log lists every command that
changed the run, from the prompt or the command line, with the entry where it
was issued. Commands that only read are not listed. --wide adds columns.
Stopping and starting
1 | machine pause app |
pause freezes a running machine and resume lets it carry on. stop shuts a
run down and keeps everything it recorded. kill asks the machine to stop and
save as stop does, and ends it if it does not stop. The next start replays
a tip that was not saved. In
a session, both wait until what the run recorded is saved to your storage,
and show how far the save has got. start brings it
back at the end of its log. If the run had gone past its last checkpoint
when it stopped, Machine replays the recorded tail from that checkpoint and
tells you how many entries it replayed. Add --no-recover to make it
refuse instead.
start can also take a run back to an earlier point. --at replays from an
earlier checkpoint, and --read-only opens a checkpoint without running it
at all.
1 | machine start app --at ^booted # replay from booted to the end of the log |
A run recorded by another version of Machine does not start. Its recording
replays only on the version that made it, so start refuses it with
it was recorded by another version, followed by the version of its run log
and the version the session reads.
--at brings the run up at the checkpoint and replays every entry after it,
so the run ends at its tip as it would have without the option, and you see
that the recording still replays. A read-only run accepts commands that read,
such as machine dump, machine hash
and machine output, and refuses run and
anything that would record. Stop it, and start it again without --read-only
to go on.
A run remembers the options you created it with: --no-daemon,
--single-step, --limit and --backstop-limit. Every start and fork
repeats them, so you type them once. start takes the same flags to change
them for that start only, so machine start app --daemon gives you a machine
to drive on a run created with --no-daemon. fork and detach take them
too, and the new run keeps what you gave. --pin-cpu is the exception: no run
remembers a CPU, so each start and fork picks the least-loaded one unless
you name it.
What a run records is fixed when you create it: --side-dump, --side-trace,
--mem-hash, --disk-hash and --repeatable. A start keeps it. A branch is
where it changes: fork, detach and go take each of them, and
--no-mem-hash, --no-disk-hash or --no-repeatable to turn one off, and a
branch given none records what its parent records. Replay says what the two hashes add.
delete only removes stopped runs, unless you add --stop. It also refuses to
remove a run that has branches, unless you add --recursive. And since
machine delete --all removes every run, it asks you first.
delete waits until the run is gone, which takes a while for a run with many
checkpoints. Add --no-wait and it returns as soon as the delete is accepted,
after stopping the machine if you asked it to. The delete carries on either
way, and machine ops shows how it went.
A delete that was interrupted, by a lost connection or a stopped service,
finishes when you run the same delete again. --cleanup -a finishes every
interrupted delete at once, and --dry-run lists them first:
1 | machine delete --cleanup -a --dry-run |
To continue a run in a different session, publish it first from the session that has it.
1 | machine sync app |
To have a run or a disk waiting in a session before you need it, preload it.
Going the other way, machine offload lists the copies in this session that
can be dropped to free space, and -a drops every one that isn’t in use. The
durable copy stays, and starting the run brings it back.
1 | machine preload run app |
machine cleanup removes stale runs and other leftovers, and --force also
kills machines that have been orphaned.
Checkpoints and branches
Branching is cheap, because a branch shares its parent’s history instead of copying it. That is what makes it practical to try many things from one point.
Checkpoints
1 | machine checkpoint app booted |
The name is optional. Leave it off and the checkpoint is named after the entry it was taken at. Checkpoints are how you mark points you plan to come back to, like “just after boot” or “right before the request”.
A name uses letters, digits, ., - and _, and starts with a letter or
digit. It can’t be a number, because numbers name the checkpoints taken without
one, and it can’t read as a time, like 10ms, because a time names a point too.
Machine also takes its own, at the interval create sets, named after the entry they were taken at
(like 10000), so machine checkpoints app lists yours next to those.
The DURABLE column says whether a checkpoint is in your account’s storage or
only in the session. A new checkpoint shows no until the run is published,
with machine sync app or by stopping the machine. On a branch, the listing
includes its ancestors’ checkpoints, named in the ANCESTOR column. The point a
branch was forked at is listed once, as its ancestor’s checkpoint, because
that is the one a fork there starts from.
--durable lists only the checkpoints in your account’s storage, the ones
another session can start the run from:
1 | machine checkpoints app --durable |
From Python, run.checkpoint("booted") saves one and returns it, and
run.checkpoints() lists them.
Starting a branch
machine fork creates a new branch at a point. The branch runs in its own
machine, alongside its parent, and the parent keeps running.
1 | machine fork app # at the current point |
Give a branch a name with --name. If you don’t, it is named after the
entry it landed on. Add --no-start to create the branch without starting it.
A branch’s full name is the name of the run its tree starts from, a slash, and
the name you gave it. So machine fork app --name try-1 makes app/try-1,
and a branch of app/try-1 named try-2 is app/try-2. Every other command
takes the name you gave it, machine lineage try-1,
as long as no other run has a branch of that name. When two trees both have a
try-1, Machine refuses the short name and lists both, so use the full name:
machine lineage app/try-1.
From Python, run.fork(at="^booted", name="try-1") returns the branch as a
Run, so you drive it like any other: branch.run(...), branch.replay().
Switching the machine
machine fork gives the new branch a machine of its own and leaves you where
you were. go, switch and rewind work in place instead: they change the
machine you are on. detach is the odd one out, and is explained below.
| Command | Does |
|---|---|
machine go app ^booted |
creates a branch at a point and switches this machine onto it, in place |
machine switch app app/try-1 |
switches the machine onto a branch that already exists, and never creates one |
machine detach app --at ^booted |
copies a point into a new root, a self-contained run in a tree of its own |
machine rewind app to ^booted |
puts the run back at a point and throws away what came after |
go is the one you will use most. It creates a branch the way fork does, but
it doesn’t start another machine. It switches the one you are on, in place.
1 | machine go app ^booted --name g1 |
1 | ✔ Branched app/g1 (whUxiqseEZ7Si8R4osT8K) at entry 16239 and switched onto it. |
After a go or a switch, the run the machine left is listed as switched
and no longer running. machine start app brings it back at its tip, and tells
you how many entries it had to replay to get there. The machine you switched is
now the new branch, so talk to it by that name: app/g1.
A switch needs a branch that isn’t running somewhere else. Pointing it at a
branch that is gets refused:
1 | ✗ Already running: app/try-1. |
Creating the branch with --no-start avoids this.
rewind changes the run itself, not a copy of it. Use it when you want to
record again from an earlier point. The run keeps its name. Machine refuses to rewind
past a branch that starts after that point, since that branch depends on entries
that would no longer exist.
switch takes a branch name, a mark (%crashed), parent or root.
detach is for leaving a tree behind. A branch normally shares its ancestors’
data, which is what keeps it cheap. detach copies everything into a new root
that owns all of its data and belongs to no tree but its own. That is much
slower than a fork, so only reach for it when you want to get rid of the
original.
If you only want a branch to have its own copy of the data, run
machine fork --rebase, which is quicker. Add --lean and the branch keeps no
history shared with its ancestors, so they can be deleted.
The tree
1 | machine branches app # the whole tree this run belongs to |
branches lists the newest 25 by default. --running shows only the ones
still running, and --depth keeps one level of the tree.
Marks
When a branch turns out to be interesting, mark it.
1 | machine marks add app/try-1 crashed |
A mark is unique within its tree, so you can use it anywhere a run is named, written with a percent sign.
1 | machine output %crashed |
From Python, run.marks.add("crashed") marks the branch, run.marks.remove("crashed")
takes the mark off, and client.marks.list() lists every mark.
Replay
Replay is what makes a recording useful. A branch can play back what its parent recorded and Machine checks, step by step, that it gets the same result.
1 | machine replay app/try-1 |
Replay is strict. An entry that does not reproduce what was recorded means
the run has diverged. Before giving up on it, Machine replays that entry
again from the nearest checkpoint before it, twice by default. Use --retry
to change that, or --retry 0 to try each entry once.
--to stops at a point instead of the end. It takes a point as every command
does: --to ^booted for a checkpoint, --to '#300' for an entry, --to 10ms
for a time. The point is one of the line being replayed, so a checkpoint the
parent took after the fork is a valid one. --trace replays one iteration at
a time and prints each entry as it goes, which is slower.
Each entry holds hashes of the machine’s state: its registers, its devices
and its timers. A replay compares them all. A run created with --mem-hash
also hashes the guest’s memory at every entry, and one created with
--disk-hash its disk, and a replay of it compares those as well. Hashing
costs time at every entry, so both are off by default. A divergence of memory
or the disk alone then shows up later, once it reaches a register or a
device. A branch can turn either on or off:
1 | machine create --disk app.img --mem-hash --disk-hash # hash both from the start |
A replay hashes memory and the disk exactly where the recording did, whatever
the replaying branch records, so a replay compares either kind of recording.
diff of an entry that hashed one against an entry that did not leaves it
out, and says Memory not compared or Disk not compared.
The host can interrupt an entry too. That is not a divergence. Replay prints
Interrupted: entry N, crossing 1: host SMI and replays the entry again,
counted against the same --retry. A recording does the same on its own, see
Driving it.
Replay margins
A replay reaches each recorded stop by letting the CPU’s counter interrupt a
margin short of it, then stepping to the stop. --limit-margin is that margin
in conditional branches, and --instr-margin in instructions, for a stop
placed by an instruction count. A replay that names either uses the values it
names. Without them, it uses the larger of the run’s margin and the host CPU’s.
A start, a start --at, a rewind and a fork --from name none, so they use
the larger of the two as well.
1 | machine replay app/try-1 --limit-margin 20000 |
A replay that passes a recorded stop diverges and says by how much. On amd-1,
the architecture of the standard class, a replay given --instr-margin 5, far below the CPU’s skid, answers:
1 | ✗ Divergence at entry 804. The replay passed the recorded stop by 3,370 instructions, with its margin 5. |
The suggested margin is twice the overshoot and the margin together, and
never less than the host CPU’s margin. Here twice the overshoot and the
margin is 6,750, so the hint is the AMD CPU’s margin, 50,000. A stop
counted in branches names --limit-margin instead. At the
control prompt the hint is
/replay instr-margin 50000. A replay that reaches the stop’s
address on a later pass, with other registers, gets no margin hint, because
no margin explains it.
If a run does diverge, machine diff shows how.
1 | machine diff app/try-1 |
With no second argument, diff compares the entry against the run it was
replayed from, and says ✔ No differences when they match. Otherwise it prints
a hash by default, and --show takes any of the
sections in its help: registers, stack, page tables, timers, devices,
memory and more. Name another run to compare two recordings. A difference
there only means the recordings differ. It says nothing about determinism.
There is also machine play, which plays a log back without checking for
divergence. It is for when you want to see what happened, not whether it
repeated. By default it plays the parent’s log, so it needs a branch. To play
another run’s log, name it: machine play app/try-1 app.
1 | machine play app/try-1 -n 16240:16243 |
--include plays only the kinds of entry you name, and --exclude plays every
kind but those.
Comparing entries
machine diff compares one entry. A difference often starts a few entries
before the one that shows it, so --context also prints the entries leading
up to it, on both sides:
1 | machine diff app/try-1 --entry 5000 --context 5 # a branch against the run it replayed |
Between two recordings, such as two branches with different seeds, a difference is expected: it shows where their paths parted.
A replay that diverges is different. Every recording must replay, so a divergence is a bug in Machine, not in your program. Keep the run and the branch that diverged: the recording is what reproduces it. Open an issue with the session, the run, the branch, the entry where it diverged and the commands you ran.
At the control prompt
Replay and diff are one loop at the control prompt.
Open it on the branch, and chain the two with ;+, so /diff runs whether
the replay converged or not:
1 | machine control app/try-1 |
1 | /replay to ^booted;+/diff |
/replay takes to, retry, trace, limit-margin and instr-margin as
words. /diff 120 regs compares
entry 120 with its recording and shows only the registers, and context 5
adds the five entries before it. /play feeds the parent’s entries 295 to 300
without checking them, here only the interrupts.
From Python
run.replay() returns the result instead of raising. Check converged, and
diverged holds what differed:
1 | kid = run.fork(at="^booted", name="check", start=False) |
run.replay(to="^booted", retry=5, trace=True) takes the options of the
command. The SDK has the rest.
Comparing two machines
machine diff compares an entry with the one it
was replayed from. machine hash compares two
machines stopped at the same point: a recording, and a branch you drove
there yourself.
1 | machine hash app |
The command computes the hashes of the machine as it stands: memory, the
disk, each device, each part of each vCPU, and irq_pending, the hash of
the interrupts --irq-delay is
holding. It changes nothing, and the run records nothing. Memory is read
from all of RAM, so it takes as long as that reading takes. The machine
refuses commands that would change it until the command returns.
machine cancel stops it.
machine dump --show hash prints what the newest
entry recorded, before its exit was handled. Memory and the disk are 0 there
unless the run hashes them at each entry. machine hash is after that entry,
with the exit handled, and it hashes memory and the disk whether or not the
run does at each entry.
A replay compares the hashes the recording stored at each entry. hash is
for two machines you have in front of you.
Steering programs
By default the machine schedules itself. Sometimes that isn’t what you want. A bug that needs one thread to run past another will not show up if the scheduler never picks that order. Machine lets you pick it.
Everything here is recorded, so a replay repeats it exactly.
Looking at tasks
1 | machine tasks app |
That lists the tasks in the machine, one row per task by its tid, with the
tgid each one belongs to. You will want a tid, or a program’s path, in the
commands below. machine syscalls app and machine ints app list the kernel’s
system calls and interrupt vectors.
Focus
Machine matches a program by its absolute path in the machine. Tell it which programs to watch and steer.
1 | machine focus app add /app |
Focus also takes a cgroup (cgroup=/path), a single process (tgid=12), or
the processes with a task of a given name (comm=cat). A name picks out one
tool of a multi-call binary such as busybox, or a script, which a path cannot.
machine tasks app exist comm=cat lists the tasks with
that name; focus takes every thread of their processes.
A program that renames itself after it starts, with prctl(PR_SET_NAME) or
/proc/self/comm, is focused by its new name at its next fork, exec or exit,
not at the rename. Until then --until comm=worker:on waits. Focus the program
by its path to steer it from the start. focus remove and focus clear take them off again.
1 | machine focus app add comm=cat |
Whatever you focus by, the conditions below and a plan’s when name it the
same way: /app:exit, cgroup=/path:on, comm=cat:kernel.
Once a program is focused you can wait for things that happen to it. The task
conditions take the form program:what, where what is one of:
| What | Means |
|---|---|
exec |
the program started |
exit |
the program exited |
sched |
one of its threads was given a CPU |
desched |
one of its threads gave its CPU up |
on |
a thread is running its own code |
kernel |
a thread is running in the kernel |
off |
the thread is not on a CPU at all |
A run can also stop where a thread reaches a location, or after it writes or touches bytes. These need no focus. They name a tid or a tgid:
1 | machine run app --until 'tid=56:at=parse' # until tid 56 reaches parse |
Each takes one of the run’s 3 hardware breakpoints and watchpoints, for that run alone. Debugging says what they take and what the run says when it stops.
Holding threads
1 | machine tasks app hold 56 # take tid 56 off the run queue |
Each of these takes effect at the next scheduling decision. If you hold the
last task that could run, the machine has nothing to run. Release it, then
force a decision with machine irq app preempt --vcpu all.
There is also pin, which places a thread on the vCPUs you choose the next
time it wakes:
1 | machine tasks app pin 56 2 # on vCPU 2 |
A set is one vCPU or a range, both ends included. A pin decides where the
thread goes when it wakes, so a thread already queued on another vCPU stays
there until then. none keeps a thread from running at all, until you pin it
somewhere again.
Two more things you can ask of tasks. machine tasks app exits lists the
tasks that have left and how each one ended, and machine tasks app cgroups
groups them by cgroup. --focused shows only the tasks you are steering.
From Python, the same verbs are run.tasks.hold(56), run.tasks.release(56),
run.tasks.yield_(56), run.tasks.slice(56, "10us") and
run.tasks.pin(56, "2"). run.tasks.exits() and run.tasks.cgroups() read
the other two lists.
Plans
When one rule isn’t enough, write a plan file. A plan is a list of clauses, and each one says when it applies and what to do.
1 | when /test:on mode step,fast fast=1:100k step=1:200 |
Machine tries the clauses in order and the last match wins, so a later
clause overrides an earlier one. else covers whatever nothing matched.
Each when names a state of a program (/test:on is the program running its
own code, and /test:kernel is the program running in the kernel). After that come the
things to choose for each stretch of execution it covers, such as the mode, how
many conditional branches to run, or what to do with the scheduler. machine plan
--help lists all of them.
1 | machine plan app plan.txt |
A clause draws any of these, once per dispatch, and irq-delay once per
interrupt:
| Axis | Draws |
|---|---|
mode |
how the machine runs: fast, step, step-into, free, fence or keep, with the length of a stretch as fast=1:100k |
task-sched |
what the task on the vCPU does: keep, hold, yield, drain, drain-all |
task-int |
whether to preempt that task: none or preempt |
task-slice |
how long that task runs between decisions: keep,default,10us:600us |
task-pin |
where a task goes when it next wakes: a vCPU set, free or none |
vtime-rate |
how fast the machine’s clock passes, as in Clock rate |
vtime-add |
time added to the clock before the dispatch, such as 10us:2ms |
timer-slack |
how late a timer may be, such as 0:20ms |
irq-delay |
how long each interrupt is held, as in Holding interrupts |
Every axis is a comma list of choices, and [N] makes a
choice N times as likely: mode step[3],fast[1] draws step three times as
often as fast. keep draws nothing and leaves the axis as it was, so a clause
can leave an axis alone part of the time. A range such as 1:200 draws one
value from it, and the plan’s --seed makes every draw repeatable.
A when names what a task is doing: /path:on for a program running its own
code, /path:kernel in the kernel, /path:syscall=openat in one system call. The
subject can also be tid=N, tgid=N, comm=NAME or cgroup=PATH, and
vcpu=1 narrows it to one vCPU. Braces may group a clause, # starts a
comment, and a file named - is read from standard input.
1 | # Find the race: interrupt the writer, and move it between vCPUs. |
The plan applies only to that machine plan. A later machine run does not
use it.
From Python, run.plan("plan.txt", seed=3) runs a plan from a file, and the
RunResult it returns lists each clause with the iterations it applied to. The
plan_file.py example shows one.
Interrupts
Inject an interrupt by name or number.
1 | machine irq app preempt --vcpu 1 |
The interrupt is queued, then delivered at the next boundary and recorded there, so a replay delivers it at the same place.
--nmi delivers a non-maskable interrupt instead, which reaches a vCPU even
while the guest has interrupts turned off, so you can test how a program or
the kernel handles one:
1 | machine irq app --nmi --vcpu 0 |
Holding interrupts
machine run can hold each interrupt a drawn time
before it is delivered, at most one second:
1 | machine run app --irq-delay 0:2ms |
A bare span holds every interrupt. Name a target to hold one kind: timer,
ipi, init, blk, rng, console, serial, keyboard, device or
all. The most specific target wins. An interrupt is never delivered before
one held earlier on its line, its MSI vector, or its vCPU’s timer, IPI
vector, or INIT and SIPI.
machine irqs lists every interrupt the machine
owes or holds, then the vectors machine irq
injects. An owed interrupt was raised by a device or a timer and is injected
at the next dispatch. A held one waits for the time the draw gave it. The
state hash irq_pending is of the held ones, and
machine state shows how many are held.
1 | machine irqs app |
The run log keeps each hold, and a checkpoint keeps what is held. At the
control prompt the same option is
irq-delay on /run. From Python it is run.run(irq_delay="0:2ms"), and
run.irq.pending() lists them.
Machine time
Machine time is not the host’s. machine vtime app rate
changes how fast it passes, and machine vtime app add pushes it forward
without running anything.
1 | machine vtime app add 5s |
There is no way back. The machine’s clock only goes forward, so to return to an
earlier time, branch there with machine go app 1.5s.
Starting date
Every machine’s clock starts at 2026-03-01 00:00 UTC, whatever the date is on your computer, so a program that reads the date reads the same one in every run. To test code that depends on the date, such as a certificate that expires or a job that runs at the end of a month, give the start as a Unix timestamp when you create the run:
1 | machine create app --disk app --vtime-base 1893456000 # 2030-01-01 00:00 UTC |
The run keeps its starting date, and every branch of it has the same one.
Clock rate
The machine’s clock moves with the work the guest does: 10 nanoseconds for
each conditional branch a vCPU retires, unless the run was created with
--vtime-rate. A higher rate makes the same work take longer in the guest’s
eyes, which brings timeouts closer, and a lower one makes it faster. The rate
is a number or a fraction:
1 | machine vtime app rate 20 # the same work takes twice as long |
The change is recorded, so a replay changes the rate at the same point. Idle time is a separate matter: Time compression skips it whatever the rate.
From Python, client.create("app", disk="app", vtime_base=1893456000) sets
the starting date, run.vtime.rate("20") the rate, and run.vtime.add("5s")
pushes the clock forward.
Fault injection makes things fail: system calls, the kernel and the disk, a service’s container, and the date the guest and its containers read.
Fault injection
Machine makes things fail inside the machine, at the point you choose or by chance, and records each fault with the run, so a replay fails exactly the same way. There are four kinds of fault, each with its own command:
| Command | What it makes fail | Section |
|---|---|---|
machine inject |
a program’s system calls and its functions, the kernel’s disk requests and memory allocations, and the disk’s data | System call faults, Function faults, Kernel faults, Disk faults |
machine containers |
one service’s container: stopped, killed, crashed, paused, throttled, or limited in memory, processes, open files or local ports | Container faults, Resource exhaustion |
machine clock |
the date the guest or one container reads: stepped, drifting, jittering, or a leap second | Clock faults |
machine net |
the links between containers: slow, lossy, out of order, rate limited, or cut | Network faults |
machine inject makes things fail inside the machine. A fault goes into a
subsystem: a program’s system calls, or one of the kernel’s own places where
things can fail, such as the disk or memory allocation, or the disk itself. You
find out what your program does when openat says EACCES, write says
ENOSPC, the disk returns an I/O error, the kernel runs out of memory, or the
disk quietly returns the wrong bytes, at the call you choose or by chance.
| Subsystem | What fails |
|---|---|
syscall |
a system call returns an error you choose, or waits first |
function |
a function of a program, such as malloc, returns a value you choose without running |
disk |
a disk read or write returns an I/O error |
disk-timeout |
a disk request never completes, and its task waits forever |
slab, pages |
a kernel memory allocation fails, as ENOMEM in most calls |
usercopy |
the kernel fails to copy to or from the program’s memory: EFAULT |
futex-access |
a futex call fails to reach the program’s memory: EFAULT |
disk-corrupt |
a disk read or write succeeds with bytes changed |
disk-torn |
a write succeeds, but only its first sectors reach the disk |
disk-lost |
a write succeeds, and none of it reaches the disk |
Every subsystem takes the same options: --in for whose calls, --prob for the
chance, --after to let calls through first, --times to stop after a number,
--seed, and --on and --off to make the fault cycle.
The disk subsystems also take --file, --op and --sectors to say which
requests. The guest kernel decides each call itself, without stopping
the machine, and a replay fails exactly the same calls.
System call faults
A syscall fault names a system call and the errors it may return, and the program’s calls of it return one of them without running.
1 | machine inject app add syscall connect ECONNREFUSED --in /usr/bin/app |
The last command lists the faults, with how many calls each one matched and fired on. For a program that connects once, it shows that one call failed. The seed is drawn, so yours differs:
1 | # INJECT ERRORS DELAY IN FIRES PROB SEED MATCHED FIRED |
The guest kernel decides at each call, and the run records every fault you add, so a replay fails the same calls with the same errors. A checkpoint keeps the faults and their counts, and a branch has its parent’s.
Which calls fail
--in says whose calls fail, and you always give it:
A program, by its absolute path:
--in /usr/bin/app. The fault applies to every task running that program, and to the processes it forks without starting another program, such as a server’s workers. It starts with the program, so you can add it before the program runs. Use the pathmachine tasks appshows as EXE: a symlink counts as the file it points to, so busybox’s/bin/catis/bin/busybox. A program that loads shared libraries opens and reads them after it starts, so add a fault onopenat,readorstatonce it is up, or it fails the loading instead.A task’s name:
--in comm=cat. The kernel names a task after the file the command named, up to 15 bytes. That picks out one tool of a multi-call binary such as busybox, and a script, whose program is its interpreter but whose name is its own. The interpreter’s first open is of the script itself, so--after 1lets it read the script and fails what the script opens:1
2machine inject app add syscall open EACCES --in comm=s.sh --after 1
machine exec app run -- /tmp/s.shThat is busybox’s shell, which calls
open. A glibc shell callsopenatand opens its shared libraries first: count them in--after, or add the fault once the script runs.One process:
--in tgid=412, its pid. Add it once the program is running. Before then, that process is still being set up to run it.One thread:
--in tid=41.Every task in the machine, the system’s own included:
--in all.
To make sure something runs before you act on it, ask first. machine tasks app exist takes the same words, lists the tasks that match, and fails when none
does. It only reads, so it changes nothing:
1 | machine tasks app exist /usr/bin/app && machine inject app add syscall openat ENOENT --in /usr/bin/app |
A rule’s --after and --times count calls across every task it matches: for
a program, the 100th write is the 100th of all its processes together.
1 | machine inject app add syscall write ENOSPC --in /usr/bin/app --after 99 --times 1 # the 100th write |
--after lets that many matching calls through first. --prob is the chance a
call fails, from 0 to 1. --times is the most calls that fail; without it there
is no end.
The chance belongs to each call, not to a stretch of time. A fault stands from
when you add it until you delete it, and every matching call draws on its own:
1,000 calls at --prob 0.1 fail about 100 of them, however long they take. To
fault a stretch, such as the next 10 seconds of machine time, add the fault, run
for that long, and delete it:
1 | machine inject app add syscall read EIO --in /usr/bin/app --prob 0.1 |
To fail one particular call, run until the program makes it, then add the fault for one call. The stop is at the start of the call, so that same call fails:
1 | machine run app --until tgid=412:syscall=write |
Name several system calls at once and each gets a fault of its own:
1 | machine inject app add syscall open,openat,creat EACCES --in /usr/bin/app |
Which errors
Any Linux error goes, by name or by number, whether or not the system call’s
manual lists it. Give a set and each failing call returns one of them. The
examples here fail openat of /usr/bin/app, so add them once it runs, after
machine tasks app exist /usr/bin/app:
1 | machine inject app add syscall openat ENOENT,EACCES,EMFILE --in /usr/bin/app # one of three, each call |
valid is the errors the system call’s manual page lists. all is every error
Linux names, which is more than the 30 a fault holds, so use --pick with it.
An error the manual does not list is one Linux would not return from that
call. getpid failing with ENOSPC never happens on a real system. Machine
allows it on purpose: it tests whether the program handles an error it did not
expect. Use valid for failures a real system produces, and an error by name
or all to test the program’s handling of any error.
Picking errors at random
--pick takes a number of errors from the set when you add the fault, and the
fault keeps those. This is swarm testing: each run fails with its own few errors.
1 | machine inject app add syscall openat,read,write valid --pick 1 --in /usr/bin/app # one error each |
Every fault has a seed, and the list shows it. The seed decides the picks,
which calls fail by chance and which error each one returns. Give --seed to
draw as another run did:
1 | machine inject app add syscall openat valid --pick 2 --seed 7 --in /usr/bin/app |
Slow calls
A fault can hold a call instead of failing it. --delay takes a time, or a
span to draw one from for each call, up to a second:
1 | machine inject app add syscall read --delay 0:20ms --in /usr/bin/app # each read waits 0 to 20ms, then runs |
The wait is on the machine’s own clock, and each call draws its time from the
fault’s seed, so a replay holds the same calls for the same time. --prob,
--after, --times and --in work as they do for errors.
Function faults
function makes a function of a program return a value without running. A
malloc that returns 0 is an allocation that failed, which no system call
fault reaches: the C library serves most allocations from memory it already
has, without asking the kernel.
1 | machine inject app add function malloc --return 0 --prob 0.01 --in /usr/bin/app |
The function is found by its name in the files a running task maps, its C
library or the program itself, and the fault acts on every task that maps
the same file and that --in names. With --in naming a program that runs
now, the function is found in that program. With one that does not run yet,
name with --at a task that maps the same library, such as a shell of the
same container.
For example, --in comm=sort with --return 0 makes BusyBox’s sort exit 2
on its first allocation, while seq and tail in the same container run as
before. With --prob 0.05, four runs of sort match 60 calls and fail 4. The
seed decides which calls fail, so a replay fails the same calls.
- errno is not set. The C library keeps errno in each thread’s own memory, so a caller that checks the return sees the failure, and one that reads errno reads what was there before.
- The function’s first instruction. The kernel puts a breakpoint there, in
the tasks
--innames alone, and a fire returns to the caller before the function sets up its frame. Each call traps into the kernel, which costs time in a function called often. - The task
--atnames. It must be a task inside the container, not thedocker execthat started it:machine tasks appshows each task’s program as the container sees it, such as/bin/sh.
Memory faults
Memory goes wrong in four places: a program’s allocator returns nothing, the kernel refuses memory, a service runs out of it, or the data in it is wrong. Each has its own fault:
| You want to know | Fault |
|---|---|
Does the program handle malloc returning NULL? (errno is not set) |
inject add function malloc --return 0 --after 5 --times 1 --in /usr/bin/app |
| Does it survive one allocation in a hundred failing? | inject add function malloc --return 0 --prob 0.01 --in /usr/bin/app |
| Does it handle the kernel refusing to grow its heap? | inject add syscall brk ENOMEM --in /usr/bin/app |
| Does a system call fail cleanly when the kernel cannot allocate? | inject add slab --after 50 --times 1 --in /usr/bin/app |
| What happens when a service runs out of memory? | containers memory db 64m |
| How does a service behave under memory pressure, short of a kill? | containers memory db 48m --soft |
| Does the program notice a flipped bit in its data? | mem 412 'checksum^=0x1' |
| What does a pointer set to 0 do? | mem 412 'head=0' |
| What does a buffer of garbage do? | mem 412 buf --fill 0xde --length 4096 |
A program’s allocations
A program asks its C library for memory, and the library serves most
requests from memory it already has, without a system call. A
function fault on malloc reaches every allocation:
1 | machine inject app add function malloc --return 0 --after 5 --times 1 --in comm=sort --at 628 |
1 | sort: out of memory |
sort failed on its sixth allocation, and the next run, with the fault used
up, sorted as before. calloc, realloc and strdup are functions of their
own: name each that the program calls.
The kernel’s answer
brk grows the heap and mmap maps new memory. A fault on them is the
kernel refusing, which the C library sees first: it turns a refused brk
into an mmap, and a refused mmap into malloc returning NULL.
1 | machine inject app add syscall brk ENOMEM --in /usr/bin/app |
A refused brk returns the break unchanged, as the kernel’s own refusal does,
and the C library moves to mmap: with every brk of sort failing, all 12,
it sorted 100,000 lines and exited 0. That is the test of a program that
assumes its heap is contiguous.
A program’s first mmap calls are its dynamic loader’s, mapping its
libraries, and a fault on those stops the program before it runs: sort
failed with error while loading shared libraries: libc.so.6: failed to map segment from shared object with mmap failing after its 13th call. A program
with a small heap makes no mmap of its own, so function malloc is the
fault that reaches its allocations.
The kernel’s own allocations
slab and pages fail allocations inside the kernel, for work a system call
or a page fault does for the program, as Kernel faults
describes. The program sees what the kernel does with the failure: a system
call that returns ENOMEM, or a page fault that cannot be served. In one
test, failing the 51st slab allocation of sort killed it with SIGSEGV.
A service out of memory
machine containers memory limits a container’s
memory. Past the limit the guest kernel reclaims memory, and its
out-of-memory killer kills a process of the container when it cannot:
1 | machine containers app memory db 32m |
1 | ⚠ Exit status 137: killed by SIGKILL, as Docker reports a signal |
--soft sets the limit past which the kernel reclaims and slows the
container’s processes, and kills none of them, as memory pressure does. What
the kernel can reclaim decides what the program sees, and the machine has no
swap:
- Files the program reads are cached in memory the kernel can drop and read again. Past a soft limit the program keeps running, slower. For example, reading a 100MB file takes 0.20 seconds of machine time against a soft limit of 16MB and 0.02 seconds without. Its cache falls from 103MB to 8MB, the cgroup counts 202 times over the limit, and nothing is killed.
- Memory the program allocates has nowhere to go without swap, so past a soft limit its processes slow to a crawl. For example, a shell holding 40MB against a soft limit of 16MB makes no progress in 2 seconds of machine time. The cgroup counts 27,935 times over the limit and kills nothing, and the shell goes on once the limit is lifted:
1 | machine containers app memory db 16m --soft |
machine containers app shows each container’s limits in its LIMITS column,
such as memory 64MB soft 48MB.
Wrong data
machine mem writes a program’s memory with the machine
stopped, ^= flips bits of a value, as C’s ^= does, and --fill writes a
pattern over a span. Stop the program where the data is live first, at a
breakpoint or a run --until. The names below stand for the program’s own
variables and function:
1 | machine break app add 412 process_record |
The answer names the bits, the address and the value before and after. Here
optind, a variable of the C library a shell of a container maps, is named
by its file, as machine symbols lists it, since the shell has one too:
1 | machine mem app 628 '/lib/libc.so.6:optind^=0x1' |
1 | ✔ Flipped 0x1 at 0x7fe1a2385248 <optind> of tid 628: 0x00000001 became 0x00000000 |
Each write is recorded with the run, so a replay corrupts the same value at the same point, and a fork from before it runs the program unharmed.
Resource exhaustion
A program runs out of a resource in two ways, and Machine tests both:
- The resource runs out. A Linux limit on one container makes its processes meet the limit, as they would on a full production host. Use it to see how a service behaves as a whole when it hits the limit. Memory, processes and CPU are shared by the container’s processes. The open-file limit is each process’s own count.
- One call fails. A system call fault makes one program get the error the resource gives, at the call chosen, while everything else runs on. Use it to test one error path, exactly and repeatably.
| Resource | Run it out | Fail one call | The error |
|---|---|---|---|
| CPU | containers throttle db --cpus 0.1 |
none: everything slows | |
| Memory | containers memory db 64m |
inject add function malloc --return 0 |
SIGKILL, or malloc returns NULL |
| Processes | containers pids db 50 |
inject add syscall clone EAGAIN |
EAGAIN |
| Open files | containers files api 64 |
inject add syscall openat EMFILE |
EMFILE |
| Ephemeral ports | containers ports api 40000-40009 |
inject add syscall connect EADDRNOTAVAIL |
EADDRNOTAVAIL |
| Disk space, inodes | fill a volume of its own | inject add syscall write ENOSPC |
ENOSPC |
A fault that fails one call also takes --in, the program it fails, as
One failed call shows. Memory faults
covers memory. The rest of this section
covers the other resources.
| You want to know | Command | Example |
|---|---|---|
| Does a service survive when it cannot start a process? | containers pids db 10 |
Processes |
| Does a program cope when it runs out of file descriptors? | containers files api 32 |
Open files |
| Does a client cope when it runs out of local ports, or churns through them? | containers ports api 40000-40001 |
Ephemeral ports |
| Does the system cope with one slow service? | containers throttle db --cpus 0.1 |
CPU |
| Does a service that crashes on a limit recover once the limit is lifted? | containers memory hog 64m, then memory hog 0 |
A service that crashes on its limit |
| Does a service survive its disk filling up? | a volume of 64MB, filled with dd |
A full disk |
| Does a program report a failed fork, a full disk or no inodes? | inject add syscall clone EAGAIN, write ENOSPC, mkdir ENOSPC |
One failed call |
Limits on a container
machine containers sets each limit on one container. The limit holds when
the container starts again, until it is lifted, so a service that crashes on
the limit and restarts meets it again. Every start has the limit before the
container’s program runs its first instruction. A limit set on a stopped
container applies when it starts. A container that Compose recreates starts
without it. Use --for 30s to set the previous limit back after 30 seconds
of machine time, on whichever start of the container runs then. A disk built
before the guest kept container limits is refused, with a hint to build the
disk again.
A limit of 0 sets back the container’s own: what its Compose file gives it,
such as mem_limit or pids_limit, or no limit. A container with
mem_limit: 512m and memory api 128m goes back to 512MB on memory api 0,
running or stopped.
After pids db 10, machine containers app shows the limit in the LIMITS
column:
1 | machine containers app pids db 10 |
1 | SERVICE CONTAINER ID STATE IMAGE ADDRESS LINK LIMITS |
The other limits, and lifting one:
1 | machine containers app pids db 20 # 20 processes and threads |
Each limit is set by a command the run records, so a replay sets it at the same point, the restarts included.
A service that crashes on its limit
A service that dies of a limit and restarts by its Compose file’s
restart: policy meets the limit again, and the run goes on. Here hog
grows to 100MB five seconds after it starts, under restart: always:
1 | services: |
1 | machine containers app memory hog 64m |
The out-of-memory killer takes it each time it grows, 3 times in 40 seconds of machine time, and the console says each time:
1 | === hog exited by itself: code 137. Its restart policy starts it again === |
memory hog 0 while it restarts lets it grow, and it prints grew. A
service with no restart: policy that exits by itself ends the run, with
its code.
Processes
pids limits the processes and threads of a container. Past the limit, a
fork or a new thread fails with EAGAIN, and the processes already running
go on. Here a shell in db starts background jobs under a limit of 10:
1 | machine containers app pids db 10 |
1 | sh: can't fork: Resource temporarily unavailable |
After pids db 0, running that command again starts all 20.
Open files
files limits the files each process of a container may hold open.
Sockets and pipes count as files. Past the limit, an open, a socket or
an accept fails with EMFILE, and files already open stay open.
The limit is set on the processes running now, and the processes they start
take it. A process that docker exec starts, machine exec --container
included, takes the container’s own limit. So set the limit while the
program runs. Here a shell in api was already running when the limit was
set:
1 | machine exec app start --container api -- sh -c \ |
The shell prints 32. Then tail opens the files f1 to f100, in the
order f1, f10 to f19, f100, f2. It holds 29 of them and its
standard input, output and error, 32 files, and every file after that fails:
1 | tail: can't open 'f35': Too many open files |
A later machine exec --container api -- sh -c 'ulimit -n' prints 1024,
because a process docker exec starts does not take the limit set on the
processes already running. It takes the container’s own limit. files api 0 sets
that own limit back, both its soft and its hard number.
Ephemeral ports
ports narrows the range of local ports a container’s connections take
from. A connection is told apart by its local and remote address and port,
so connections to different servers can share a local port. The range is
not a count of connections: it runs out for one destination at a time. Here
db runs a server on port 9000, and api connects to
db’s address from the listing above, 172.18.0.4, with 2 ports:
1 | machine exec app start --container db -- tcpsvd 0 9000 cat |
1 | telnet: can't connect to remote host (172.18.0.4): Cannot assign requested address |
Two connections take both ports, and the third fails with EADDRNOTAVAIL. Four things follow from how the kernel gives out ports:
- A closed connection keeps its port for 60 seconds. Its side stays in
TIME_WAIT. Short connections in a loop therefore run out of a small range
even though no more than one is open at a time. Right after two
connections close,
netstat -tanshows both ports in TIME_WAIT, and 65 seconds later neither. - Name lookups need a port too. A DNS query takes a UDP port from the
same range. With both ports held, a client connecting to
dbby name failed withtelnet: bad address 'db', before it tried to connect. - A program that binds port 0 before it connects gets EADDRINUSE.
BusyBox
ncdoes this:nc: bind: Address already in use. - A program that binds port 0 uses an even count of the range. Of a
range of 3 ports,
ncused 2.
The range belongs to the container’s network. The command refuses a container on the machine’s network, whose range is every process’s. It also refuses a container on another container’s network, with a hint to set the range on the container whose network it is:
1 | ✗ Not set: sidecar is on the network of compose-db-1. |
A full disk
A container’s own files are on the machine’s disk, shared by every container, so filling them fills the machine. A volume that is a filesystem of its own fills alone: only the services that mount it meet the full disk. A Compose file makes one in two ways:
| Volume | Full as a disk is | After a restart | Costs |
|---|---|---|---|
A volume named in x-machine, with its size |
yes: an ext4 filesystem on a block device, with a journal, metadata and fsync |
its files are kept | room on the disk, up to its size |
A tmpfs volume, driver_opts with size= |
no: in memory, where fsync does nothing and no write is delayed |
its files are gone | the machine’s memory, up to its size |
Use the x-machine volume to see what a service does on a full disk. A
tmpfs volume answers whether it handles ENOSPC on a write, with no change to
how the disk is built. Here db has a volume of 64MB at /data:
1 | services: |
Filling it with one file runs it out of space, and many empty files run it out of inodes:
1 | machine exec app run --container db -- dd if=/dev/zero of=/data/fill bs=1M |
The fill stops at 54.7MB of the filesystem’s 57MB, and the write after it stores one block of 10 and fails. The files stop at the 16,373rd, with blocks still free:
1 | sh: can't create f16372: No space left on device |
Meanwhile db writes its own files outside /data, api writes its files,
and the machine’s disk is untouched. Deleting the files ends the full disk.
A tmpfs volume of 40MB stops a dd at 40MB with No space left on device,
and is empty after its container restarts.
Filling the machine’s own disk instead makes every container meet it, and
the container runtime with them: machine exec --container then fails with
OCI runtime exec failed. machine exec without --container still runs.
CPU
throttle gives a container a share of a CPU. Nothing fails: every process
of the container runs slower. For example, a loop in db that counts to
200,000 takes 4.58 seconds of machine time, and 46.80 seconds with
--cpus 0.1:
1 | machine containers app throttle db --cpus 0.1 |
One failed call
A system call fault gives one program the error a
resource gives, at the call chosen. Nothing runs out, and every other
program runs on. Each fault below was added, then the program after it was
run, then the fault was deleted with machine inject app clear. The
programs are BusyBox’s, in the containers of the
ports example, and tail reads the files of the
open-files example. worker.sh starts 3 background jobs and
waits for them, and machine put copies it into db:
1 |
|
1 | machine put app ./worker.sh /tmp/worker.sh --container db --mode 755 |
Each program reported its error:
1 | /tmp/worker.sh: line 0: can't fork: Resource temporarily unavailable |
fork in the C library is the clone system call. A script run by its own
path, /tmp/worker.sh, has its file name as its comm, so the fault takes
that script and no other shell. Running out of inodes gives ENOSPC from the
call that creates a file or a directory, as running out of space does.
A count of calls includes the program’s start. The dynamic loader opens the
program’s libraries with openat before main runs: 12 calls for tail
here, of 20 in all for 8 files. --after 9 made the loader fail with
error while loading shared libraries: libm.so.6, and --after 14 failed
the third file. Read MATCHED in machine inject app to
count a program’s calls, or use --prob 0.01 to fail calls at random.
Kernel faults
The other subsystems fail work inside the kernel, after a system call has started, where a syscall fault never reaches. They take no errors: each fails the way the kernel would.
1 | machine inject app add disk --times 1 --in /usr/bin/app # one disk request fails with EIO |
Five things decide whether you see the failure:
- Who sends a disk request.
--innames the task that sends the request, not the program the data belongs to. Your program sends its reads, itsO_DIRECTwrites and the writesfsyncforces, so--in /usr/bin/appfails those. Ordinary writes wait in memory and the kernel’s flusher writes them later, so only--in allreaches them. To fail a program’s writes, name its files with--fileand use--in all, as Disk faults does. - A program’s first moments. When a program starts, the kernel copies its
arguments in and allocates for it. A
usercopyor allocation fault that fires then kills the program before your code runs. Use--after, or add the fault once the program is up. - The kernel copes with some failures. A failed page allocation is often
retried, and the program never sees it.
MATCHEDandFIREDin the list still show that the fault fired. - The kernel reads a page more than once. A buffered read goes out as
readahead, and when that fails the kernel reads the pages again before it
gives up. How many requests it makes depends on how it splits the read: for
a file of 8KB,
catread the file fine with--times 1and--times 2, and gotI/O errorwith--times 3, all 3 of its requests failed. Leave out--timesfor the program to getEIO. AnO_DIRECTread fails on its first request. disk-timeouthangs. The request is lost, and the task waiting for it stays stuck until the machine stops. This is a hung device: no timeout fires and no reset follows. That is the point: you find out what else in the program notices. Name your program in--in: with--in allthe lost request may be the kernel’s own, and whatever waits on that disk hangs with it.
Disk faults
The faults above are errors: your program is told something failed. Real disks
also fail silently. A write reports success and the data is wrong, half there,
or not there at all. disk-corrupt, disk-torn and disk-lost are those
failures. The request succeeds, and the damage is found later, or never.
| You want to know | Fault |
|---|---|
| Does the database notice a bad block when it reads its data files? | disk-corrupt --file '/var/lib/db/*/data.bin' --in all |
| Does the program trust bytes it read without a checksum? | disk-corrupt --op read --file /srv/app/state.json --in all |
| Does recovery handle a log record only half written? | disk-torn --file /var/lib/db/wal --times 1 |
| Does it notice a committed write that never arrived? | disk-lost --file /var/lib/db/wal --after 10 --times 1 |
| Does the filesystem survive damage to its own metadata? | disk-corrupt --sectors LO:HI, the sectors of its inode table or journal |
1 | machine inject app add disk-corrupt --file '/var/lib/db/*/data.bin' --op write --bytes 8 --times 1 --in all |
--fileis a pattern.*matches within a directory and**across directories. Programs often name their files as they run, and a pattern names them before they exist. A file the program already holds open or maps, such as a running database’s WAL, matches too. A file inside a container is named by its path in the container.--in allwith--file. Ordinary writes reach the disk from the kernel’s flusher, not from your program, so--in allis what reaches them.--filekeeps the fault on your file.- The page cache hides a damaged write. The kernel keeps the bytes your
program wrote in memory, so the program reads them back clean until they are
read from the disk again: with
O_DIRECT, afterecho 3 > /proc/sys/vm/drop_caches, or in a branch forked later. A damaged read is the other way round: the bad bytes go into the page cache, and every reader sees them until the kernel drops the page. - A tear is whole sectors. Machine tears a write at sector granularity: each 512-byte sector is written whole or not at all, so a write of one sector does not tear. A torn write keeps at least one sector of the new data and loses at least one. Real disks differ in what they write atomically; this is the model Machine applies.
--sectorsaims below the files. It names 512-byte sectors of the whole disk, so it reaches what--filecannot: the filesystem’s inodes, bitmaps and journal. The filesystem’s own tools say where those are, such asdumpe2fsanddebugfsfor ext4, in blocks of its partition. With 4 KiB blocks, block N starts at sector 8 × N plus the partition’s first sector, which/sys/class/block/vda1/startholds, or 0 for a filesystem on the whole disk. Or stop at a write withcatch add io, below, and read its sectors off the stop.disk-lostis one write, not a crash. The lost write reports success, and the writes after it still land. A power cut loses every write since the last flush, and the disk faults do not model one. For that, usemachine containers crash: it loses the writes a service’s volumes had not synced.
A replay damages the same bytes, so a bug you find this way is found again on every replay of the run.
A worked example: ClickHouse. ClickHouse writes each part of a table into a
directory it names as it goes, such as all_12_12_0, and checks the part’s
checksums when it reads it back. To damage one column file of the next part
written and see what ClickHouse does:
1 | machine inject app add disk-corrupt --file '/var/lib/clickhouse/data/db/t/*/data.bin' \ |
To damage one part you have already seen, find its name first on a branch, and keep your run where it is:
1 | machine fork app --name peek |
Then add the fault on app with the exact path. The run is deterministic, so it
writes the same part. An insert’s part name holds; a merge’s can change, because
any fault changes how the machine runs a little. A pattern covers both.
Faults that cycle
Real outages come in bursts: a disk that is flaky for two seconds, a network
that drops for five. --prob fails calls one at a time and independently,
which rarely exercises retries, backoff or a circuit breaker. --on and
--off make a fault active for a stretch of machine time, rest for another,
and repeat:
1 | machine inject app add disk --file /var/lib/db/data --on 2s --off 5s --in all |
The first fails the data file’s disk requests for 2 seconds in every 7. The
cycle starts when the fault is added, active first, on the machine’s own
clock, so a slow host changes nothing and a replay crosses every edge at the
same instruction. The machine’s clock is not the date the guest reads, so a
machine clock change does not move a window. While a fault rests it matches
no call and counts none, so
--after and --times count only the calls in its active windows. Both
options are needed, each a whole number of milliseconds from 1ms to a day.
The listing says which window each fault is in and how long it has left. After the second command it reads:
1 | # INJECT ERRORS DELAY IN FIRES PROB SEED MATCHED FIRED |
machine inject app later shows the window
moving: now on, 996ms left one second in, and now off, 2496ms left at two
and a half seconds.
A shell loop that opens a file twice a second under the second command sees
4 failures, then 6 successes, over and over: 2 seconds failing, 3 seconds
working. Each window has a fixed length. To try outages of other
lengths, fork branches from one checkpoint and give each its own --on and
--off.
Stopping at a fault
To look at the program at the moment a fault fires, add a catchpoint for it:
1 | machine catch app add fault openat |
The run stops where the fault fires, before a system call’s wait and error, and
says which: Stopped at catchpoint 1: fault openat EACCES, in tid 229, or
fault disk, in tid 77. See Debugging for catchpoints.
Stopping at a file’s I/O
An io catchpoint stops the run at a disk request, with no fault. --file
takes the same patterns as a disk fault, and --op takes read or write:
1 | machine catch app add io --file '/var/lib/clickhouse/data/db/t/*/data.bin' --op write |
The run stops as the request is sent, before any disk fault decides on it, and
names the request: Stopped at catchpoint 1: io write 1058144:1058151, in tid 266. A fault added there decides on that same request. Name it by its file:
1 | machine inject app add disk-lost --file '/var/lib/clickhouse/data/db/t/*/data.bin' --times 1 --in all |
Or by its sectors and its task, which name this request and no other:
1 | machine inject app add disk-corrupt --sectors 1058144:1058151 --times 1 --in tid=266 |
An ordinary write is sent by the kernel’s flusher, so the stop names a kernel
task such as kworker/u4:1. --in tid=N names that task as well as any
other.
Changing faults
1 | machine inject app delete 2 # by its number in the list |
At the control prompt it is /inject,
with the same words, and /tasks exist checks first:
1 | /tasks exist /usr/bin/app |
Faults and plans work together. Add the faults, then run the plan: the faults apply to every stretch of execution, whatever the plan chooses. To change them partway, stop, change them, and run on.
From Python, run.inject.add(...) takes the same subsystem, errors and
options, with in_= for --in, and run.inject() lists the faults with
their counts. The SDK’s Faults has an example.
What a fault misses
- A C function is not always the system call of its name, and C libraries
differ. glibc’s
open()callsopenat, musl’s callsopen, and glibc’sstat()callsnewfstatat. Name both,open,openat, to cover either.machine syscalls applists the names the kernel uses. - Give
--seedin a script that records again. Without it each run draws its own seed. A replay needs none: the run keeps the fault as it was added. - 64-bit system calls only. A 32-bit call, made with
int 0x80, never fails. - Calls the vDSO answers never reach the kernel.
clock_gettime,gettimeofday,timeandgetcpureturn from the vDSO without a system call, so a fault on them matches nothing and MATCHED stays 0. - 16 faults, and 30 errors in one. Delete a fault to add another, and use
--pickto take 30 or fewer ofall. - A fault that fired is not a failure the program saw. The kernel retries
an allocation, rereads a page and recovers a filesystem, and a program
retries a call.
FIREDsays the fault fired, andMATCHED0 says it never met a call, so nothing was tested. Whether the program failed is a question for the program: check its output, its exit status or its data after the fault, as well as the counts. - A fault matches the system call, whose call it is and the count of calls. It does not look at the arguments, so “openat of /etc/passwd” is not a fault. Stop at the call and fail that one instead.
Container faults
A project run as containers is a small cluster, and each service is a node.
machine containers puts a fault on one
container while the others keep running:
1 | machine containers app stop db # SIGTERM, then SIGKILL after 10s |
| You want to know | Fault |
|---|---|
| Does the cluster keep serving while one node restarts cleanly? | stop db --restart 10s |
| Does a client retry when its server dies mid-request? | kill db --restart 5s |
| Does the database keep what it committed through a power cut? | crash db --restart 0 |
| Do the other services notice a node that hangs without dying? | pause db --for 30s |
| Does the system cope with one slow node? | throttle db --cpus 0.1 --for 1m |
| What happens when a node runs out of memory? | memory db 64m |
| What happens when a node cannot start a process, open a file or connect? | pids db 20, files api 64, ports api 40000-40009; see Resource exhaustion |
| Does the service keep state it never put in a volume? | kill db --restart 0 --fresh |
A container you stop, kill or crash stays down. --restart 5s brings it back
after 5 seconds of machine time, and --restart 0 at once. The other
containers keep running through the outage:
1 | machine containers app kill db --restart 5s |
1 | ✔ Killed compose-db-1 (db) |
A restart is a reboot: the container comes back with every file it had.
--fresh replaces it instead, as a failed node is replaced by a new one from
the same image. The files it wrote outside its volumes are gone, and its
volumes are kept. Use it to find state a service keeps in its own files when it
should keep it in a volume.
A limit holds when the container starts again, as a limit in its Compose file
would: a service the out-of-memory killer stops comes back to the same limit.
Every start has the limit before the container’s program runs. Use
throttle db --cpus 0 to set back the container’s own limit, its Compose
file’s or none, or --for 30s to restore the previous
limit after 30 seconds of machine time, on whichever start of the container
runs then. A limit set on a stopped container applies when it starts. A
container that Compose recreates, or start --fresh replaces, is a new
container and starts without the limit.
memory limits the memory a container’s processes may use, as its cgroup’s
memory.max. A limit below what they use makes the guest kernel reclaim
memory, and its out-of-memory killer kill a process of the container when it
cannot. A process killed that way exits 137, by SIGKILL:
1 | machine containers app memory db 32m |
memory takes --for, and its limit holds through restarts, as
throttle‘s does. machine containers app lists each container’s limits in its
LIMITS column, cpus 0.1 memory 64MB or - for none. A disk built before
container limits is refused, with a hint to build it again.
An out-of-memory kill is part of the run: a replay kills the same process at
the same point.
Each fault is recorded with the run, so a replay faults the same container at
the same point. Put one where the run is stopped, at a breakpoint, a catchpoint
or a run --until, to fault a service at the moment that matters.
A crash
kill loses nothing the program wrote: the kernel inside the machine keeps
running and writes it out later. crash is a power cut of the service’s node.
Its volumes keep what was synced and lose the writes still in memory. When the
service starts again, each volume replays its journal and the service runs its
own recovery. That finds a commit acknowledged before its data was synced, a
rename whose directory was never synced, and a log record that survived while
the data it describes did not.
A crash cuts the volumes the project names in x-machine; see
Volumes a crash cuts. The cut is when the
volumes stop writing to the disk, just after the container is frozen; a write
the kernel finishes in between is kept, as one a disk finished before the power
went. A volume another running container also mounts is refused, since cutting
it would cut that container too.
1 | machine containers app crash db /var/lib/postgresql/data --restart 0 # that volume alone |
1 | ✔ Crashed compose-db-1 (db) |
The volume is named as Docker names it, after the Compose project.
A crash test, start to end. Stop where the database syncs, keep the run there, and crash a branch of it. The branch loses what the database had not synced yet; the query tells you whether it had already told a client the data was safe:
1 | machine catch app add syscall fsync fdatasync --in comm=postgres # where it syncs |
The run is deterministic, so the branch crashes at exactly that sync every time. Run on to the next sync and fork again to cut at another point.
From Python, run.containers.crash("db", restart="0") and the other verbs
take the same options as keywords. The SDK’s Container faults
has an example.
Clock faults
Software reads two kinds of time. Elapsed time, CLOCK_MONOTONIC and the TSC,
is the machine’s own, and Machine time
controls it. The date, CLOCK_REALTIME, is what date, time() and
gettimeofday return, and on a real fleet it jumps, drifts and disagrees
between hosts. machine clock changes the date
inside the machine, for the whole guest or for one container, and leaves
elapsed time alone.
| Command | What the date does |
|---|---|
jump +5s |
steps forward or back, as an NTP step or date -s does |
set 2027-03-28T00:59:50Z |
goes to that date |
drift +200ppm --container db |
gains 200 parts per million of machine time, 0.72 seconds an hour |
jump +4s --slew 30s --container db |
moves 4 seconds over 30 seconds of machine time, with no step |
jitter -200ms:200ms --every 10m |
is set to a drawn offset every 10 minutes of machine time |
leap insert |
repeats a second, 23:59:60, at the next UTC midnight |
reset |
returns to machine time, for the guest and every container |
The guest kernel makes each change, so a change takes effect when the run next executes. The run records every change: a replay makes it at the same instruction, a checkpoint keeps the date, and a branch starts with its parent’s date.
The guest’s date
jump, set and reset change the date every process in the guest reads,
in every container. The step is a real step of the guest kernel’s clock: a
timer armed for an absolute date sees it, and a timerfd with
TFD_TIMER_CANCEL_ON_SET is woken, which is how systemd and cron notice a
step.
1 | machine clock app jump +59m50s |
The answer shows the date as it stands and the change waiting for the run:
1 | date: 2026-03-01T00:01:29.614903749Z |
Steps add up: jump +5s and then jump -2s leave the date 3 seconds ahead.
set puts the date at a value, whatever steps came before it. Machine refuses
a date before 1970, or one on or after 2232-04-18T23:47:16Z, because the
kernel cannot set either.
A container’s date
With --container, a change moves one container’s date alone, as an offset
from the guest’s date. Every process of the container reads it, including
those docker exec and machine exec --container start. The other
containers keep the guest’s date.
1 | machine clock app jump -6m --container client |
The client now reads a date 6 minutes behind the server, so a certificate the server issued a minute ago is not valid yet for the client.
The offset reaches every way a program reads the date. clock_gettime stays
as fast as it is outside the machine, because the container’s processes still
read the date in the vDSO, without a system call. System calls, POSIX timers,
timerfd and clock_nanosleep agree with the container’s date, and so do the
futex waits behind pthread_cond_timedwait and sem_timedwait and message
queue timeouts. A restarted container keeps its offset, while a container
created again starts on the guest’s date.
--container takes a container’s id, its name or its Compose service. Machine
finds the container without running the machine, so a clock change made at a
breakpoint leaves the run at that breakpoint. Prefer the id: Machine knows it
as soon as Docker creates the container, and learns the name shortly after the
container starts. machine containers app lists both.
A stopped container that has an offset keeps it, and takes it again when it
starts. machine clock app reset --container db returns it to the guest’s
date, stopped or running. A container Docker removes, with docker rm or a
--fresh restart, loses its offset, and the container that replaces it starts
on the guest’s date. At most 8 containers have dates of their own at once;
the next is refused until one is reset or removed.
Drift and slew
A clock that runs fast or slow is the fault a jump cannot fake: nodes do not
jump apart, they wander apart until an assumption breaks. drift makes one
container’s date gain or lose time at a rate of machine time:
1 | machine clock app drift +5000ppm --container node2 # 0.5% fast: 3s in 10 minutes |
No read sees a jump, and a timer armed in the container agrees with its drifting date: it fires when the container’s own date reaches the deadline, never before. At +100000ppm, a timerfd armed for 2 seconds ahead by the container’s date fired after 1.821 seconds of machine time, which is 2 seconds of a clock running 10% fast.
--slew moves a container’s date to a new value over a time instead of in one
step, as NTP slews a small correction:
1 | machine clock app jump +4s --for 2m --container db # off and back, two steps |
A slew runs until it arrives. A later change to the same container, a drift
included, stops it where it has got to, so set a drift before you start a
slew. A slew moves the date at most half a second for each second of machine
time, so jump +4s --slew 8s is the fastest slew of 4 seconds. Machine
refuses a shorter one:
1 | ✗ A slew of +4s takes 8s at the least. |
The guest’s date has no drift. To make the whole machine’s time run fast or slow, use the clock rate.
Jitter
An NTP-disciplined fleet is never exactly in sync: each host’s offset wobbles
within a bound. jitter sets the date to a value drawn around it each period:
1 | machine clock app jitter -200ms:200ms --every 10m --container api |
Every 10 minutes of machine time, api’s offset is set to a value from -200ms
to 200ms around the date without the jitter. The first step is taken when the
guest takes the command. A weight in brackets draws a choice more often:
-1s[1],0[8],1s[1] is in sync most of the time and a second off now and then.
A jitter takes 4 choices at most, each a whole number of microseconds.
The draws come from --seed, or from a seed drawn at random and shown. Give
that seed with --seed to step the date the same way again. A replay steps by
the same amounts at the same points, and a branch draws on from where its
parent was. A jump or a set moves the date under the jitter, which goes on
around the new date. machine clock app jitter off stops the jitter and
returns the date to where it would be without it.
Leap seconds
In 2012 a leap second livelocked Linux futexes, and services across the
industry went down with it. leap insert makes the guest kernel repeat a
second, 23:59:60, at the next UTC midnight of the guest’s date, and
leap delete makes it skip 23:59:59:
1 | machine clock app set 2026-12-31T23:59:50Z |
After the 20 seconds the date is a second behind where it would have been
without the leap. With leap delete it is a second ahead. A leap second moves the guest’s date and every container
with it, so leap takes no --container. leap off cancels a leap still
pending, and so does reset.
Undoing a change after a time
--for 2m takes a change back after 2 minutes of machine time, at the same
instruction in every replay. jump +4s --for 2m is a step forward and, 2
minutes later, a step back: the NTP correction that follows a host running
with a bad clock. jump, set, drift and jitter take --for.
The undo takes back the command’s own change. What else moved the date meanwhile stays: a drift’s accrual, a leap second, or another program’s step. A later change cancels an undo still pending.
Reading the date
machine clock app reads the date without running the machine. It shows the
guest’s date and its offset from machine time, then each container whose date
is off the guest’s, with its offset, drift, slew and jitter. To see the date a
program reads, ask from inside its container, which runs the machine:
1 | machine exec app run --container db -- date -u |
The reading looks like this:
1 | date: 2026-12-31T23:59:54.977514123Z |
--container db shows one container, and machine -o json clock app gives
every field.
A change is taken when the run next executes, so the command that asks for it cannot say whether the guest kernel accepts it. Read the clock after the run that takes it. The reading names each change of the last batch the guest kernel refused, as you typed it, and why:
1 | machine clock app set 1970-01-01T00:00:01Z |
1 | date: 2026-03-01T00:00:40.556767815Z |
The jitter applied and the set did not. A refused change changes nothing.
Clock fault use cases
Each block below is a question a clock fault answers, and the commands that
ask it. Run the machine afterwards, with machine run app --within 2m or with
an --until the program prints, and read what it did.
1 | # Does the program refresh a token before it expires? |
How much skew the system tolerates
Fork the same checkpoint into branches, one skew each, and run each to the same point. Every branch starts from the identical machine, so the skew is the only difference:
1 | machine checkpoint app steady |
A jump inside one request
Most time bugs are check-then-use: the process checks a token’s expiry, the clock steps, and it uses what it checked. Stop the run where the check is done, jump, and go on:
1 | machine containers app # auth is 3f2a91c0d4e7 |
The jump finds the container by its id without running the machine, so it
lands while the run is stopped at check_expiry.
A step back reorders events
A database that stamps rows with the time they were written sorts them by that stamp. Step its container’s date back between two writes, and the second row sorts before the first:
1 | machine exec app run --container db -- psql -U postgres -tAc "create table ev(id serial, at timestamptz default clock_timestamp())" |
1 | 2|2026-03-01 00:50:42.230825+00 |
Any code that takes the newest stamp as the latest event, such as
last-write-wins or a cursor over at, now picks row 1.
Because every clock change is part of the run’s recording, a failure found
under skew comes back when you replay the recording; there is no timing to
reconstruct. Investigate it as any other failure, with
rewind, diff and the debugger.
What a clock change misses
- File timestamps. The kernel stamps files with its own clock, so
makein a container behind the guest reports clock skew. - Socket timestamps.
SO_TIMESTAMPandSIOCGSTAMPcarry the guest’s date, sopingin a skewed container reports the offset as round trip time. - Timers already armed. A container’s change does not move a deadline
armed before it; a timerfd with
TFD_TIMER_CANCEL_ON_SETis woken so the program can re-arm. - A container that sets the date. A container given
CAP_SYS_TIMEthat callssettimeofdaysets the guest’s date, as it would set a host’s; its own offset stays.
Network faults
A distributed system fails at its links: one that is slow, one that loses
packets, one that stalls and then delivers everything at once, a node cut
off from one peer while it still reaches the others, a connection reset in
the middle of a transaction. machine net puts
those faults on the links between the containers of a run.
Each container has one link: everything it sends. A fault on the whole link
acts on every packet the container sends. A match narrows a fault to some of
those packets: the ones to one peer, of one protocol, of one port, or the
TCP SYNs that open a connection. Each match has faults, a seed and counts of
its own. Every net command runs the machine until the guest has made the
change, so the run moves forward.
1 | machine net app # each link and its faults |
Use cases
Each command follows machine net <run>.
| You want to know | Command |
|---|---|
| Does a client time out and retry when its server answers slowly? | delay db 2s |
| Does the protocol survive lost and duplicated packets? | loss db 10%, then duplicate db 5% |
| Does a leader election hold when one node is cut off? | partition node1 node2, then partition node1 node3 |
| What happens when only one direction fails? | partition db api --one-way |
| Does replication keep up over a slow link? | rate primary 1mbit --to replica |
| Does the protocol recover when the network damages packets? | corrupt db 1% |
| Does the client reconnect, and retry safely, when its connection is reset mid-transaction? | reset api --to db |
| What happens when a server’s responses to one client are lost, and nothing else? | loss db 100% --to api --port 5432 |
| Do messages that arrive late, after a leader change or a timeout, do harm? | hold node1 --for 10s |
| Does the cluster recover when the whole network stalls and comes back at once? | hold, run --within 3s, release |
| What happens if the network stalls at one exact point? | run --until <point>, then hold |
| Do established connections keep working while new ones cannot open? | loss api 100% --to db --syn |
| Does the cluster settle when a link keeps going down and coming back? | a loop of partition --for, below |
| What happens when a node cannot resolve a peer’s address? | loss db 100% --proto arp --to api, then arp flush api and arp flush db |
| What happens when a node holds a wrong MAC for a peer? | arp set api db 02:42:ac:12:00:99 |
| Did the fault drop the packets, or did the program never send them? | machine net app: each fault’s SENT and DROPPED, below |
| Which exact packet did the fault drop, and why? | loss db 30% --log, then read the decision log, below |
| What exactly went between two containers, to open in Wireshark? | tcpdump in the machine, in Capturing the packets, below |
Other commands cover the failures around a network fault:
- A refused connection, rather than one that times out:
machine inject app add syscall connect ECONNREFUSED --in /usr/bin/app, in Fault injection. - A node that freezes rather than one that is cut off:
machine containers app pause db --for 5s, in Container faults. - Clocks that disagree between nodes:
machine clock app jump +2s --container db, in Clock faults. - A cluster split into two groups: one
partitionfor each pair of containers across the split.
One link or all of them
Without a match, a fault is on everything the container sends. With --to,
it is on what the container sends to that one peer, and its other links stay
as they are:
1 | machine net app delay db 100ms --to api |
1 | SERVICE CONTAINER TO MATCH FAULTS SEED SENT DROPPED QUEUED |
A ping from api to db now takes 103ms, because db‘s replies to api
wait 100ms. A ping from nest to db still takes 0.04ms.
Packets a match takes get only the match’s faults, not the whole link’s:
after loss db 10% and then delay db 100ms --to api, what db sends to
api is delayed and none of it is lost. A packet two matches take gets the
faults of the one listed first. Each verb changes the fault it names and
keeps the others, so delay and then loss give a link that is slow and
lossy. A value of 0, or 0%, takes that fault away. A jitter also reorders
packets, since each one leaves at its own time.
Matching packets: protocol, port and SYN
--proto tcp, --proto udp or --proto icmp takes the packets of one
protocol. --port 5432 takes the packets with 5432 as their source or
destination port, so on a server it takes its responses, and on a client its
requests; without --proto it takes both TCP and UDP. --syn takes the
TCP SYNs that open a connection. The options combine with each other and
with --to, and a packet must satisfy all of them. Every verb that sets a
fault takes them:
1 | machine net app loss db 100% --to api --port 5432 # every packet db sends api on port 5432 |
The first loses db’s responses to api on that port, and its SYN-ACKs too, so
api can open no new connection there either. The last leaves every
connection api already has to db working: only the packets that open a new
one are lost. A port belongs to TCP and UDP and a SYN to TCP, so
--proto icmp --port 5432 and --proto udp --syn are refused. A container
holds at most 15 matches besides its whole link.
Partitions
partition cuts two containers apart: everything each sends the other is
lost, and both still reach the rest of the cluster. --one-way cuts what the
first sends the second, and leaves the reverse. --proto, --port and
--syn cut those packets alone:
1 | machine net app partition db nest |
With --for 10s, both directions come back together after 10 seconds of
machine time.
A link that keeps failing and recovering is a loop of partitions, each followed by a stretch of the run:
1 | for i in 1 2 3 4 5; do |
Holding packets, then releasing them
hold keeps every packet the link sends, and release lets them all go at
once, in the order they were sent. Nothing is lost, so the peer receives
messages that were sent seconds ago: a stalled link that recovers, a
message that arrives after a leader change or after its sender timed out:
1 | machine net app hold node1 --to node2 # node1's messages to node2 wait |
--for 10s releases them after 10 seconds of machine time. A link holds 1000
full-size packets and drops the ones past them.
A held packet still counts against the send buffer of the socket that sent
it, about 208KB by net.core.wmem_default. Once that buffer is full, the
socket’s sends fail, or for TCP block, before the link’s queue is reached,
so one socket cannot fill a hold by itself. For example, when api sends db
1200 pings of 1400 bytes during a hold of db, db receives all 1200 and its
link holds 117 replies. The other 1084 replies fail in db’s kernel and count
as OutErrors in its /proc/net/snmp. With pings of 56 bytes, 321 replies
are held.
Without a container, hold holds every link at the instruction the
machine is stopped at, and all networking between the containers stops
there. release or heal without a container lets every packet go:
1 | machine net app hold # the whole cluster's network stops here |
The hold is part of the recording, so a replay holds at the same point, and a checkpoint taken during a hold keeps it. Run the machine to an exact point first, and the hold lands there:
1 | machine run app --until 'console~ready' # stop where the cluster is ready |
For example, with api pinging db every 100 milliseconds, db receives none of
api’s requests while the links are held and the fault is set. After the
release, db receives the 47 held and later requests, the new fault drops
all 47 replies, and the listing’s DROPPED reads 47. No packet crosses a
link under the old faults after the hold.
The guest’s programs and timers keep running during a hold. A retransmission or an application timeout can expire while the links are held. To hold for a time, run the machine for that time between the two commands:
1 | machine net app hold |
For example, with api pinging db every 250 milliseconds, the 3 replies sent before the hold arrive, none arrives in 3 seconds of hold, and all 20 arrive after the release. A container on another’s network, the machine’s or none has no link of its own and is left out, and so is a process outside every container. The listing names a container whose link could not be held.
Resetting connections
reset aborts TCP connections. The container’s side of each connection is
aborted, and the peer receives a TCP reset, as from a middlebox that dropped
the connection or a server that restarted. What a program sees depends on
the call it makes. On the aborted side, the first call after the data it
already received fails with ECONNABORTED; after that a read returns end of
file and a write fails with EPIPE. On the peer, the first call fails with
ECONNRESET, and the same follows.
Unlike a partition, nothing waits for a timeout:
1 | machine net app reset api --to db # api's connections to db |
A listening socket stays open, so the server takes new connections at once.
Address resolution
A container finds a peer’s MAC by ARP and keeps it in its neighbour table.
--proto arp puts a fault on the ARP frames a container sends, and with
--to on those about one peer: its requests for the peer’s address and its
replies to the peer. A fault on ARP acts when a container resolves a peer
again, so flush the tables to see it at once. With db’s replies lost, api
cannot resolve db, while nest still reaches it:
1 | machine net app loss db 100% --proto arp --to api |
machine net app arp shows and changes a container’s neighbour table:
1 | machine net app arp show api |
1 | INTERFACE ADDRESS PEER MAC STATE |
1 | machine net app arp set api db 02:42:ac:12:00:99 # api sends db's frames to a MAC nobody has |
An entry made by arp set is permanent, so ARP never corrects it: api
reaches db again only after arp delete. A stale entry is still used; at
the next packet the kernel checks it by ARP within seconds, unless the
connection’s own traffic confirms the peer first. The table is part of the
machine, so a replay makes the same changes. A container that docker starts
again begins with an empty table, as a host that restarts does.
The decision log
Run the machine for 10 milliseconds before reading a log just after a run stops: the last millisecond’s lines reach the run when the machine runs again.
--log writes each decision a fault makes to a log: which packet it took,
and whether it dropped it, delayed it, duplicated it, corrupted it or sent it
ahead. It answers which exact packet a fault dropped, which the counts and a
capture do not. --no-log with the same fault stops it:
1 | machine net app loss db 30% --to nest --seed 7 --log # the link's decisions are logged |
The guest kernel writes one line per decision, where the fault decides, to
/output/drop/.machine/net/<container>.log. The .machine folder holds the
files the machine itself writes, apart from a program’s own, and it is under
drop, so a run stores the log only when it keeps that output. A run that
keeps it from the start is read at any time:
1 | machine create app --disk app.img --keep-output /output/drop/.machine/net |
machine output prints the last 25 lines unless -n says otherwise; -n :
prints every line. The guest kernel writes the lines in batches, at most 1
millisecond of machine time after the first line of a batch. Lines decided
in the last millisecond before the machine stops reach the run when the
machine runs again, and a checkpoint keeps them: run the machine for a
moment, as machine run app --within 10ms, before reading the last
packets’ lines.
A run that does not keep the log still makes it, and replay brings it back. Fork a branch from a checkpoint before the stretch, keeping that output, and the branch replays the stretch and stores the same lines:
1 | machine checkpoint app before |
1 | 2026-03-01T00:01:31.691986Z #8 to compose-nest-1 pass ICMP 172.18.0.4 > 172.18.0.3 echo reply id 25 seq 0 len 84 |
Each line has the machine’s date, the packet’s number on its link, the
match, the decision and the packet with its IP length. The number counts
every packet the link carried since the container started, logged or not,
so a later stretch starts at a higher number: here the link had carried 7
packets before the ping. The 8 drops of the 20 lines are the 8 pings the
ping lost, and DROPPED in the listing reads 8. A replay from a checkpoint
writes the same log, byte for byte.
The decisions are pass, drop, delay with the time the fault drew,
ahead for a packet reorder sent before the delayed ones, and duplicate
and corrupt beside the first word. drop limit is a packet dropped
because the fault’s queue already held 1000 delayed packets, and drop hold
one dropped because a hold already held 1000 full-size packets; DROPPED
counts both. Every packet the link carries has its line and its number,
and a packet a hold drops has a second line, drop hold, under the number
of its first. A duplicated packet has two lines: the
original says duplicate, and the copy, logged first, says copy. A delay
is what the fault drew; the packet can leave later by the machine’s timer
slack, as a timer can. A container on two networks writes both links’
decisions to the one file.
Every decision is logged; none is dropped however fast the packets come. For
example, a 200MB TCP transfer under a logged fault decides 19,586 packets,
the log holds 19,586 lines, and the listing’s SENT and DROPPED add up to
them.
Each decision costs about 32 microseconds of machine time to write, and a decision is a packet as the kernel hands the link, whatever its size. A 50MB TCP transfer under a logged fault takes 0.66 seconds of machine time on average, against 0.47 seconds under the same fault without the log, because its packets are large. Small packets cost the same each, so a stream of them spends far more of its time on the log; 10,000 packets cost about 0.32 seconds. Log the stretch being examined, not a whole run. A decision is a packet’s, never a program message’s: see the limits below on TCP units and retransmission.
Seeds, replay and –for
Loss, jitter, duplicate, corrupt and reorder decide each packet by chance,
from the link’s seed. Give one with --seed, or the link draws one and shows
it. A delay or a rate alone draws nothing, and shows no seed. A link keeps
its seed through later changes. The run records every change, so a replay
delays and drops exactly the same packets, and so does a branch forked from a
checkpoint before them.
A change to a link starts its draws again from the seed: loss db 10% --seed 7, then delay db 50ms, loses packets from the start of seed 7’s
sequence again. Which packet corrupt changes, and which byte and bit
of it, follow the seed.
A checkpoint keeps a link’s faults: a rewind to it brings them back. A
container that docker starts again, after containers kill --restart or a
crash, gets its faults back, toward each peer at the address it has now.
--for 30s sets back what its command set after 30 seconds of machine time,
and leaves the rest of the link as it is then:
1 | machine net app loss api 100% --for 3s # everything api sends lost for 3 seconds |
A heal takes every fault off and releases what a hold keeps. A heal, and a
--for that ends, act on new packets: a packet the link already delays
still leaves at its time.
What a fault did
SENT, DROPPED and QUEUED count what each fault did with the packets
it took. They count from when the match, or the whole link, first got a
fault, and a later change to it keeps counting. SENT is the packets it
passed on, a duplicate counting once more. DROPPED is the packets it lost,
by loss, or past what a hold has room for. QUEUED is the packets it
holds now, delayed or held: neither sent nor dropped yet. After 50 pings
from api to db with loss db 30% --to api --seed 9, the row of that match:
1 | SERVICE CONTAINER TO MATCH FAULTS SEED SENT DROPPED QUEUED |
The ping prints 50 packets transmitted, 36 packets received, so the 14
db dropped are the 14 api lost. A hold shows what it holds as QUEUED until
the release. The counts are what the fault processed, packet by packet, not
a measure of its probability: over many packets 30% loss drops close to 30%
of them, and over a few the count can be anything. A container that docker
starts again counts from 0, and so does one a network is disconnected from.
The counts are packets as the fault decides them, the unit of the log’s
lines: a large TCP write that reaches the fault as one packet is one.
The counts are packets on the link, not sends by a program. A TCP write can
wait in the socket’s buffer, and the kernel retransmits without a system
call. machine catch add syscall sendto --in /usr/bin/app shows what a
program sends, and a capture shows the packets.
Capturing the packets
Every disk has tcpdump in the machine itself, at /usr/sbin/tcpdump, so a
capture needs nothing in the containers’ images. Write the capture under
/.machine/output/keep, and the run keeps it as its output, named
/output/keep/db.pcap.
machine containers lists what a capture needs: each container’s ADDRESS,
and its LINK, the machine’s interface at the other end of the
container’s own:
1 | machine containers app |
1 | SERVICE CONTAINER ID STATE IMAGE ADDRESS LINK |
To capture one container’s packets, capture on its LINK:
1 | machine exec app start -- tcpdump --immediate-mode -U -i veth45c3e2e -w /.machine/output/keep/db.pcap |
--immediate-mode and -U write each packet as it is captured: without
them, the packets tcpdump has not written yet when it stops are lost. The
stop waits for tcpdump to exit, so the file is whole before it is read.
Open db.pcap in Wireshark or read it with tcpdump on this computer.
On its link each packet db sends or receives appears once, as an Ethernet
frame, ARP included. Broadcasts from the other containers appear too, since
the bridge sends them to every link. A packet a fault on db dropped or still
holds is not there: the capture sees what db’s faults let through, and what
reaches db. The LINK changes when docker starts the container again, and
so do addresses from run to run, so read them in the run you capture.
--net db runs tcpdump on db’s own network instead, so its eth0 is db’s
link, and the name stays the same across restarts:
1 | machine exec app start --net db -- tcpdump --immediate-mode -U -i eth0 -w /.machine/output/keep/db.pcap |
It captures the same packets as a capture on the LINK. For example, with
db’s ICMP lost by loss db 100% --proto icmp, both show the pings reaching
db and none of db’s replies.
To capture every container at once, use -i any and filter by ADDRESS:
1 | machine exec app start -- tcpdump --immediate-mode -U -i any -w /.machine/output/keep/all.pcap host 172.18.0.2 |
With -i any, each packet between two containers appears twice: as it
arrives from the sender’s link, P, and as it leaves toward the receiver’s,
Out. The file records each packet’s interface by an index that means
nothing on this computer, so filter by address rather than by interface.
Finding the address and the link by hand
machine containers reads both from the machine; they can be read by hand
as well. Docker knows each container’s address by the container’s name:
1 | machine exec app run -- docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' compose-db-1 # 172.18.0.2 |
A container’s eth0 is one end of a pair, and the container’s iflink is
the number of the machine’s interface at the other end:
1 | machine exec app run --container db -- cat /sys/class/net/eth0/iflink # 6 |
A replay writes the same capture, byte for byte, with loss, delay and clock
jumps in effect. The timestamps are the machine’s date: clock jump of the
whole guest moves them, and a container’s offset does not. A link between
containers carries TCP segments larger than its MTU, since the kernel
segments them only where a real interface needs it.
What a network fault misses
Traffic inside one container, over its loopback, has no link to fault.
DNS between containers is answered on each container’s loopback, so no fault reaches it. Point the container’s resolver at an address that does not answer, and its lookups time out; write the file back to restore them:
1
2machine exec app run --container api --user root -- sh -c 'cp /etc/resolv.conf /etc/resolv.keep; echo nameserver 192.0.2.1 > /etc/resolv.conf'
machine exec app run --container api --user root -- sh -c 'cat /etc/resolv.keep > /etc/resolv.conf'Address resolution follows the whole link’s faults unless a match names
--proto arp: a match of another protocol takes IP packets alone. Withloss db 30%, db’s ARP requests are lost too, and right after a start, with nothing resolved yet, db can take seconds to reach a peer.A fault is on what a container sends. To fault what
dbreceives, put the fault on each container that sends to it, with--to db. To slow both directions between two containers, put a fault on each, aspartitiondoes.A container on another’s network, with
network_mode: service:db,hostornone, has no link of its own, and a fault on it is refused. Put the fault on the container whose network it uses.A process outside every container, on the machine itself, sends on no container’s link. What it sends to a container takes no fault, and only the container’s replies do.
A rate is per sender. Three containers each limited to 1mbit toward
dbgivedb3 megabits a second, not one link of 1 megabit.A damaged packet rarely reaches the program.
corruptchanges one bit anywhere in a packet’s frame, and the IP, TCP and UDP checksums drop nearly every damaged packet; TCP then sends it again. A bit in the Ethernet addresses is not caught by a checksum. No fault hands a damaged payload to the program.Losing exactly the Nth packet, bursts of loss, and delays other than a uniform jitter: a fault decides each packet on its own, by its chance.
IPv4 alone. Ports and SYNs are matched in packets without IP options, which Docker’s networks send.
The path’s MTU stays as Docker sets it.
Loss of TCP is per unit the kernel hands the link, not per wire segment. Between containers the kernel sends a large TCP write as one packet many times the MTU, segmented only where a real interface needs it: for example, most are 8244 bytes.
loss 1%drops 1% of those units, each as many segments as it holds, and TCP sends the whole unit again. The log’slenshows each unit’s size.
Debugging
There are two ways to debug a program in a machine. The machine’s own
debugger works from outside the guest: machine regs, mem, step, break
and the rest read and change a task while the machine is stopped, and they
work on any program, including one with no debugger on its disk. GDB works
from inside: install it on the disk, attach it with gdb -p, and use it as on
any Linux host.
Both are recorded. A replay repeats every step, every write and every stop, and a checkpoint taken before a session lets you run it again from the same state.
The machine’s debugger
Each command takes the run and a task by its tid, which
machine tasks lists. A task’s tgid is its
process. The commands work while the machine is stopped, between runs, on a
task that is on a vCPU or off one.
Symbols, types and source lines come from the program files on the machine’s disk: the program’s symbol table, and its debug information when it has some.
Debug information
What the debugger shows depends on what the program’s files on the disk hold:
| On the disk | What you get |
|---|---|
| The symbol table alone | Function names in a backtrace, a stop at a function by its name, registers, memory, and a global at its symbol’s address |
| Debug information too | Source lines, a stop at FILE:LINE, line steps, arguments in a backtrace, print and locals by name, and a watch of a variable by its name |
Frame 1 of the same stop, in pg_parse_query, of PostgreSQL as it ships and
with its debug package:
1 | 1 0x56365898a1af ?? |
A program built with -g has its debug information in itself. A
distribution ships it apart, in a package such as -dbg, -dbgsym or
debuginfo, which puts a debug file under /usr/lib/debug. Install the
package of the program’s exact build in the image the disk is built from:
the debugger takes a debug file only when its build id is the program’s, and
finds it in any layer of a container’s image.
A server is built optimized, as it ships. Its arguments and locals are where
the compiler kept them at each instruction: on the stack, in registers, or in
pieces of both, as clang keeps a C++ std::string_view in two registers. One
kept nowhere is optimized out. A C++ function is named short, such as
DB::executeQuery. machine --signatures names it by its whole signature.
Without debug information, read the arguments from the registers the ABI passes them in, and give a global a type with a cast:
1 | machine mem app 412 '$rsi' s # the second argument, a string |
The debug examples build ClickHouse and PostgreSQL both ways, and show a session of each.
Locations
Every command that takes a location takes one of these. A location names an address in the task’s process:
| Location | What it names |
|---|---|
0x401136 |
an address |
*0x401136 |
the same address, written with a * |
main |
a function or a variable, by its name |
libc.so.6:malloc |
a name in one file of the process, by its path or its file name |
init.c:106 |
a source line, at its first code |
$rsp |
a register of the task, as it is now |
main+4, $rsp-8 |
any of these, moved by +N or -N |
When the program has debug information, a breakpoint at a function by its name is placed past its prologue. Without it, the breakpoint is at the function’s first instruction.
Registers and memory
1 | machine regs app 412 # the general registers of tid 412 |
machine mem shows memory in the format given
after the location: a count, a letter and a unit, such as 8xg, 4i or s:
1 | machine mem app 412 '$sp' # eight giant words at the stack |
A write is part of the run, and a replay makes it again. A page the tgid has
not written yet is shared, and is refused for a write. A name two files of the program define is
refused with each file to name, /lib/libc.so.6:optind, by its path in the
container as machine symbols lists it. ^= flips the bits a
mask sets, at the variable’s width or a unit’s, such as
machine mem app 412 '$sp^=0x1' b; it reads, flips and writes with the
machine stopped, and says the value before and after.
The stack, symbols and variables
1 | machine backtrace app 412 # the stack, the innermost frame first |
print takes a variable of the frame, a global or a $register, with .m,
->m, [i], *, & and casts, and prints it by its type. print and
locals need the program’s debug information.
Stepping
machine step steps one task. The machine runs
whichever task the guest schedules, so other tasks run meanwhile:
1 | machine step app 412 # one instruction, as stepi |
A step of an instruction that faults, such as a write to a copy-on-write page, ends on the next instruction.
Breakpoints, watchpoints and catchpoints
A breakpoint stops the run when a task of its tgid reaches its location. A watchpoint stops it after an instruction writes the bytes it watches. A catchpoint stops it at an event of a task: an exec, an exit, a fork or vfork, a system call, a signal, a file mapped or unmapped, an injected fault, a disk request or a C++ exception.
1 | machine break app add 412 main # stop tgid of 412 at main |
watch add takes --on write, change, read or access, and --size for
a location that is not a variable. An exit catchpoint takes --in tgid=N,
since an exit is the end of a process. A throw, catch or rethrow
catchpoint needs --in tgid=N.
--in comm=NAME stops at the events of the tasks with that name, the same
tasks machine tasks app exist comm=NAME
lists. That catches one tool of a busybox, or a script, before you know its
ids. An exec catchpoint sees the new program’s name. An exit catchpoint
sees the name of the process’s main thread, which ends last, so a worker
thread named cat inside a differently named process does not stop it. A
syscall catchpoint with --in comm=NAME makes the machine report every task’s calls
while it is set, which costs time on a busy machine.
Each stop has a number, and the same verbs change any kind:
1 | machine points app # every breakpoint, watchpoint and catchpoint |
machine break, watch and catch each list their own kind.
A run that reaches a stop ends there and says which. The run log keeps the stops: a branch has the ones its parent had where it was forked, and a replay applies the adds and deletes. The stops are armed while the run records, and not while it replays.
Running to a place
1 | machine run app --iters 0 # until the next hit, as continue |
tid=N:at=LOC and tgid=N:at=LOC take any location. write=LOC and
access=LOC take a variable or an expression, watched at its size of 1, 2, 4
or 8 bytes. A write the kernel makes into the program counts, as read makes
into its buffer. The address is resolved when the run starts, and the stop
lasts for that run alone.
The run says the tid it stopped in and where that task stands. A watch also says the value the access left in the bytes:
1 | ✔ Written counter at main+116, watched.c:63, in tid 191 |
Going back
Going back puts you on a new branch, at an earlier point of the run, and leaves the run as it was:
1 | machine run app --back # onto a branch at the newest hit, as reverse-continue |
The branch keeps the run’s stops as they are now.
Calling, returning and signals
1 | machine call app 412 square 3 # call a function in the task, as call |
call puts the task’s registers back after the call. Each of these is part of
the run, and a replay makes it again.
At the control prompt
Every command here has a slash command at the
control prompt, with the run left
out: /regs 412, /mem 412 '$pc' 4i, /step 412 --line, /break add 412 main, /run until 'tid=412:at=main'. The prompt also has /display, an
expression printed after each run and step, and /skip, what a line step of
the session steps out of:
1 | /display 412 counter # printed after every /step and /run |
/help lists them all.
From Python
The Python SDK has the same debugger: run.task(tid) gives a
task with registers(), read(), backtrace(), print(), step(),
breakpoint() and watch(), and run.breakpoints lists and changes the
stops.
How many stops a run holds
A run holds 64 breakpoints. The machine puts each one into the tgid’s own copy
of its code as an int3 instruction. Another process that maps the same library does not stop, and
other threads of the tgid stop at it too. A read of the task’s memory, with
mem or the instruction a step shows, shows the program’s byte, not the
int3. A write over that byte is refused while the breakpoint is there.
The throw, catch and rethrow catchpoints are breakpoints too. The other
catchpoints are events the machine reports, and a run holds 8 of them.
A run has 3 debug registers. These take one each:
- a watchpoint of
machine watch add - a breakpoint added with
--hardware - the stop of
run --until tid=N:at=LOC,write=LOCoraccess=LOC, for that run - the stop of a step:
step --overandstep --outtake one, and two in a C++ program, where the second stops at a catch the exception reaches - a hardware breakpoint or watchpoint of GDB inside the machine,
hbreakorwatch
When the 3 are taken, the next is refused. A machine watch add refused while
GDB holds some says how many:
1 | No room: a run holds 3 hardware breakpoints and watchpoints. The debugger in the guest holds 1. |
A step of one instruction takes none, from a breakpoint’s hit too. The one exception is a task that returns into a signal handler. The step stops where the handler returns and takes a debug register for that stop.
A breakpoint in a library every process maps
A breakpoint at malloc stops one process, though every process of the
machine calls libc’s malloc. The others run on, and pay nothing for it:
1 | $ machine break app add 218 malloc |
A breakpoint in a library the program loads later is placed when it loads, and again each time it is loaded after being unloaded.
Many breakpoints at once
A pattern adds a breakpoint at each function that matches. For example, 38 of libc’s string functions:
1 | machine break app add 251 --regex '^str[a-n]' |
A pattern that matches more functions than the run has room for adds none, and says how many it matched and how many the run holds:
1 | ✗ 79 functions match: strcasecmp, strcasecmp_l, strcasestr, strcat, strchr and 74 more. The run holds 64 more breakpoints. |
Threads
Every thread of the tgid stops at a breakpoint, each with its own tid. For example, two worker threads that run the same function:
1 | $ machine break app add 251 worker_tick |
A child the process forks, or vforks, is another tgid and runs on, though it shares the parent’s copy of the code until it writes.
When to use –hardware
A software breakpoint is the right one almost always. --hardware is for the
cases an int3 in the program cannot cover:
Code in memory the program allocated, as a JIT writes. The listing says the breakpoint is not placed, with the command that puts it in a register:
1
2
3
4$ machine break app
# KIND ADDRESS SIZE WHERE TGID HITS WHEN
1 breakpoint 0x7f35d1580000 1 *0x7f35d1580000 251 /usr/bin/bpprobe 0 not placed: the address maps no file
Put it in a debug register machine break app add 251 '*0x7f35d1580000' --hardwareCode the program reads. The program sees the
int3, so a checksum of its own code changes, for example. A hardware breakpoint changes no byte.An instruction the kernel does not let a breakpoint sit on, such as
syscall. The listing saysnot placed: the kernel refused the instruction.
A hardware breakpoint lists as hw breakpoint and takes one of the 3 debug
registers.
A GDB hbreak or watch refused for the same reason makes GDB say
Couldn't write debug register: No space left on device at the continue
that inserts it. An x86 computer has 4, so a GDB session that uses all 4 there
gets 3 in the machine.
GDB inside the machine
GDB runs in the machine as it does on any Linux host: breakpoints, stepi,
next, finish, hbreak, watch and detach.
Getting GDB onto the disk
Put GDB in the image the disk is built from. For a server in a container, put it in the server’s image, so GDB sees the server’s files:
1 | FROM clickhouse/clickhouse-server:26.8.31.2 |
GDB reads the program’s symbol table for function names. For arguments, locals and source lines it needs the program’s debug information too, in a format it reads. ClickHouse’s debug information is DWARF 5, which GDB 12 and later read. An older image’s GDB may not: GDB 9.2, Ubuntu 20.04’s, does not.
GDB holds what it reads in memory, and the machine needs room for it beside the
program. ClickHouse’s 1.7GB of debug information grows to about 3.8GB in GDB,
so that machine needs --mem 12288; with 4GB the guest’s out-of-memory killer
ends GDB.
The ClickHouse example builds both kinds of image for ClickHouse 26.8: one with the symbol table alone, and one with the debug information. Its server prints the commit it was built from as it comes up, with a link to that source on GitHub, and the example shows how to put a source file where GDB finds it.
A batch of commands
Build the example’s disk from its server image, and create a run on it.
The example’s README
builds the image:
1 | machine disks build --image clickhouse-server:26.8.31.2 clickhouse-server |
Run the machine to the point you want to debug, and keep a checkpoint there.
Then run GDB with machine exec. Here the
example’s server prints clickhouse: ready once it answers queries:
1 | machine run ch --until 'console~clickhouse: ready' |
1 | [New LWP 78] |
exec run runs the machine until GDB exits, and prints what GDB printed.
-batch makes GDB exit after its commands, and an attached process is
detached as GDB exits. A file of commands works the same way, with -x FILE.
-c gives the line to the container’s shell, which finds the process id.
ClickHouse signals its own threads with SIGUSR1 and SIGUSR2. Without the
handle line, continue stops at the first of those signals instead of at
the breakpoint, and bt shows whichever thread got it.
At the control prompt the line after -c needs no outer quotes:
1 | /run until 'console~clickhouse: ready' |
Use --user to run GDB as the user of the process it attaches to. A
container has no CAP_SYS_PTRACE by default, so root in it cannot attach to
another user’s process.
Keeping GDB attached
-batch runs a list of commands chosen in advance. To decide the next
command from what GDB just printed, keep GDB attached and give it its
commands through a named pipe in the container.
This works on the server’s own image with GDB added, the example’s server
image: ClickHouse 26.8 with no debug information. GDB finds functions by
name in the program’s symbol table, and attaches in about 16 seconds. With
the debug information of the debug image it takes minutes instead.
Make the pipe, keep it open, and start GDB with its input on it.
exec start returns at once and prints the id of the process it started:
1 | machine exec ch run --container app --user clickhouse -- mkfifo /tmp/gdb.in |
1 | ✔ Sent 3615877d to ch (whvR8O5Y1otCgBTbNi3rc) |
The sleep holds the pipe open. Without it, GDB reads the end of its input
after the first command and exits.
Every step below is the same three commands. exec start queues a printf
that writes the step’s GDB commands to the pipe, the last of them an echo
of a marker. run --until runs the machine until GDB prints that marker,
which is when GDB has done every command before it. proc reads what GDB
printed, by the id exec start gave:
1 | machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "handle SIGUSR1 SIGUSR2 nostop noprint" "echo step-1\n" > /tmp/gdb.in' |
1 | Attaching to process 7 |
The handle line keeps continue from stopping at ClickHouse’s own signals,
as in the batch session. This first step waits for GDB to read the program’s
symbols, 10 seconds here. The others take a second or two.
A few things keep the steps right:
- Each step has a marker of its own.
run --untilends on text the run prints from its start, sostep-1printed before never ends a later run. - Use
exec start, notexec run, to write to the pipe.exec runruns the machine until theprintfexits, and GDB may print the marker during it. Therun --untilafter it then waits for a marker that is already there. printf "%s\n"writes each argument as a line and leaves\nin it as it is, so GDB’sechoends the marker’s line. A$in a GDB command is written\$inside the double quotes, as"x/s \$rsi"below.--within 60sbounds a step whose marker never comes. Without it a mistyped marker runs the machine until it is cancelled.
Each read shows the whole session so far. The output below is what each step added.
A breakpoint. GDB writes an int3 into the program, as on any host, and
uses no debug register:
1 | machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "break DB::executeQuery" "info breakpoints" "echo step-2\n" > /tmp/gdb.in' |
1 | (gdb) Breakpoint 1 at 0x1d386340 (3 locations) |
Continue to it. The example’s server sends itself a query every second,
and a thread that handles it stops. GDB reads the marker after c stops, so
the marker follows the hit:
1 | machine exec ch start --container app --user clickhouse -c 'printf "%s\n" c "echo step-3\n" > /tmp/gdb.in' |
1 | (gdb) Continuing. |
Every later continue works the same way. Each hit prints a new
hit Breakpoint line, so --until '/output/exec/3615877d/stdout~hit Breakpoint'
stops at the next hit too.
Read the query. executeQuery returns its result by value, so rdi
holds where to put it, and the query follows in rsi and rdx, a pointer
and a length:
1 | machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "info registers rsi rdx" "x/s \$rsi" "echo step-4\n" > /tmp/gdb.in' |
1 | (gdb) rsi 0x7f8482a32cc0 140207104142528 |
Instructions and single steps:
1 | machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "x/4i \$pc" stepi "nexti 3" "echo step-5\n" > /tmp/gdb.in' |
1 | (gdb) => 0x1d386340 <_ZN2DB12executeQuery...>: jmp 0x1d38634b <_ZN2DB12executeQuery...+11> |
The stack, and running to the return:
1 | machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "bt 3" finish "echo step-6\n" > /tmp/gdb.in' |
1 | (gdb) #0 0x000000001d386350 in DB::executeQuery(...) () |
A hardware breakpoint. Delete the first breakpoint, and put one at the
address finish returned to. It takes one of the run’s 3 debug registers,
and stops where the next query returns:
1 | machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "delete 1" "hbreak *0x1ed12ec3" c "echo step-7\n" > /tmp/gdb.in' |
1 | (gdb) (gdb) Hardware assisted breakpoint 2 at 0x1ed12ec3 |
delete prints nothing, so its prompt stands alone at the start.
A hardware watchpoint. It takes another debug register, and stops after the next call writes its return address below the stack pointer:
1 | machine exec ch start --container app --user clickhouse -c 'printf "%s\n" "watch -l *(long *)(\$rsp - 8)" c "echo step-8\n" > /tmp/gdb.in' |
1 | (gdb) Hardware watchpoint 3: -location *(long *)($rsp - 8) |
A system call. Delete every breakpoint and watchpoint, stop at the next
sendto, and read its socket, its length and the first bytes it sends.
GDB asks before it deletes them all, and answers itself, since its input is
not a terminal:
1 | machine exec ch start --container app --user clickhouse -c 'printf "%s\n" delete "catch syscall sendto" c "info registers rdi rdx" "x/16xb \$rsi" "echo step-9\n" > /tmp/gdb.in' |
1 | (gdb) Delete all breakpoints? (y or n) [answered Y; input not from terminal] |
That is the query’s result going back to the client, 34 bytes on socket 58.
Leave. Delete the catchpoint, detach and quit. GDB prints no marker after
quit, so the run waits for the process’s status file to say it exited:
1 | machine exec ch start --container app --user clickhouse -c 'printf "%s\n" delete detach quit > /tmp/gdb.in' |
1 | (gdb) Delete all breakpoints? (y or n) [answered Y; input not from terminal] |
GDB has exited, and the server carries on.
The session is recorded with the run. Rewind to the checkpoint taken before it, and the same commands give the same answers.
From the checkpoint
The checkpoint holds the machine as it was before GDB attached. Rewind to it, and the next session starts from the same instruction:
1 | machine rewind ch to ^ready |
Or fork a branch at it and keep both:
1 | machine fork ch --at ^ready --name again |
Reading the disk
You can read files from a run’s disk without touching the run.
machine fs reads the filesystem on the disk as a
checkpoint saved it, so it never changes how the machine executes.
1 | machine fs ls app /var/log --at ^done |
Every read names its checkpoint with --at, as a point: ^booted for a
checkpoint by name, or '#4200' for the checkpoint at an entry. There is no
default. This is not a view of the running machine. A listing that did not
say how old it is would look like one.
1 | machine fs ls app /tmp |
A point with no checkpoint at it is refused, and the refusal names the checkpoint before it.
Unsynced writes
Data the machine has not synced to its disk is not visible. A file a program wrote a moment before the checkpoint can be missing, short or empty, because the machine still holds it in memory.
To read the files as they are, sync in the machine, take a checkpoint, and read
at now:
1 | machine exec app run -- sync |
now is the checkpoint the run stands at. It names one until the machine runs
again; after that, name the checkpoint itself.
A run created with --fs-sync writes every change of a file to the disk as it
is made, and one created with --fs-dirsync does so for every change of a
directory. Both are options of machine create
and are off by default.
To read the files of the machine as it runs, execute a command in it:
machine exec app run -- ls /tmp. That reads now,
synced or not, and is recorded in the run.
At the control prompt the command is
/fs, and the point follows the word at:
1 | /fs ls /var/log at ^done |
From Python, run.fs.ls("/var/log", at="^done") lists a directory,
run.fs.cat("/var/log/app.log", at="^done") returns a file’s bytes, and
run.fs.cp("/var/log", "./logs", at="^done", recursive=True) copies it out.
Account and billing
Most of what the account page on the website does, the client does too.
| Command | Does |
|---|---|
machine account show |
shows your account |
machine account provision |
sets up account storage ahead of time; the first session or disk build does it otherwise |
machine account credits |
shows your credits |
machine account activity |
lists what they were spent on |
machine account products list |
lists what is for sale |
machine account buy-usage |
opens checkout for a pack of usage credits, as listed by products |
machine account buy-plan |
subscribes to a plan, or switches tier if you already have a paid one |
machine account upgrade-plan, machine account downgrade-plan |
moves you to a higher or lower plan |
machine account cancel |
cancels the subscription: the plan stays until it renews, then drops to Free, and your usage credits stay |
machine account invoices |
lists your invoices |
machine account orders |
lists your purchases, and resumes or cancels a pending checkout |
machine account payment |
updates your payment method |
machine account portal |
opens your billing settings |
machine account api-keys |
creates, lists, deletes and sets the expiry of API keys |
machine account devices |
lists where you are signed in (--revoke signs one out) |
machine account email |
changes your email address |
machine account close |
closes the account, in two steps (--cancel stops it) |
payment, portal, cancel, upgrade and downgrade open a browser. Add
--link to print the address instead.
Sessions use credits. Machine writes an amount of credits with the sign ₡,
so ₡0.15 is 0.15 credits. The price is per vCPU-hour, and
depends on the hardware class. A session is created on standard-ip unless you
pick another class, and it uses ₡0.15 a vCPU-hour. standard-ie uses ₡0.12.
Each vCPU comes with 2G of memory and 16G of disk. A session that asks for more
is charged for the vCPUs that would bring it.
machine sessions classes lists the
current price of each class. Your plan includes plan credits every month, and you buy usage credits
at €1 a credit. A credit has no cash value.
Pricing lists the plans. For dedicated machines, reserved for your team alone, contact sales@testflows.com.
A running session is charged an hour ahead. When it stops, the unused part
of its last hour is settled back. In the activity, a Charged row’s TIME is the
hour and a Settled row’s is negative, the part given back, so a session’s
rows add up to the time it ran, as their CHANGE adds up to what it used.
machine account activity <session>
lists a session’s charges and what they add up to. How credits are granted and used
is in the usage credit terms.
Runs, logs and disks take up space in your account. machine storage show says
how much, and machine storage prune frees the space that deleted runs and
disks left behind. A prune does nothing while that space is less than 5% of
your storage. machine storage prune --status shows the current or last
prune.
machine storage check reads everything in your account’s
storage back and reports what it could not read, showing its progress as it
goes. It writes nothing.
Your plan’s storage quota counts TOTAL: what your runs and disks hold, and the
space deleted ones have not given back yet. machine account show shows the
same figure. Deleting a run or a disk moves its space to PRUNABLE, which still
counts until you prune. The figures are from the last measurement, and
machine storage --refresh measures again.
At the quota, a session starts no machine and creates no branch:
machine create, machine start, machine fork, machine go,
machine detach and machine disks build are refused.
A machine that is already running keeps recording and saving. To get under
the quota, delete runs or disks, stop your sessions, and prune.
Scripting
Everything the client does can be done from a script.
-o jsonprints machine-readable output.-o rawprints text without formatting values. Put the flag before the command, likemachine -o json runs.-qturns off progress output and spinners.--timeout sgives up after that many seconds. It goes before the command too. The default is no limit, and0means check once.--no-colorsturns off colors.- A command exits with 2 when it fails or is refused, and with 1 when it cannot reach the service or a session. A waiting command exits with 3 on a timeout. Ctrl-C exits with 130.
1 | machine -q --timeout 300 run app --until tasks || echo "boot did not finish" |
Commands
machine --help lists these commands, in this order. The options apply to every
command. Add --help after a command to list its own options. Where a command
takes further commands, those follow it, so each one has a heading.
Options
| Option | Does |
|---|---|
-o, --format |
Output format: text, raw, or json. The default is text. JSON carries every field. |
--version |
Show version and license. |
--no-colors |
Disable terminal color highlighting. |
--signatures |
Name functions by their whole signature, parameters included. Without it a C++ function is named short, as GDB names it: DB::executeQuery. |
--debug |
Show the messages exchanged with the service. |
--api-url |
Cloud API base URL. Overrides TESTFLOWS_MACHINE_API_URL and the default. |
-s, --session |
Target session, by name, id, or unique prefix. Goes before the command, as every option here does. Commands that need no session ignore it. |
--timeout |
Give up after S seconds, for commands that wait. The default is no limit. 0 means check once. |
-q, --quiet |
No progress or spinners. |
machine login
Sign in. See Signing up.
A sign-in code is emailed. Use the account signup command if there is no account yet.
Or set TESTFLOWS_MACHINE_API_KEY instead.
1 | machine login [-h] |
Examples:
1 | machine login sign in |
See also:
machine sessions
List and manage sessions. See Sessions.
A session is a reserved machine in the cloud. It holds CPUs, memory and disk space for runs. A run is created, started and stored in a session.
The create subcommand creates a session and uses it, so later commands in this terminal go to it. The use subcommand switches to another session.
Use -s to select a session for a single command. A session bills while it runs.
1 | machine sessions [-h] [-w] command ... |
| Option | Does |
|---|---|
-w, --watch |
Watch for changes |
Examples:
1 | machine sessions list sessions |
See also:
machine sessions list
List sessions. See Sessions.
The current session is tagged. A session that stopped shows how it stopped. Use
--all to list the stopped sessions of the account as well.
1 | machine sessions list [-h] [-w] [-a] |
| Option | Does |
|---|---|
-w, --watch |
Watch for changes |
-a, --all |
List stopped sessions as well, the newest 50 |
Examples:
1 | machine sessions list list sessions |
See also:
machine sessions create
Create a session and use it. See Sessions.
The command uses the new session in this terminal, as sessions use does. Use
--no-use to keep using the session in use before.
The command waits until the session is running. Use --no-wait to return once
the request is accepted. A session created with --no-wait is not used.
1 | machine sessions create [-h] [--cpus n] [--mem size] [--storage size] [--class type] |
| Argument | Does |
|---|---|
[name] |
Session name (default: auto-generated) |
| Option | Does |
|---|---|
--cpus n |
Number of vCPUs (default: 1) |
--mem size |
Memory, MB or with a K/M/G/T suffix (default: the class’s memory for each vCPU, as sessions classes lists it) |
--storage size |
The session’s storage, GB or with a K/M/G/T suffix (default: the class’s storage for the session’s size, up to the plan’s limit) |
--class type |
Hardware class, as sessions classes lists it (default: the plan’s class, which sessions classes tags) |
--no-wait |
Return without waiting |
--no-use |
Create the session without using it |
Examples:
1 | machine sessions create work create the session work and use it |
Refuses, and the way past:
1 | a class with no capacity now machine sessions classes |
See also:
machine sessions use
Make a session the default for later commands. See Sessions.
The command connects first if needed. It checks that the session responds before
switching. With --no-wait it fails if a connect is needed and the session is
not running yet.
1 | machine sessions use [-h] [--no-wait] [session] |
| Argument | Does |
|---|---|
[session] |
Session name, id, or prefix (default: the only one) |
| Option | Does |
|---|---|
--no-wait |
Fail if a connect is needed and the session is not running yet |
Examples:
1 | machine sessions use work use work in this terminal |
See also:
machine sessions show
Show session details. See Sessions.
1 | machine sessions show [-h] [session] |
| Argument | Does |
|---|---|
[session] |
Session name, id, or prefix (default: the only one) |
Examples:
1 | machine sessions show work details of work |
See also:
machine sessions classes
List the hardware classes a session can be created on.
Each class names its region and its CPU architecture. A session runs in its
class’s region, and its runs are stored there. A run starts only in a session of
the architecture it was recorded on. Use the class name with create --class.
A session created without --class runs on the plan’s default class. The list
tags that class default.
1 | machine sessions classes [-h] |
Examples:
1 | machine sessions classes list the classes |
See also:
machine sessions delete
Delete a session. See Sessions.
The first operand names the session. Use -a to delete every session.
The command stops the machines of the session and syncs their runs to shared storage. The runs start again in another session. A session already gone counts as success. Its connection on this computer is cleared.
The command waits until a named session is gone. Use --no-wait to return once
the request is accepted.
1 | machine sessions delete [-h] [-a] [-y] [--no-wait] [session] |
| Argument | Does |
|---|---|
[session] |
Session name, id, or prefix (default: the only one) |
| Option | Does |
|---|---|
-a, --all |
Delete every session |
-y, --yes |
Skip the confirmation prompt |
--no-wait |
Return without waiting |
Examples:
1 | machine sessions delete work -y stop and delete work |
See also:
machine sessions rename
Rename a session. See Sessions.
Only the name changes. The session keeps running.
1 | machine sessions rename [-h] session new-name |
| Argument | Does |
|---|---|
session |
Session name, id, or prefix |
new-name |
New session name |
Examples:
1 | machine sessions rename work build rename work to build |
See also:
machine sessions keep
Keep a session running while it is idle.
A session is idle while it serves no command. An idle session stops after the plan’s idle time. Keep holds it for the duration, up to the plan’s limit. A later keep replaces the hold.
1 | machine sessions keep [-h] session duration |
| Argument | Does |
|---|---|
session |
Session name, id, or prefix |
duration |
How long to keep it, such as 30m or 1h |
Examples:
1 | machine sessions keep work 30m keep work running idle for 30 minutes |
See also:
machine sessions connect
Connect to a session. See Sessions.
The command connects this computer to the session and uses it in this terminal. The use subcommand does the same and also checks a connection already open. Use sessions use instead.
The command waits until the session is running. Use --no-wait to return once
the request is accepted.
1 | machine sessions connect [-h] [--no-wait] [session] |
| Argument | Does |
|---|---|
[session] |
Session name, id, or prefix (default: the only one) |
| Option | Does |
|---|---|
--no-wait |
Fail if the session is not running yet |
Examples:
1 | machine sessions connect work connect to work |
See also:
machine sessions disconnect
Disconnect from a session. See Sessions.
The session keeps running. The connection is shared on this computer, so every terminal loses it. Use -a to disconnect every session.
1 | machine sessions disconnect [-h] [-a] [session] |
| Argument | Does |
|---|---|
[session] |
Session (default: the current one) |
| Option | Does |
|---|---|
-a, --all |
Disconnect every session |
Examples:
1 | machine sessions disconnect work disconnect from work |
See also:
machine ping
Check reachability and latency. See The client.
The command pings the API and the connected sessions. It reports the state, round-trip time, jitter, setup time and certificate lifetime of each target.
1 | machine ping [-h] [-c n] |
| Option | Does |
|---|---|
-c, --count n |
Round-trips per target (default: 3) |
Examples:
1 | machine ping reachability and latency of the API and sessions |
Prints:
1 | text TARGET STATE RTT JITTER SETUP CERT LEFT |
See also:
machine account
Show and manage the account. See Account and billing.
The account holds the plan, the credits, the API keys and the signed-in devices. The subcommands buy credits or a plan, manage invoices and payment, and close the account.
1 | machine account [-h] command ... |
Examples:
1 | machine account plan, credits and quotas |
See also:
machine account devices
List signed-in devices. See Account and billing.
Use --revoke to sign out one device by id.
1 | machine account devices [-h] [--revoke id] |
| Option | Does |
|---|---|
--revoke id |
Sign out one device by its id (or a unique id prefix) |
Examples:
1 | machine account devices signed-in devices |
See also:
machine account buy-usage
Buy usage credits. See Account and billing.
The command opens checkout for the pack of the named number of credits. A credit costs one euro. A credit has no cash value.
1 | machine account buy-usage [-h] [-y] credits |
| Argument | Does |
|---|---|
credits |
Credits to buy, one of the amounts account products list shows. A credit costs one euro |
| Option | Does |
|---|---|
-y, --yes |
Skip the confirmation prompt |
Examples:
1 | machine account buy-usage 10 buy 10 usage credits for €10 |
See also:
machine account buy-plan
Subscribe to a plan. See Account and billing.
On a paid plan, the command upgrades or downgrades to the named tier instead.
1 | machine account buy-plan [-h] [-y] plan |
| Argument | Does |
|---|---|
plan |
Plan to subscribe to: starter, plus or pro |
| Option | Does |
|---|---|
-y, --yes |
Skip the confirmation prompt |
Examples:
1 | machine account buy-plan starter subscribe to starter |
See also:
machine account upgrade-plan
Upgrade to a higher plan. See Account and billing.
An omitted tier is the next one up. Use --link to print the URL without
opening a browser.
1 | machine account upgrade-plan [-h] [--link] [-y] [plan] |
| Argument | Does |
|---|---|
[plan] |
Plan to move to (default: the next plan up) |
| Option | Does |
|---|---|
--link |
Print the link only (don’t open a browser) |
-y, --yes |
Skip the confirmation prompt |
Examples:
1 | machine account upgrade-plan upgrade to the next plan |
See also:
machine account downgrade-plan
Downgrade to a lower plan. See Account and billing.
An omitted tier is the next one down. Use --link to print the URL without
opening a browser.
1 | machine account downgrade-plan [-h] [--link] [-y] [plan] |
| Argument | Does |
|---|---|
[plan] |
Plan to move to (default: the next plan down) |
| Option | Does |
|---|---|
--link |
Print the link only (don’t open a browser) |
-y, --yes |
Skip the confirmation prompt |
Examples:
1 | machine account downgrade-plan downgrade to the previous plan |
See also:
machine account payment
Update the payment method. See Account and billing.
Use --link to print the URL without opening a browser.
1 | machine account payment [-h] [--link] |
| Option | Does |
|---|---|
--link |
Print the link only (don’t open a browser) |
Examples:
1 | machine account payment update the payment method |
See also:
machine account portal
Open billing settings. See Account and billing.
Use --link to print the URL without opening a browser.
1 | machine account portal [-h] [--link] |
| Option | Does |
|---|---|
--link |
Print the link only (don’t open a browser) |
Examples:
1 | machine account portal open billing settings |
See also:
machine account invoices
List and manage invoices. See Account and billing.
The listing shows the newest 25. Use --limit for a different page size.
1 | machine account invoices [-h] [--since date|age] [--until date|age] |
| Option | Does |
|---|---|
--since date|age |
Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--until date|age |
Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--status {draft,open,paid,uncollectible,void} |
Filter by invoice status |
--limit n |
Cap to most recent n (default: 25, max 100) |
--sort field |
Sort by created or amount |
--reverse |
Reverse sort order |
Examples:
1 | machine account invoices list invoices |
See also:
machine account invoices list
List invoices.
The listing shows the newest 25. Use --limit for a different page size.
1 | machine account invoices list [-h] [--since date|age] [--until date|age] |
| Option | Does |
|---|---|
--since date|age |
Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--until date|age |
Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--status {draft,open,paid,uncollectible,void} |
Filter by invoice status |
--limit n |
Cap to most recent n (default: 25, max 100) |
--sort field |
Sort by created or amount |
--reverse |
Reverse sort order |
Examples:
1 | machine account invoices list list invoices |
See also:
machine account invoices download
Open an invoice.
The hosted page has the PDF and receipt. Use --link to print the URL without
opening a browser.
1 | machine account invoices download [-h] [--link] invoice-id |
| Argument | Does |
|---|---|
invoice-id |
Invoice id (in_…) |
| Option | Does |
|---|---|
--link |
Print the link only (don’t open a browser) |
Examples:
1 | machine account invoices download in_123 open invoice in_123 |
See also:
machine account cancel
Cancel the subscription. See Account and billing.
The plan stays until renewal, then drops to Free. Usage credits stay. Use
--link to print the URL without opening a browser.
1 | machine account cancel [-h] [--link] |
| Option | Does |
|---|---|
--link |
Print the link only (don’t open a browser) |
Examples:
1 | machine account cancel cancel the subscription |
See also:
machine account email
Change the account email. See Account and billing.
A code goes to the current inbox. A link goes to the new one.
1 | machine account email [-h] [-y] email |
| Argument | Does |
|---|---|
email |
New email address |
| Option | Does |
|---|---|
-y, --yes |
Skip confirmation prompts |
Examples:
1 | machine account email me@example.com change the email |
See also:
machine account products
List products for sale. See Account and billing.
1 | machine account products [-h] command ... |
Examples:
1 | machine account products list products for sale |
See also:
machine account products list
List products for sale.
1 | machine account products list [-h] |
Examples:
1 | machine account products list products for sale |
See also:
machine account show
Show the account. See Account and billing.
1 | machine account show [-h] |
Examples:
1 | machine account show plan, credits and quotas |
See also:
machine account credits
Show the account’s credits. See Account and billing.
1 | machine account credits [-h] |
Examples:
1 | machine account credits the credit balance |
See also:
machine account activity
Show credit activity. See Account and billing.
A session is charged an hour ahead, as Charged rows. When it stops, a Settled row gives back the unused part of its last hour. Naming a session ends the listing with what the session used over all of its rows.
The listing shows the newest 25. Use --limit for a different page size.
1 | machine account activity [-h] [--since date|age] [--until date|age] [--period date] |
| Argument | Does |
|---|---|
[session] |
Show one session’s transactions, by name, id or id prefix |
| Option | Does |
|---|---|
--since date|age |
Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--until date|age |
Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--period date |
Show the plan period that contains this date, such as 2026-10-01 |
--type {granted,purchased,charged,settled,expired,removed,adjusted} |
Filter by activity type |
--credit-type {plan,usage} |
Credit type to filter by: plan or usage |
--limit n |
Cap to most recent n (default: 25, max 500) |
--sort field |
Sort by sequence or amount |
--reverse |
Reverse sort order |
Examples:
1 | machine account activity charges and credits |
See also:
machine account orders
List and manage orders. See Account and billing.
The listing shows the newest 25. Use --limit for a different page size.
1 | machine account orders [-h] |
| Option | Does |
|---|---|
--status {pending,processing,completed,cancelled,expired,failed} |
Filter by status |
--limit n |
Cap to most recent n (default: 25, max 200) |
Examples:
1 | machine account orders list list orders |
See also:
machine account orders list
List orders.
The listing shows the newest 25. Use --limit for a different page size.
1 | machine account orders list [-h] |
| Option | Does |
|---|---|
--status {pending,processing,completed,cancelled,expired,failed} |
Filter by status |
--limit n |
Cap to most recent n (default: 25, max 200) |
Examples:
1 | machine account orders list list orders |
See also:
machine account orders resume
Resume a pending checkout.
The command resumes the pending order. An account has one at most. Use --link
to print the URL without opening a browser.
1 | machine account orders resume [-h] [--link] |
| Option | Does |
|---|---|
--link |
Print the link only (don’t open a browser) |
Examples:
1 | machine account orders resume resume the pending order |
See also:
machine account orders cancel
Cancel a pending order so another buy can start.
The command cancels the pending order. An account has one at most.
1 | machine account orders cancel [-h] |
Examples:
1 | machine account orders cancel cancel the pending order |
See also:
machine account api-keys
List and manage API keys. See Account and billing.
A key controls runs, disks and sessions. It cannot change billing or the account.
Set TESTFLOWS_MACHINE_API_KEY to use one.
1 | machine account api-keys [-h] command ... |
Examples:
1 | machine account api-keys list list API keys |
See also:
machine account api-keys list
List API keys.
1 | machine account api-keys list [-h] |
Examples:
1 | machine account api-keys list list API keys |
See also:
machine account api-keys create
Create an API key.
The secret is shown once. A sign-in code is required.
1 | machine account api-keys create [-h] [--expiry days|never] name |
| Argument | Does |
|---|---|
name |
Label to recognize this key later |
| Option | Does |
|---|---|
--expiry days|never |
Expire after N days (e.g. 30), or “never” (default: never) |
Examples:
1 | machine account api-keys create ci create the key ci |
See also:
machine account api-keys delete
Delete an API key.
A sign-in code is required.
1 | machine account api-keys delete [-h] id |
| Argument | Does |
|---|---|
id |
Key id, as account api-keys list shows it |
Examples:
1 | machine account api-keys delete 3 delete key 3 |
See also:
machine account api-keys update
Set the expiry of an API key.
A sign-in code is required.
1 | machine account api-keys update [-h] id days|never |
| Argument | Does |
|---|---|
id |
Key id, as account api-keys list shows it |
days|never |
Number of days from now (e.g. 30) or “never” |
Examples:
1 | machine account api-keys update 3 30 expire key 3 in 30 days |
See also:
machine account provision
Set up the account’s shared storage. See Signing up.
The first session create or disk build sets up storage itself. This command sets it up ahead of them. The command is safe to repeat while it is still working.
1 | machine account provision [-h] |
Examples:
1 | machine account provision set up shared storage |
See also:
machine account signup
Create an account. See Signing up.
The command shows the Terms of Service and the Client EULA in full and asks for their acceptance. The account is made accepting them. A link is emailed to finish signing up.
1 | machine account signup [-h] |
Examples:
1 | machine account signup create an account |
See also:
machine account close
Close the account. See Account and billing.
The command emails a code that starts the close. It waits until the machines of every session stop. It then emails a second code that confirms the close.
A close that is not confirmed stays in progress. Run the command again to finish
it. Use --cancel to stop it.
1 | machine account close [-h] [-y] [--cancel] |
| Option | Does |
|---|---|
-y, --yes |
Skip confirmation prompts |
--cancel |
Abort closing and return the account to active |
Examples:
1 | machine account close -y close the account |
See also:
machine logout
Sign out. See Signing up.
The command revokes the access token. Use --everywhere to sign out of every
device and browser.
1 | machine logout [-h] [--everywhere] |
| Option | Does |
|---|---|
--everywhere |
Sign out of every device and browser, not just this one |
Examples:
1 | machine logout sign out |
See also:
machine upgrade
Install the latest client. See Download.
The command runs the installer at https://testflows.com/machine/install. The installer downloads the release and checks its signed checksum. It puts the client where it puts every install. It removes the files older versions unpacked.
The installer uses the client’s own sign-in. A client signed in asks for nothing and stays signed in. A client signed out signs in at the terminal for the download. It stays signed out after.
The latest is not downloaded again when it is installed already. A version named
is installed even when it is installed already. An older version installs the
same way. Use --list to see the versions there are.
1 | machine upgrade [-h] [--list | version] |
| Argument | Does |
|---|---|
[version] |
Version to install, YYYYMMDD-HHMM (default: the latest) |
| Option | Does |
|---|---|
--list |
List the versions there are |
Examples:
1 | machine upgrade install the latest client |
See also:
machine completion
Print the shell completion script. See The client.
The script completes commands, options and their values as the client reads them. The client answers each Tab itself, so a newer client needs no newer script.
Add the line for the shell to its startup file:
1 | bash eval "$(machine completion bash)" in ~/.bashrc |
Run the same line in a shell to complete in that shell alone. The completion lasts until the shell exits.
machine-env loads the script itself. On macOS, machine outside machine-env is the system’s command. There the line belongs in no startup file.
1 | machine completion [-h] {bash,zsh,fish} |
| Argument | Does |
|---|---|
{bash,zsh,fish} |
Shell to print the script for: bash, zsh or fish |
Examples:
1 | machine completion bash the script for bash |
See also:
machine disks
List and manage disks. See Disks.
A disk is the filesystem image that a run is created from. Build one from a static binary, a Docker image or a Compose project.
The listing shows the newest 25. Use --limit for a different page size.
1 | machine disks [-h] [-q] [--no-trunc] [--deleted] [--label name=value] [--since date|age] |
| Option | Does |
|---|---|
-q, --quiet |
Only show ids |
--no-trunc |
Full disk id |
--deleted |
Show deleted disks |
--label name=value |
Filter by a label (repeatable; matches all) |
--since date|age |
Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--limit n |
Show at most n disks (default: 25) |
--offset n |
Skip the first n disks |
-w, --watch |
Watch for changes |
Examples:
1 | machine disks list disks |
Refuses, and the way past:
1 | a disk with runs machine disks delete hello -f -y |
See also:
machine disks list
List disks. See Disks.
The listing shows the newest 25. Use --limit for a different page size.
1 | machine disks list [-h] [-q] [--no-trunc] [--deleted] [--label name=value] |
| Option | Does |
|---|---|
-q, --quiet |
Only show ids |
--no-trunc |
Full disk id |
--deleted |
Show deleted disks |
--label name=value |
Filter by a label (repeatable; matches all) |
--since date|age |
Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--limit n |
Show at most n disks (default: 25) |
--offset n |
Skip the first n disks |
-w, --watch |
Watch for changes |
Examples:
1 | machine disks list list disks |
See also:
machine disks build
Build a disk from a binary, image or Compose project. See Disks.
Exactly one source is required. --binary wraps a static executable in an image
built around it. --from names a public image that the build pulls. No Docker
is needed for --from.
--image takes a Docker image as a save tar or a name in the local Docker.
--compose takes a project directory and the images its services name.
A binary or an image runs as one container. Use --entrypoint to set the
executable. Pass arguments after –.
Use --env to set a variable in the environment of the container. The disk
holds each --env value as written.
A project runs every service in it. It takes no --entrypoint, no --env and
no arguments after –.
With --from or --binary, --add puts a file or a directory in the image, as
one more layer. Without a destination the path lands in the working directory of
the image, under its own name. Write path:dest to name another place. A dest
ending in / is a directory.
The name defaults to what the source is called. The size defaults to the size of
the images once loaded, plus 1 GB, rounded up to a whole GB. Use --dry-run to
see those numbers without building anything.
The build writes the images into the Docker store of the disk. A machine created
from the disk has every image present and loads nothing. Use --load-at-boot to
keep them on the disk as save tars. Docker then loads them each time a machine
starts from the disk.
A guest that runs docker save or docker push on one of the images needs the save tars. Such a disk holds each image twice and is sized for both.
1 | machine disks build [-h] (--binary path | --image ref | --from ref | --compose dir) |
| Argument | Does |
|---|---|
[name] |
Disk name |
[-- args ...] |
Arguments after – become the service’s command |
| Option | Does |
|---|---|
--binary path |
Static x86_64 executable to wrap |
--image ref |
Docker image, a save tar or a name |
--from ref |
Public image, pulled by the build |
--compose dir |
Compose project directory |
--size size |
Disk size, MB or with a K/M/G/T suffix (default: the loaded images plus 1 GB, rounded up to a GB) |
--dry-run |
Report what the disk would hold and build nothing |
--load-at-boot |
Keep the images as tars on the disk and load them each time a machine starts from the disk |
--entrypoint path |
Executable the service runs |
--add path[:dest] |
File or directory to put in the image, repeatable; dest defaults to the image’s working directory |
--env name=value |
Environment variable the service runs with, repeatable |
--label name=value |
Attach a label, repeatable; an empty value removes one the name carried |
Examples:
1 | machine disks build --binary ./prog hello from a binary |
See also:
machine disks show
Show disk details. See Disks.
1 | machine disks show [-h] name |
| Argument | Does |
|---|---|
name |
Disk name |
Examples:
1 | machine disks show hello details of hello |
See also:
machine disks transfers
Show disk transfers. See Disks.
The listing shows the active transfers by default. Use --since to include
history. Use --since with --active for transfers in flight that started in
that window.
1 | machine disks transfers [-h] [--active] [--since date|age] [-w] |
| Option | Does |
|---|---|
--active |
Only in-flight transfers |
--since date|age |
Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
-w, --watch |
Watch for changes |
Examples:
1 | machine disks transfers uploads and downloads |
See also:
machine disks delete
Delete disks. See Disks.
Deletion is permanent. The name is free at once. The bytes count against the storage quota until a prune removes them.
A disk that a run was created from is refused. Use --force to delete it
anyway. Those runs stay and can no longer start.
1 | machine disks delete [-h] [-y] [-f] name [name ...] |
| Argument | Does |
|---|---|
name [name ...] |
Disk names |
| Option | Does |
|---|---|
-y, --yes |
Skip the confirmation prompt |
-f, --force |
Delete a disk that runs were created from |
Examples:
1 | machine disks delete hello -y delete hello |
See also:
machine disks rename
Rename a disk. See Disks.
1 | machine disks rename [-h] name new-name |
| Argument | Does |
|---|---|
name |
Current disk name |
new-name |
New disk name |
Examples:
1 | machine disks rename hello hello2 rename hello to hello2 |
See also:
machine disks verify
Check a stored disk against its id. See Disks.
The command reads the disk back out of shared storage. It computes the id, which is what the disk is stored under. A disk that reads as stored is verified. A disk that does not is reported as damaged.
The check is a storage operation and takes about as long as reading the disk. It keeps running if the command is interrupted. Run machine storage ops to list it. One check runs at a time for an account.
The command waits for the result. Use --no-wait to return once the request is
accepted.
1 | machine disks verify [-h] [--no-wait] name |
| Argument | Does |
|---|---|
name |
Disk name |
| Option | Does |
|---|---|
--no-wait |
Return without waiting |
Examples:
1 | machine disks verify hello check hello against its id |
See also:
machine disks cancel
Cancel a transfer in progress. See Disks.
1 | machine disks cancel [-h] name |
| Argument | Does |
|---|---|
name |
Disk name |
Examples:
1 | machine disks cancel hello cancel the transfer of hello |
See also:
machine preload
Fetch a run or a disk onto this session. See Stopping and starting.
The operand is any run or disk of the account. A run fetched here starts later without a fetch. A disk fetched here is ready before a run is created from it.
Use -a to fetch every run that this session’s runs branched from, and their disks. A copy already here is skipped. A session is required.
1 | machine preload [-h] [-a] command ... |
| Option | Does |
|---|---|
-a, --all |
Everything this session’s runs need |
Examples:
1 | machine preload run run1 fetch run1 onto the session |
See also:
machine preload run
Preload a run into a session. See Stopping and starting.
The command fetches the run from shared storage onto the session. The run is there before it is needed. A run already there is skipped. The disk of the run is fetched with it.
1 | machine preload run [-h] [-a] [run] |
| Argument | Does |
|---|---|
[run] |
Run name, required without -a |
| Option | Does |
|---|---|
-a, --all |
Every run this session’s runs branched from |
Examples:
1 | machine preload run run1 fetch run1 onto this session |
See also:
machine preload disk
Preload a disk into a session. See Stopping and starting.
The command fetches the account disk onto the session. The disk is there before a start. A disk already there is skipped.
1 | machine preload disk [-h] [-a] [disk] |
| Argument | Does |
|---|---|
[disk] |
Disk name or id |
| Option | Does |
|---|---|
-a, --all |
Every disk this session’s runs need |
Examples:
1 | machine preload disk hello fetch the disk hello |
See also:
machine fs
List, copy or print files of a run’s disk.
The command reads the filesystem on the disk as a checkpoint saved it. It never reads the live machine. It needs no program in the machine and does not change how the machine executes.
Every read names its checkpoint with --at. A point that is not a checkpoint is
refused, and the refusal names the checkpoint before it. The point now is the
checkpoint the run stands at, until the machine runs again.
Data the machine has not synced to its disk is not visible. A file a program wrote a moment before the checkpoint can be missing, short or empty.
To read the files as they are, execute sync in the machine with exec. Then take
a checkpoint and read with --at now. Use exec run with cat to read a file of
the live machine.
A run created with --fs-sync writes every change of a file to the disk as it
is made. A run created with --fs-dirsync does so for every change of a
directory. Both are options of create and are off by default.
1 | machine fs [-h] command ... |
Examples:
1 | machine fs ls run1 /etc --at ^cp1 list a directory at cp1 |
Refuses, and the way past:
1 | a read with no checkpoint named machine checkpoints run1 |
See also:
machine fs ls
List files on a run’s disk at a checkpoint.
The files are read from the disk as a checkpoint saved it, not from the live
machine. Use --at to name the checkpoint. Data the machine has not synced to
its disk is not visible.
1 | machine fs ls [-h] [--at point] [-l] [-a] run [path] |
| Argument | Does |
|---|---|
run |
Run name |
[path] |
Path (default: /) |
| Option | Does |
|---|---|
--at point |
Required. The checkpoint to read the disk at: ^name, ‘#entry’ of a checkpoint, or now |
-l |
Long format (mode, size, mtime, name) |
-a, --all |
Include the current and parent directory entries |
Examples:
1 | machine fs ls run1 /etc --at ^cp1 list a directory at cp1 |
Prints:
1 | text checkpoint <entry> path <path>, then a name a line |
See also:
machine fs cp
Copy a file or directory from a run’s disk at a checkpoint.
The files are read from the disk as a checkpoint saved it, not from the live
machine. Use --at to name the checkpoint. Data the machine has not synced to
its disk is not visible. Use -r to copy a directory.
1 | machine fs cp [-h] [--at point] [-r] run path [dest] |
| Argument | Does |
|---|---|
run |
Run name |
path |
Path to copy |
[dest] |
Local destination file or directory (default: .) |
| Option | Does |
|---|---|
--at point |
Required. The checkpoint to read the disk at: ^name, ‘#entry’ of a checkpoint, or now |
-r, --recursive |
Copy directories recursively |
Examples:
1 | machine fs cp run1 /etc/hostname . --at ^cp1 copy a file out |
See also:
machine fs cat
Print a file from a run’s disk at a checkpoint.
The files are read from the disk as a checkpoint saved it, not from the live
machine. Use --at to name the checkpoint. Data the machine has not synced to
its disk is not visible.
1 | machine fs cat [-h] [--at point] run path |
| Argument | Does |
|---|---|
run |
Run name |
path |
File path |
| Option | Does |
|---|---|
--at point |
Required. The checkpoint to read the disk at: ^name, ‘#entry’ of a checkpoint, or now |
Examples:
1 | machine fs cat run1 /etc/hostname --at ^cp1 print a file |
See also:
machine sync
Publish a run to shared storage. See Stopping and starting.
Any session can then start the run. The command runs in the session that holds the run.
1 | machine sync [-h] run |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine sync run1 publish run1 to shared storage |
See also:
machine offload
List or drop this session’s local copies. See Stopping and starting.
Offload alone lists the copies this session can drop. It counts the copies in use.
The run and disk subcommands drop one kind. Use -a to drop every run and disk not in use. The copy in shared storage stays.
Start or preload brings a run back. The next start fetches a disk again. A session is required.
1 | machine offload [-h] [--runs] [--disks] [-a] command ... |
| Option | Does |
|---|---|
--runs |
Only session run folders |
--disks |
Only session disk files |
-a, --all |
Every run and disk not in use |
Examples:
1 | machine offload list the copies that can be dropped |
See also:
machine offload list
List the copies that can be dropped. See Stopping and starting.
The listing shows the copies on this session that are not in use. The rest are
counted below the table. Use --runs or --disks to show one kind. Omit both,
or pass both, to show both.
1 | machine offload list [-h] [--runs] [--disks] |
| Option | Does |
|---|---|
--runs |
Only session run folders |
--disks |
Only session disk files |
Examples:
1 | machine offload list copies that can be dropped |
See also:
machine offload run
Offload a run from a session. See Stopping and starting.
The command drops the session’s copy of the run and frees its storage. The run itself stays in shared storage. Stop does not drop the copy.
Start or preload brings the run back. The command does not drop the disk of the run.
1 | machine offload run [-h] [-a] [run] |
| Argument | Does |
|---|---|
[run] |
Run name |
| Option | Does |
|---|---|
-a, --all |
Every run not in use |
Examples:
1 | machine offload run run1 drop the copy of run1 |
See also:
machine offload disk
Offload a disk from a session. See Stopping and starting.
The command drops the session’s copy of the disk. The account disk stays. A disk in use by a live machine is refused. The next start fetches the disk again.
1 | machine offload disk [-h] [-a] [disk] |
| Argument | Does |
|---|---|
[disk] |
Disk name or id |
| Option | Does |
|---|---|
-a, --all |
Every disk not in use |
Examples:
1 | machine offload disk hello drop the copy of the disk hello |
See also:
machine storage
Show and manage storage. See Account and billing.
Storage holds the runs and disks of the account. The subcommands report its use, list its operations, reclaim space and read every object back.
1 | machine storage [-h] [--refresh] command ... |
| Option | Does |
|---|---|
--refresh |
Measure again instead of using the last measurement |
Examples:
1 | machine storage show storage the account uses |
See also:
machine storage show
Show the storage the account uses. See Account and billing.
1 | machine storage show [-h] [--refresh] |
| Option | Does |
|---|---|
--refresh |
Measure again instead of using the last measurement |
Examples:
1 | machine storage show storage the account uses |
See also:
machine storage ops
List operations on the account’s storage.
The listing shows the newest 25 operations in every state. Use --limit for a
different page size. Use --state to show some states.
STATUS says why an operation failed. It also says how many objects a running check has read.
These operations act on the account’s storage, not on a run. The ops command does not list them. The command needs no session.
1 | machine storage ops [-h] [--op-id id] [--state [^]state] [--type [^]type] |
| Option | Does |
|---|---|
--op-id id |
Show one operation, by its id (ignores --state and --type) |
--state [^]state |
Filter by state: pending, running, done, failed (default: all; repeat = OR; ^ excludes) |
--type [^]type |
Filter by type: measure, prune, verify, check (default: all; repeat = OR; ^ excludes) |
--since date|age |
Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--until date|age |
Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--limit n |
Show at most n operations (default: 25) |
--offset n |
Skip the first n operations |
-w, --watch |
Watch for changes |
Examples:
1 | machine storage ops the newest operations |
See also:
machine storage prune
Reclaim space held by deleted runs and disks. See Account and billing.
The command does nothing while unused data is below 5% of storage.
1 | machine storage prune [-h] [-y] [--status] |
| Option | Does |
|---|---|
-y, --yes |
Do not ask for confirmation |
--status |
Show the current or last prune and exit |
Examples:
1 | machine storage prune -y reclaim space without asking |
See also:
machine storage check
Check that every stored object can be read. See Account and billing.
The command reads everything in the account’s storage back and reports what could not be read.
Nothing is written. The check can run while sessions and a prune run. An object deleted while the check runs is counted apart, not as unreadable.
The command waits for the check. It shows the objects and bytes read as it goes.
Use --no-wait to return once the request is accepted.
The check is an operation. It goes on if the command is interrupted. Run machine storage ops to list it.
The command fails when an object could not be read. It names each one with the id of the failed request.
1 | machine storage check [-h] [--status] [--no-wait] |
| Option | Does |
|---|---|
--status |
Show the current or last check and exit |
--no-wait |
Return once the check is started |
Examples:
1 | machine storage check read every stored object |
See also:
machine create
Create a run and start it. See Creating a run.
Options configure the machine: disk, memory, CPUs, devices and virtual time. The RNG seed and the virtual time base, rate and costs fix the sources of nondeterminism.
Use --auto-checkpoints to set how often checkpoints are taken. Use --like to
inherit the options of another run. The disk is not inherited, so --disk is
always given.
The machine waits for commands. Use --no-daemon to run it to completion
instead.
The run keeps its execution options. Start and fork repeat them unless given others.
A source is a device the machine prints on, such as ttyS1, or a file under
/output. Use --keep-output to store a source that would be dropped. Use
--drop-output to store less. The run keeps both lists for its whole life, and
the describe command shows them.
A run stores every source but the files under /output/drop. A file there has a name and no bytes. A program writes under /output/drop through MACHINE_DROP_OUTPUT_DIR. It writes under /output/keep through MACHINE_OUTPUT_DIR.
A name given to either option may be a pattern, such as /output/drop//.log. A star matches within one directory name, a leading dot included, and never a slash. Two stars match any number of directories.
A name given to --keep-output or --drop-output decides before the directory
does. When a kept name and a dropped name cover one source, a name spelled out
decides over a pattern. Of two names spelled out, the longer decides, and a tie
keeps the source. Of two patterns, the kept one decides.
The name none, given alone, is an empty list. Use it to keep or to drop nothing where the run would have the names of another run.
The command waits until the machine accepts commands. Use --no-wait to return
once the request is accepted.
1 | machine create [-h] [--like run] [--label name=value] [--no-wait] [--keep-output name] |
| Argument | Does |
|---|---|
[name] |
Name for the new run, defaulting to a generated one |
| Option | Does |
|---|---|
--like run |
Inherit options from another run, its disk aside, overridden by explicit ones |
--label name=value |
Attach a label, repeatable; an empty value removes an inherited one |
--no-wait |
Return without waiting |
--keep-output name |
Source that the run stores though it is dropped: a path under /output, a device such as ttyS1, a pattern, or none (repeatable) |
--drop-output name |
Source that the run does not store: a path under /output, a device such as ttyS1, a pattern, or none (repeatable) |
--disk disk |
Disk to create the run from |
--mem size |
Memory, MB or with a K/M/G/T suffix (default: 256) |
--cpus n |
Number of CPUs (default: 1) |
--console {serial,virtio} |
Device the console is: serial for ttyS0 (default) or virtio for hvc0 |
--mem-type {auto,hugepages} |
Machine RAM backing: auto (default) or hugepages (local only) |
--mem-mode {auto,map,lazy} |
How memory loads when the run starts from a checkpoint: auto (default), map to load it first, lazy to load each page at first use |
--fs-sync, --no-fs-sync |
Mount filesystem with sync (default: off) |
--fs-dirsync, --no-fs-dirsync |
Mount filesystem with dirsync (default: off) |
--fb type |
Framebuffer type: vnc, gtk, sdl, or none |
--rng, --no-rng |
Give the guest an RNG device, whose seed a branch can change (default: on) |
--rng-seed n |
Seed of the RNG device, 0 to 2^64-1 (default: 0) |
--daemon, --no-daemon |
Wait for commands instead of running to completion (default: on) |
--single-step, --no-single-step |
Run in step mode, with --no-daemon. A run that waits for commands takes its mode from machine run --mode |
--rdtsc-exit, --no-rdtsc-exit |
Record an entry for every RDTSC exit, with --no-daemon only |
--repeatable, --no-repeatable |
Stop fast exactly at the limit, so recording again gives the same entries (default: the session’s) |
--limit-margin n |
Conditional branches short of a stop where the CPU’s counter interrupts (default: the host’s) |
--timer-slack duration |
How late a timer may be in code that takes no exit, 0 for none (default: 20ms) |
--instr-margin n |
Instructions short of a stop placed by an instruction count where the CPU’s counter interrupts, 0 for the host’s (default: the host’s) |
--dispatch-bound duration |
Host time a fast dispatch may run before it stops, 0 for 1s. A stop it places does not repeat (default: 1s) |
--fast-backstop n |
Instructions a fast dispatch may run with no other stop, 0 for 100000000 (default: 100000000) |
--irq-wait n |
Conditional branches a dispatch runs while an interrupt waits for the guest to take it, 0 for 200 (default: 200) |
--limit n |
Conditional branches per dispatch in fast mode, steps in step mode, 0 to leave it to the machine (default: 0) |
--backstop-limit n |
Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: 0) |
--pin-cpu n |
Pin to this host CPU (default: the least-loaded) |
--vtime-base n |
The vtime base (Unix timestamp, default: 2026-03-01) |
--vtime-rate rate |
Nanoseconds of machine time for each conditional branch the guest retires, a number or num/den, and N=rate for vCPU N’s own, such as 10,1=40 (default: 10) |
--vtime-io-cost ns |
IO exit vtime cost in ns (default: 2000) |
--vtime-rdtsc-cost ns |
RDTSC exit vtime cost in ns (default: the IO cost) |
--vtime-pause-cost ns |
Vtime cost of each PAUSE in ns (default: 2000) |
--auto-checkpoints n |
Checkpoint every n entries, 0 for never (default: scaled by memory, 50000 per 512MB) |
--side-dump, --no-side-dump |
Record a per-entry debug dump in the run log (default: off) |
--side-trace, --no-side-trace |
Record the hypervisor’s trace of each entry, for diagnosis (default: off) |
--mem-hash, --no-mem-hash |
Hash guest memory at every entry, for a replay to compare (default: off) |
--disk-hash, --no-disk-hash |
Hash the guest’s disk at every entry, for a replay to compare (default: off) |
Examples:
1 | machine create run1 --disk hello create run1 from hello |
See also:
machine start
Start an existing run. See Stopping and starting.
Use --at to replay the tip from an earlier checkpoint. Use --read-only to
open a checkpoint that cannot run or record. Use -f to stop a running machine
first.
A stopped run resumes at its tip. A run that stopped past its last checkpoint
replays its recorded tail to reach the tip. The replay reports how many entries
it replayed. Use --no-recover to refuse instead.
A branch that never ran starts in its own machine. Its parent keeps running.
The machine runs with the execution options of the run. Options given here replace them for this start only.
The command waits until the machine accepts commands. Use --no-wait to return
once the request is accepted.
1 | machine start [-h] [-f] [--read-only] [--recover | --no-recover] [--at checkpoint] |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
-f, --force |
Stop a running instance first, then start |
--read-only |
Start read-only at a checkpoint, unable to run or record |
--recover, --no-recover |
Replay a tip that was never checkpointed from the last checkpoint (default: on) |
--at checkpoint |
Checkpoint to replay to the tip from, or to open with --read-only: ^name, or #entry of a checkpoint |
--parents mode |
Where to read the runs it branched from: local for this session, remote for shared storage |
--daemon, --no-daemon |
Wait for commands instead of running to completion (default: the run’s) |
--single-step, --no-single-step |
Run in step mode, with --no-daemon. A run that waits for commands takes its mode from machine run --mode (default: the run’s) |
--limit n |
Conditional branches per dispatch in fast mode, steps in step mode, 0 to leave it to the machine (default: the run’s) |
--backstop-limit n |
Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: the run’s) |
--pin-cpu n |
Pin to this host CPU (default: the least-loaded) |
--no-wait |
Return without waiting |
Examples:
1 | machine start run1 resume run1 at its tip |
Refuses, and the way past:
1 | a run that is running machine start run1 -f |
See also:
machine pause
Pause a running machine. See Stopping and starting.
The command waits until the machine is paused. Use --no-wait to return once
the request is accepted.
1 | machine pause [-h] [--no-wait] run |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--no-wait |
Return without waiting |
Examples:
1 | machine pause run1 pause run1 |
See also:
machine resume
Resume a paused machine. See Stopping and starting.
The command waits until the machine accepts commands. Use --no-wait to return
once the request is accepted.
1 | machine resume [-h] [--no-wait] run |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--no-wait |
Return without waiting |
Examples:
1 | machine resume run1 resume the paused run1 |
See also:
machine stop
Stop a running machine gracefully. See Stopping and starting.
Use -r to stop a branch with its descendants. Use -a to stop every running
machine. Use --dry-run to list what would stop.
A machine that refuses to stop stays up and is reported. A machine that does not stop in time stays up too. Use -f with -a or -r to kill those machines.
With -f, the command lists the machines it would kill and asks first. Use -y to kill without asking.
The command waits until the machine is gone. A machine that already exited
counts as success. Use --no-wait to return once the request is accepted.
1 | machine stop [-h] [-a] [-r] [-f] [-y] [--dry-run] [--no-wait] [run] |
| Argument | Does |
|---|---|
[run] |
Run name |
| Option | Does |
|---|---|
-a, --all |
Stop all running machines |
-r, --recursive |
Stop branch and all descendants |
-f, --force |
With -a or -r, kill a machine that refuses to stop or does not stop in time |
-y, --yes |
With -f, kill without asking first |
--dry-run |
List what would be stopped without stopping it |
--no-wait |
Return without waiting |
Examples:
1 | machine stop run1 stop run1, wait until it is gone |
See also:
machine kill
Stop a machine, and end it if it does not stop. See Stopping and starting.
The machine is asked to stop and save, as stop does. It is ended if it does not stop.
The next start replays the tip of a machine that did not save. The command says when the machine did not save.
The command waits until the machine is gone. A machine already gone counts as
success. Use --no-wait to return once the request is accepted.
1 | machine kill [-h] [--no-wait] run |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--no-wait |
Return without waiting |
Examples:
1 | machine kill run1 stop run1, end it if it hangs |
See also:
machine delete
Delete a run that is not running, with its log and checkpoints. See Stopping and starting.
Use -r to delete a branch with its descendants. Use --dry-run to list what
would be deleted.
The first operand names the run to delete. Use -a to delete every run, or
--disk to delete every run created from one disk. Use --cleanup -a to resume
interrupted deletes.
The command waits until the run is gone. Use --stop to stop a running machine
first. Use -f to also delete a run whose machine is gone. Use --no-wait to
return once the request is accepted.
A delete is an operation. It goes on if the command is interrupted. Run machine ops to list it.
1 | machine delete [-h] [-a] [--disk name|id] [-r] [-y] [--stop] [-f] [--cleanup] [--full-scan] |
| Argument | Does |
|---|---|
[run] |
Run name |
| Option | Does |
|---|---|
-a, --all |
Delete all runs, or cleanup all interrupted deletes (with --cleanup) |
--disk name|id |
Delete every run created from this disk |
-r, --recursive |
Delete branch and all descendants |
-y, --yes |
Skip the confirmation prompt |
--stop |
Stop running machines before deleting |
-f, --force |
Implies --stop, and deletes a run whose machine is already gone |
--cleanup |
Resume interrupted delete operations |
--full-scan |
With --cleanup, check every run, not only those recorded as interrupted |
--dry-run |
List what would be deleted/recovered without acting |
--no-wait |
Return once the delete of one run is accepted |
Examples:
1 | machine delete run1 delete a stopped run |
Refuses, and the way past:
1 | a run with branches machine delete run1 -r |
See also:
machine cleanup
Remove stale runs and their leftovers. See Stopping and starting.
A stale run is listed as running but has no machine. Use -f to kill machines that belong to no run, and to remove those runs from the list.
1 | machine cleanup [-h] [-f] |
| Option | Does |
|---|---|
-f, --force |
Also kill machines that belong to no run, and remove runs whose machine is already gone |
Examples:
1 | machine cleanup remove stale runs |
See also:
machine run
Run one or more vCPU iterations. See Driving it.
The machine executes and the run log records each iteration, so the run can be replayed.
The mode sets how an iteration executes. Fast mode stops near the limit, a little late. Step mode stops exactly. Step-into mode also steps into syscalls, gates and injected events.
An iteration is one dispatch of a vCPU. It ends where the guest next exits to the machine, or at the limit. Its length is not fixed. One may be five instructions and the next a million.
The run stops after --iters iterations or when the --until predicate holds.
No count is right in advance, because how long a program runs is not known. Use
--until to stop where something happens, with --iters or a vtime predicate
as a bound.
A run that went too far is not lost. Use rewind or go to return to an earlier point.
A vtime predicate holds once the clock of the machine reaches a time. vtime>=T counts from the start of the run. vtime=+T counts from where the run is. The run stops at the end of the iteration that reaches the time.
Options vary the run. They draw the mode, the limit and the virtual time rate. They also draw a scheduling mark, slice and interrupt for a task. The same seed draws the same values.
The --until predicates are those of wait. The console~ predicate signals that
boot is done. The halted and activity=idle predicates do not.
A console~ predicate takes a regular expression. The run stops at the entry that ends the line the match is on. A match on a line with no end, such as a prompt, stops the run where the machine goes idle.
The match is within a line, as output search finds it, and has at least one character. Use the point console~REGEX to go to the entry that printed the first byte of the match.
The name of another source takes the place of console, as in ttyS1~REGEX. The run then stops for a line that source prints. A line another source prints does not stop it.
The path of an output file takes the same place, as in /output/drop/app/debug.log~REGEX. A file that is dropped stops the run the same way. The run stores none of the file. A path that holds a ~ is quoted, as in ‘“/output/keep/app.log~“~ERROR’.
Only text the run prints from its start ends it. A line printed before does not.
Use machine wait --for to ask whether text was printed.
tid=N:at=LOC runs until tid N reaches LOC in its own code. tgid=N:at=LOC runs until any thread of the process does. The address is resolved when the run starts. The stop takes one of the 3 debug registers the run’s watchpoints use.
tid=N:write=LOC runs until an instruction writes the bytes LOC names while tid N is on a vCPU. tid=N:access=LOC runs until one reads or writes them. tgid=N:write=LOC and tgid=N:access=LOC stop for any thread of the process.
The run stops after the access. A write the kernel makes for the task counts. A variable or an expression is watched at its size of 1, 2, 4 or 8 bytes. The watch takes one of the 3 debug registers the run’s watchpoints use.
An address is watched through a cast that gives it a type, as GDB’s watch takes one. The address is a multiple of the size. tgid=N also stops for a kernel thread that works in the process’s memory.
A run that stops at LOC says the tid that reached it and where it stands. A watch also says the value the access left in the bytes. The machine records both at the stop, so a task in the kernel still shows them.
LOC is one of these, in the task’s process:
1 | 0x401136 an address |
The run log records each iteration as an entry. It also records interrupts and device input, so entries outnumber iterations.
The global timeout cancels the run in flight. Without one there is no limit. A timeout of 0 checks once without batching. Exit 3 means the predicate was not reached.
--iters is an upper bound, not a count. A batch is the iterations one request
of the command runs. It ends at every context switch of a focused task, a task
the focus command names. It also ends where a task reaches its own code.
Without --until, a batch that ends early ends the run. With --until, the run
goes on with the next batch. The response reports how many iterations ran.
Every parameter is drawn for one iteration. The mode, the limit, the task marks and the interrupt apply to the task on that one vCPU.
Without --vcpu, a scheduling cycle dispatches every vCPU. It spends up to one
iteration for each. --iters 100 on four vCPUs buys 100 dispatches, not 400.
With --until, --iters and the predicate are both bounds. The first to fire
wins. --iters 500 --until on stops at 500 iterations or when a focused task
takes a vCPU.
Use --within to bound the run by the machine’s clock. The machine checks the
bound at every iteration, so a slow host does not end the run sooner. With
--until, the first of the two to hold ends it. Without --until, the run goes
on until the clock has moved that far.
Without --iters, only the timeout bounds the run, so --until tasks runs a
whole boot. A span is lo:hi, a range one value is drawn from for each dispatch.
A span bounds nothing either, so --iters 100:500 --until on keeps going.
--when bounds nothing. It names what must be true for the clause to apply to a
dispatch. A dispatch it excludes runs free, up to the crossing into the subject.
So --when /test:on --until /test:exit perturbs that program while its own
code runs. It runs free while the program is in the kernel or off the CPU. It
stops when the program is gone. --iters counts every dispatch, the ones that
ran free too.
--when takes on, kernel and off. Each is a state, not an edge.
The options of run make one clause of a plan. Use machine plan <run> <file>
for a run of more than one clause.
A hit of a breakpoint, watchpoint or catchpoint ends the run. Use --iters 0 to
run until one does, as GDB’s continue does.
Use --back to go back to the newest breakpoint, watchpoint or catchpoint hit
the run recorded, as reverse-continue does. The machine switches onto a branch
at that entry, as go does. The branch keeps the run’s stops as they are now. The
run stays as it was.
1 | machine run [-h] [--back] [--limit span] [--backstop-limit n] [--retry n] [--strategy name] |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--back |
Go back to the newest hit before now, onto a branch |
--limit span |
Default branches per dispatch under fast and fence, steps under step; 0 means no limit under fast, one step under step and the machine’s fence under fence; default 0 |
--backstop-limit n |
Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: 0) |
--retry n |
Retries for an entry the host interrupted, default 2; 0 does not retry |
--strategy name |
How spans resolve when they name none: random, golden, sweep (default: random) |
--seed n |
Seed for the random and golden draw strategies (default: 0) |
--vcpu span |
The vCPU, or a span to draw one per dispatch (default: all) |
--iters span |
Iterations, an upper bound; 0 is unlimited (default: 1; with --until, unbounded) |
--until predicate |
Drive until this is true: a predicate of wait --for, or tid=N:at=LOC, tid=N:write=LOC or tid=N:access=LOC, or the same with tgid=N |
--within duration |
End the run when the machine’s clock has moved this far, such as 10s |
--irqblk |
Block IRQs during run |
--rdtsc-exit |
Record an entry for every RDTSC exit |
--no-idle-skip |
Keep machine time crawling while every vCPU is idle |
--auto-checkpoints n |
Auto-checkpoint every N entries; 0 turns it off, default the machine’s |
--mode mode,... |
Execution mode: fast, step, step-into, free, fence, or a comma list to draw among, each with an optional [weight] (default: fast) |
--fast span |
Limit fast draws when it wins (its member), overriding --limit |
--step span |
Limit step draws when it wins (its member), overriding --limit |
--step-into span |
Limit step-into draws when it wins (its member) |
--fence span |
Conditional branches to stop at when fence mode is drawn, overriding --limit |
--vtime-rate span |
Vtime rate for this run: n, n/d, or a span of either |
--vtime-add span |
Nanoseconds to push the clock forward before each dispatch |
--timer-slack span |
How late a timer may be in code that takes no exit, drawn per dispatch, at most 1s |
--irq-delay [target=]span,... |
Hold each interrupt a time drawn from its target’s span, at most 1s. A target is timer, ipi, init, blk, rng, console, serial, keyboard, device or all, and a bare span is all |
--vtime-io-cost span,... |
Machine time an I/O, MMIO or hypercall exit costs, drawn per dispatch from spans such as 2us or 50:20us, 1ns to 1s |
--vtime-rdtsc-cost span,... |
Machine time an RDTSC; it follows the I/O cost unless given costs, drawn per dispatch from spans such as 2us or 50:20us, 1ns to 1s |
--vtime-pause-cost span,... |
Machine time each PAUSE a spin loop executes costs, drawn per dispatch from spans such as 2us or 50:20us, 1ns to 1s |
--task-pin [tid=]set,... |
Draw per dispatch where a task is placed when it next wakes: a vCPU set such as 1-3, free or none; tid= names the task |
--task-sched mark,... |
Draw a scheduler mark per dispatch for the task on that vCPU: keep, hold, yield, drain, drain-all |
--task-slice slice,... |
Draw that task’s slice per dispatch: keep, default, or a duration range such as 10us:600us |
--task-int int,... |
Draw an interrupt for that task per dispatch: none, preempt |
--when predicate |
Apply the clause only while this holds |
Examples:
1 | machine run run1 run one iteration |
Prints:
1 | progress <done>/<batch> vtime=<time> entry=<id> ic=<instructions> |
See also:
machine plan
Run a multi-clause plan from a file. See Plans.
A plan varies how the run executes. Its clauses draw the mode, the virtual time,
and the scheduling and interrupts of tasks. Use --iters and --until to bound
the run, as with run.
A plan file is a list of clauses. A clause is an optional when predicate and the per-dispatch axes it draws for a dispatch it matches. Clauses are tried in order. The last to match supplies each axis, so a later clause overrides an earlier one.
The else clause applies to a dispatch that no when matched. It is the last clause of the file.
Braces may group a clause. # begins a comment. Blank lines are ignored.
A file of - reads the plan from standard input. The plan applies to the run of this command only. A later machine run runs without the plan. Run machine plan again to go on with it.
Example:
1 | when /test:on mode step,fast fast=1:100k step=1:200 |
Example: place tid 56 on vCPU 2 or 3 when it wakes, 2 three times as often as 3:
1 | else task-pin 56=golden:2[3],3[1] |
A clause names one when predicate, then any of these axes. An axis is a comma list of choices, one drawn per dispatch. [N] is an optional weight on a choice:
1 | when <predicate> |
A weight makes a choice win more often. mode step[3],fast[1] draws step three times as often as fast. keep is the choice that draws nothing. It holds the last value of the axis, so a clause can leave an axis alone part of the time.
A span is a range written lo:hi[:strategy[:step]]. The strategy is random, golden or sweep. One value is drawn from the span per dispatch.
A span is accepted by the mode member limit, vtime-rate, vtime-add, task-slice,
timer-slack, irq-delay and the run-level --limit. The three exit costs take
one too. The limit itself is not a clause axis. It is the default for a mode
with no member limit of its own.
The options of this command are plan-level. --limit and --strategy are the
defaults for a clause that names no span of its own. A clause axis overrides
them for the dispatches it matches. A dispatch that no clause matches runs free.
--seed seeds the draw strategies. --until, --iters and --vcpu bound and
steer the run as on run. The clause axes and when belong in the file, not here.
1 | machine plan [-h] [--limit span] [--backstop-limit n] [--retry n] [--strategy name] |
| Argument | Does |
|---|---|
run |
Run name |
file |
Plan file of when and else clauses, and their axes, or - to read standard input |
| Option | Does |
|---|---|
--limit span |
Default branches per dispatch under fast and fence, steps under step; 0 means no limit under fast, one step under step and the machine’s fence under fence; default 0 |
--backstop-limit n |
Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: 0) |
--retry n |
Retries for an entry the host interrupted, default 2; 0 does not retry |
--strategy name |
How spans resolve when they name none: random, golden, sweep (default: random) |
--seed n |
Seed for the random and golden draw strategies (default: 0) |
--vcpu span |
The vCPU, or a span to draw one per dispatch (default: all) |
--iters span |
Iterations, an upper bound; 0 is unlimited (default: 1; with --until, unbounded) |
--until predicate |
Drive until this is true: a predicate of wait --for, or tid=N:at=LOC, tid=N:write=LOC or tid=N:access=LOC, or the same with tgid=N |
--within duration |
End the run when the machine’s clock has moved this far, such as 10s |
--irqblk |
Block IRQs during run |
--rdtsc-exit |
Record an entry for every RDTSC exit |
--no-idle-skip |
Keep machine time crawling while every vCPU is idle |
--auto-checkpoints n |
Auto-checkpoint every N entries; 0 turns it off, default the machine’s |
Examples:
1 | machine plan run1 plan.txt run the clauses of plan.txt |
Prints:
1 | tally one line for each clause, numbered from 1 |
See also:
machine control
Control a run interactively. See Interactive mode.
The prompt takes slash commands. They map to the commands of machine, with a syntax of their own.
Type /help at the prompt to list the commands. Use -c to send a slash command from the command line, without opening the prompt. Use -c /help to list the commands that way.
A semicolon between two slash commands runs the second after the first, and stops when the first fails. The prompt reads quotes as a shell reads them. A semicolon inside quotes is part of the word.
Use control log with a run to read the command history instead of driving the
machine. Use --entries to select the commands issued at given run entries. Use
-n to select commands by number.
1 | machine control [-h] [--entries range] [--wide] [-n range] [-c cmd] run|log [run] |
| Argument | Does |
|---|---|
run|log |
Run name, or log to read history |
[run] |
With log: the run whose history to read |
| Option | Does |
|---|---|
--entries range |
With log: commands at one entry, or an inclusive range |
--wide |
With log: show the entries, vCPU, opcode and size of each command |
-n, --lines range |
With log: commands: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end |
-c, --command cmd |
Slash command to run without opening the prompt, such as “/f@100” |
Examples:
1 | machine control run1 open the prompt for run1 |
See also:
machine wait
Wait until a run satisfies a predicate. See Waiting.
Use wait to script against a run. It waits for a state, an entry, a console line or a task event. Several runs, or every run of the session, may be waited for.
The predicates are running, ready, tasks, stopped, paused, halted,
activity=<name>, entry>=N, entry=+N, vtime>=T, vtime=+T, console~REGEX and a
task predicate. The tasks predicate holds once the operating system is up and
its task table is readable. That is before it execs anything in userland.
A console~REGEX predicate holds once the console prints a match. The name of another source takes the place of console, as in ttyS1~REGEX.
A task predicate is [who:]what. The what is exec, exit, sched, desched, on, kernel or off. The who is /path, cgroup=path, comm=name, tgid=N or tid=N. Omitting it means any focused task.
A path, a cgroup or a comm must already be focused. Such a subject means its
tgid. tgid=N means the tgid for exit and any of its tasks for exec. Use machine
tasks <run> --focused to see which task is on a vCPU.
Four of the seven are edges. exec fires where a task matching who entered the task table. exit fires where one left it.
sched fires where a task is given the CPU. desched fires where one gives it up.
Three are states, and they partition time. on holds while the task is on cpu in its own code, at CPL 3. That is where a forced decision can land between two of its own instructions.
kernel holds while that task is on cpu in the kernel. off holds while it is not the task on cpu at all.
Named runs must be in the current session. Use --all to wait for every run in
it.
The global timeout bounds the wait. Without one there is no limit. A timeout of 0 checks once.
Exit codes: 0 success, 2 error, 3 timeout.
1 | machine wait [-h] [--all] --for predicate [--interval s] [run ...] |
| Argument | Does |
|---|---|
[run ...] |
One or more runs (omit with --all) |
| Option | Does |
|---|---|
--all |
Wait for every run in the session |
--for predicate |
Predicate such as running, stopped, entry=+N or console~REGEX |
--interval s |
Poll every S seconds (default: 1) |
Examples:
1 | machine wait --for running run1 until run1 runs |
See also:
machine cancel
Cancel the run or replay in progress. See Cancelling a drive.
A run stops where it is. The stop falls where the host’s clock put it, so another recording of the same state does not stop there. Describe reports the run as not repeatable from that entry. A replay stops before its next entry.
Use --soft to stop the run at the end of the dispatch in progress. That stop
is an ordinary entry, and the run stays repeatable. It takes at most the run’s
dispatch bound, 1s by default. Ctrl-C stops a run this way, and a second Ctrl-C
stops it now.
1 | machine cancel [-h] [--soft] run |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--soft |
Stop at the end of the dispatch in progress, which keeps the run repeatable |
Examples:
1 | machine cancel run1 cancel what run1 is running |
Refuses, and the way past:
1 | a run with nothing in progress machine state run1 |
See also:
machine replay
Replay recorded entries from the parent lineage. See Replay.
The command executes the recorded entries again. During replay, Machine checks each entry against the original recording. If they don’t match, Machine reports a divergence. It names the entry, with the expected and actual values.
Without a line, the parent’s entries are replayed. A fork at an entry leaves that entry with the parent. The branch starts at the next entry, so the entries of the parent end at the fork. Replaying past them reports no forward entries.
Use --to to stop at a point of the line that is replayed. The point is an
entry, a checkpoint or a time of that line. A checkpoint the parent took after
the fork is such a point.
The line of a run is its entries from the root on, across every fork. Naming a run replays the entries of its line. The run must be in the same tree. The entry this run stands on must be on the line replayed.
An entry that diverges is replayed again from the newest checkpoint before it, at most twice. It is not replayed again once it diverges the same way twice. An entry that the host interrupted is replayed again the same way. It is not a divergence.
Use --retry to set how many retries. --retry 0 replays each entry once. Use
--limit-margin to set how many conditional branches short of each recorded
entry the CPU’s counter interrupts. A smaller margin makes a miss more likely.
The replay stores what the branch keeps of its output. A branch that keeps an
output file its parent dropped is how the file is read. Use fork with
--keep-output to make one. What is stored does not change what the machine
executes.
1 | machine replay [-h] [--to point [point ...]] [--trace] [--retry n] [--limit-margin n] |
| Argument | Does |
|---|---|
run |
Run name |
[line] |
Run whose line to replay, defaulting to the parent’s |
| Option | Does |
|---|---|
--to point [point ...] |
Point of the line to replay to: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, now, parent [n], root, or %mark (default: the end of the line) |
--trace |
Trace each iteration one by one |
--retry n |
Retries for an entry that diverges or the host interrupted, default 2; 0 does not retry |
--limit-margin n |
Conditional branches short of each recorded stop where the CPU’s counter interrupts (default: the run’s) |
--instr-margin n |
Instructions short of each stop placed by an instruction count where the CPU’s counter interrupts (default: the run’s) |
Examples:
1 | machine replay try2 replay the parent's entries |
Prints:
1 | text Replayed <n> entries to entry <id>. Converged. |
Refuses, and the way past:
1 | a run off the line it would replay machine replay try2 run1 |
See also:
machine play
Play recorded entries without checking divergence. See Replay.
Use --include and --exclude to select the event types to play. The entries
no type names are always played. They are the clock’s rate and steps, the focus,
the stops and the debugger’s writes. The RUN entries after them need them.
A play that leaves out irq delivers the interrupts the machine raises itself, as a recording does. The run records them as its own.
The line of a run is its entries from the root on, as replay says. The run named must be in the tree of this run, and need not be on its line. Divergence is not checked.
1 | machine play [-h] [-n range] [--include type [type ...]] [--exclude type [type ...]] |
| Argument | Does |
|---|---|
run |
Run name |
[line] |
Run whose line to play, defaulting to the parent’s |
| Option | Does |
|---|---|
-n, --entries range |
Entries: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end |
--include type [type ...] |
Play only these event types: run, irq, exception, random, device_in, block_done, input, exit_shutdown, system_shutdown, system_reset, checkpoint, cpuid, rng_seed |
--exclude type [type ...] |
Leave these event types out |
Examples:
1 | machine play run1 -n 295:300 play entries 295 to 300 of run1 |
See also:
machine diff
Compare a run log entry against another. See Replay.
The command compares the state recorded at two entries. It breaks the combined
hash into machine state, memory and disk, with register detail, so the layer
that differs shows. It reads the recorded log, so no replay is needed. Use
--show to choose sections and --context to add the entries before.
With no second operand, the entry is compared against the run it was replayed from. The answer says whether the replay converged. Naming a run compares against that run, at its own entry when given as other:m. A difference between two recordings says that they differ, not that a replay diverged.
1 | machine diff [-h] [--show sections] [--entry n] [--context n] run [other[:m]] |
| Argument | Does |
|---|---|
run |
Run name |
[other[:m]] |
Run to compare against, at its entry m; m defaults to --entry |
| Option | Does |
|---|---|
--show sections |
One or more, comma-separated: code, regs, stack, pt, instr, trace, events, timers, debug, xsave, irq, pic, ioapic, devs, mem, disk, hash, counts, entry, io, msrs, fullhash, all; default hash |
--entry n |
Compare specific entry N (0 = last entry) |
--context n |
Also show the N entries leading up to it, on both sides |
Examples:
1 | machine diff try2 compare try2 with the run it replayed |
See also:
machine checkpoint
Create a checkpoint of the machine’s state. See Checkpoints.
A checkpoint is a point to return to, written ^name. It saves the machine state. Fork, go and rewind return to it.
The machine also checkpoints on its own, at the interval that create sets.
1 | machine checkpoint [-h] run [name] |
| Argument | Does |
|---|---|
run |
Run name |
[name] |
Checkpoint name (auto-generated if omitted) |
Examples:
1 | machine checkpoint run1 checkpoint run1 under a generated name |
See also:
machine fork
Fork a branch and start it alongside the parent. See Starting a branch.
A branch shares the history of its parent up to the fork point. Then it runs on
its own path. Use --rebase to copy the data of the ancestors instead of
sharing it. Use --no-start to create the branch without starting it.
The parent keeps running. The terminal stays attached to the parent.
The branch forks at the current point and reuses a checkpoint there if one
exists. Use --at to fork at another point.
The branch keeps the execution options of the parent. Options given here replace them for the branch.
The new run keeps and drops the output that the run it comes from does. Use
--keep-output to replace the names it keeps. Use --drop-output to replace
the names it drops. Each option changes its own list only.
Use --from to replay the new run from an earlier point to its point. The new
run stores the output of that stretch by its own names. It is forked at the
checkpoint at or before the earlier point.
A run stores every source but the files under /output/drop. A file there has a name and no bytes. A program writes under /output/drop through MACHINE_DROP_OUTPUT_DIR. It writes under /output/keep through MACHINE_OUTPUT_DIR.
A name given to either option may be a pattern, such as /output/drop//.log. A star matches within one directory name, a leading dot included, and never a slash. Two stars match any number of directories.
A name given to --keep-output or --drop-output decides before the directory
does. When a kept name and a dropped name cover one source, a name spelled out
decides over a pattern. Of two names spelled out, the longer decides, and a tie
keeps the source. Of two patterns, the kept one decides.
The name none, given alone, is an empty list. Use it to keep or to drop nothing where the run would have the names of another run.
The command waits until the branch accepts commands. Use --no-wait to return
once the request is accepted.
1 | machine fork [-h] [--at point [point ...]] [--from point [point ...]] [--name name] |
| Argument | Does |
|---|---|
run |
Run to fork from |
| Option | Does |
|---|---|
--at point [point ...] |
Point the new run is at: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, now, parent [n], root, or %mark |
--from point [point ...] |
Earlier point to replay the new run from: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, parent [n] or root. The new run starts at the checkpoint at or before it |
--name name |
Name for the new run (default: a name made from where it starts) |
--rebase |
Copy the ancestors’ data into the new branch instead of sharing it |
--depth n |
With --rebase, how many root-side ancestors to keep (default 1: the root) |
--lean |
With --rebase, keep no history shared with the ancestors, so they can be deleted |
--no-start |
Create the new run without starting it |
--side-dump, --no-side-dump |
Record a per-entry debug dump, whatever the run it comes from records |
--side-trace, --no-side-trace |
Record the hypervisor’s trace of each entry, for diagnosis, whatever the run it comes from records |
--mem-hash, --no-mem-hash |
Hash guest memory at every entry, whatever the run it comes from records |
--repeatable, --no-repeatable |
Stop fast exactly at the limit, so recording again gives the same entries, whatever the run it comes from does |
--limit-margin n |
Conditional branches short of a stop where the CPU’s counter interrupts (default: the run’s) |
--timer-slack duration |
How late a timer may be in code that takes no exit, 0 for none (default: the run’s) |
--instr-margin n |
Instructions short of a stop placed by an instruction count where the CPU’s counter interrupts, 0 for the host’s (default: the run’s) |
--dispatch-bound duration |
Host time a fast dispatch may run before it stops, 0 for 1s. A stop it places does not repeat (default: the run’s) |
--fast-backstop n |
Instructions a fast dispatch may run with no other stop, 0 for 100000000 (default: the run’s) |
--irq-wait n |
Conditional branches a dispatch runs while an interrupt waits for the guest to take it, 0 for 200 (default: the run’s) |
--disk-hash, --no-disk-hash |
Hash the guest’s disk at every entry, whatever the run it comes from records |
--rng-seed n |
Seed the new run’s RNG device from n once it is up, so its guest draws other random numbers |
--keep-output name |
Source that the run stores though it is dropped: a path under /output, a device such as ttyS1, a pattern, or none (repeatable) |
--drop-output name |
Source that the run does not store: a path under /output, a device such as ttyS1, a pattern, or none (repeatable) |
--no-wait |
Return without waiting |
--daemon, --no-daemon |
Wait for commands instead of running to completion (default: the run’s) |
--single-step, --no-single-step |
Run in step mode, with --no-daemon. A run that waits for commands takes its mode from machine run --mode (default: the run’s) |
--limit n |
Conditional branches per dispatch in fast mode, steps in step mode, 0 to leave it to the machine (default: the run’s) |
--backstop-limit n |
Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: the run’s) |
--pin-cpu n |
Pin to this host CPU (default: the least-loaded) |
Examples:
1 | machine fork run1 fork here and start the branch |
See also:
machine go
Branch at a point and switch onto it. See Switching the machine.
Go starts another path from a point. The branch shares the history of the run up to the point.
The machine moves onto the branch. The run it leaves is listed as switched. A switched run keeps its history and has no machine.
Go is the only command that branches and switches in one step. The point operand lists the forms a point takes. Each names a point on the own line of this run.
console~REGEX is where the console first printed a match. It is the entry that printed the first byte of the match. The name of another source takes the place of console, as in ttyS1~REGEX.
A time such as 10ms counts from the start of the run. A minus sign, as in -10ms, counts back from now. A time may carry +N, as machine now prints it, to name one entry among several at that time.
A minus sign before a number with no unit counts entries back from now. The point -10 is #-10, 10 entries before now.
Go reaches a point that is not a checkpoint by branching at the nearest checkpoint at or before it. It then replays the difference.
Every use creates a branch, also for a point gone to before. What runs from a point is a path of its own. Use switch to return to a branch that exists. Use rewind to put this run back at a point.
The new run keeps and drops the output that the run it comes from does. Use
--keep-output to replace the names it keeps. Use --drop-output to replace
the names it drops. Each option changes its own list only.
Use --from to replay the new run from an earlier point to its point. The new
run stores the output of that stretch by its own names. It is forked at the
checkpoint at or before the earlier point.
A run stores every source but the files under /output/drop. A file there has a name and no bytes. A program writes under /output/drop through MACHINE_DROP_OUTPUT_DIR. It writes under /output/keep through MACHINE_OUTPUT_DIR.
A name given to either option may be a pattern, such as /output/drop//.log. A star matches within one directory name, a leading dot included, and never a slash. Two stars match any number of directories.
A name given to --keep-output or --drop-output decides before the directory
does. When a kept name and a dropped name cover one source, a name spelled out
decides over a pattern. Of two names spelled out, the longer decides, and a tie
keeps the source. Of two patterns, the kept one decides.
The name none, given alone, is an empty list. Use it to keep or to drop nothing where the run would have the names of another run.
1 | machine go [options] <run> <point> |
| Argument | Does |
|---|---|
run |
The running machine that moves |
[point ...] |
Where to branch: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, now, parent [n], root, or %mark |
| Option | Does |
|---|---|
--from point [point ...] |
Earlier point to replay the new run from: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, parent [n] or root. The new run starts at the checkpoint at or before it |
--name name |
Name for the new run (default: a name made from where it starts) |
--rebase |
Copy the ancestors’ data into the new branch instead of sharing it |
--depth n |
With --rebase, how many root-side ancestors to keep (default 1: the root) |
--lean |
With --rebase, keep no history shared with the ancestors, so they can be deleted |
--side-dump, --no-side-dump |
Record a per-entry debug dump, whatever the run it comes from records |
--side-trace, --no-side-trace |
Record the hypervisor’s trace of each entry, for diagnosis, whatever the run it comes from records |
--mem-hash, --no-mem-hash |
Hash guest memory at every entry, whatever the run it comes from records |
--repeatable, --no-repeatable |
Stop fast exactly at the limit, so recording again gives the same entries, whatever the run it comes from does |
--limit-margin n |
Conditional branches short of a stop where the CPU’s counter interrupts (default: the run’s) |
--timer-slack duration |
How late a timer may be in code that takes no exit, 0 for none (default: the run’s) |
--instr-margin n |
Instructions short of a stop placed by an instruction count where the CPU’s counter interrupts, 0 for the host’s (default: the run’s) |
--dispatch-bound duration |
Host time a fast dispatch may run before it stops, 0 for 1s. A stop it places does not repeat (default: the run’s) |
--fast-backstop n |
Instructions a fast dispatch may run with no other stop, 0 for 100000000 (default: the run’s) |
--irq-wait n |
Conditional branches a dispatch runs while an interrupt waits for the guest to take it, 0 for 200 (default: the run’s) |
--disk-hash, --no-disk-hash |
Hash the guest’s disk at every entry, whatever the run it comes from records |
--rng-seed n |
Seed the new run’s RNG device from n once it is up, so its guest draws other random numbers |
--keep-output name |
Source that the run stores though it is dropped: a path under /output, a device such as ttyS1, a pattern, or none (repeatable) |
--drop-output name |
Source that the run does not store: a path under /output, a device such as ttyS1, a pattern, or none (repeatable) |
--no-wait |
Return without waiting |
Examples:
1 | machine go run1 ^cp1 branch at checkpoint cp1 and switch |
Prints:
1 | result Branched <parent>/<name> (<id>) at entry <n> |
See also:
machine switch
Switch a running machine onto a branch that exists. See Switching the machine.
The first operand names the run that switches, as with fork and detach. The second names where it goes: a branch, a mark written %mark, parent or root. Switch never creates a branch.
The run that the machine leaves is listed as switched. A switched run keeps its history and has no machine.
Go forks a branch and switches onto it. Fork forks a branch without switching.
1 | machine switch [-h] [--no-wait] run [branch ...] |
| Argument | Does |
|---|---|
run |
The running machine that switches |
[branch ...] |
Branch to switch onto: a name, a mark written %mark, parent [n], or root |
| Option | Does |
|---|---|
--no-wait |
Return without waiting |
Examples:
1 | machine switch run1 try2 switch run1's machine onto try2 |
See also:
machine rewind
Put a run back at a point and discard what follows. See Switching the machine.
The run keeps its id and its name. The entries, checkpoints and console output after the point are discarded.
The point is required. It is any entry from #1 to the tip. Write an entry id after #, a checkpoint name after ^, or a machine time. The run lands on it exactly.
The point becomes a checkpoint. Rewinding to the tip changes nothing unless a replay diverged there. A point before the run was forked puts the run back along the line it was forked from. A run with a branch forked after the point is refused.
1 | machine rewind <run> to <point> |
| Argument | Does |
|---|---|
run |
The machine that rewinds |
[point ...] |
Point to go back to, from #1 to the tip: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, now, parent [n], root, or %mark |
Examples:
1 | machine rewind run1 to ^cp1 put run1 back at checkpoint cp1 |
Refuses, and the way past:
1 | a branch forked after the point machine delete try2 |
See also:
machine detach
Detach a new root from a run and start it. See Switching the machine.
Without --at, the root is detached at the current point. The copy includes
everything the root still owns. It shares nothing with the origin tree. It is
slower than fork --rebase.
The new root is named for the run it copied and the point. A second copy of the same point gets a suffix.
The new root keeps the execution options of the run it copied. Options given here replace them for the root.
The new run keeps and drops the output that the run it comes from does. Use
--keep-output to replace the names it keeps. Use --drop-output to replace
the names it drops. Each option changes its own list only.
Use --from to replay the new run from an earlier point to its point. The new
run stores the output of that stretch by its own names. It is forked at the
checkpoint at or before the earlier point.
A run stores every source but the files under /output/drop. A file there has a name and no bytes. A program writes under /output/drop through MACHINE_DROP_OUTPUT_DIR. It writes under /output/keep through MACHINE_OUTPUT_DIR.
A name given to either option may be a pattern, such as /output/drop//.log. A star matches within one directory name, a leading dot included, and never a slash. Two stars match any number of directories.
A name given to --keep-output or --drop-output decides before the directory
does. When a kept name and a dropped name cover one source, a name spelled out
decides over a pattern. Of two names spelled out, the longer decides, and a tie
keeps the source. Of two patterns, the kept one decides.
The name none, given alone, is an empty list. Use it to keep or to drop nothing where the run would have the names of another run.
1 | machine detach [-h] [--at point [point ...]] [--from point [point ...]] [--name name] |
| Argument | Does |
|---|---|
run |
Run to detach from |
| Option | Does |
|---|---|
--at point [point ...] |
Point the new run is at: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, now, parent [n], root, or %mark |
--from point [point ...] |
Earlier point to replay the new run from: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, parent [n] or root. The new run starts at the checkpoint at or before it |
--name name |
Name for the new run (default: a name made from where it starts) |
--no-start |
Create the new run without starting it |
--side-dump, --no-side-dump |
Record a per-entry debug dump, whatever the run it comes from records |
--side-trace, --no-side-trace |
Record the hypervisor’s trace of each entry, for diagnosis, whatever the run it comes from records |
--mem-hash, --no-mem-hash |
Hash guest memory at every entry, whatever the run it comes from records |
--repeatable, --no-repeatable |
Stop fast exactly at the limit, so recording again gives the same entries, whatever the run it comes from does |
--limit-margin n |
Conditional branches short of a stop where the CPU’s counter interrupts (default: the run’s) |
--timer-slack duration |
How late a timer may be in code that takes no exit, 0 for none (default: the run’s) |
--instr-margin n |
Instructions short of a stop placed by an instruction count where the CPU’s counter interrupts, 0 for the host’s (default: the run’s) |
--dispatch-bound duration |
Host time a fast dispatch may run before it stops, 0 for 1s. A stop it places does not repeat (default: the run’s) |
--fast-backstop n |
Instructions a fast dispatch may run with no other stop, 0 for 100000000 (default: the run’s) |
--irq-wait n |
Conditional branches a dispatch runs while an interrupt waits for the guest to take it, 0 for 200 (default: the run’s) |
--disk-hash, --no-disk-hash |
Hash the guest’s disk at every entry, whatever the run it comes from records |
--rng-seed n |
Seed the new run’s RNG device from n once it is up, so its guest draws other random numbers |
--keep-output name |
Source that the run stores though it is dropped: a path under /output, a device such as ttyS1, a pattern, or none (repeatable) |
--drop-output name |
Source that the run does not store: a path under /output, a device such as ttyS1, a pattern, or none (repeatable) |
--no-wait |
Return without waiting |
--daemon, --no-daemon |
Wait for commands instead of running to completion (default: the run’s) |
--single-step, --no-single-step |
Run in step mode, with --no-daemon. A run that waits for commands takes its mode from machine run --mode (default: the run’s) |
--limit n |
Conditional branches per dispatch in fast mode, steps in step mode, 0 to leave it to the machine (default: the run’s) |
--backstop-limit n |
Step modes only: machine instructions a dispatch may run without stepping before it ends, 0 for the machine default (default: the run’s) |
--pin-cpu n |
Pin to this host CPU (default: the least-loaded) |
Examples:
1 | machine detach run1 --at ^cp1 copy run1 at cp1 to a new root |
See also:
machine checkpoints
List the checkpoints of a branch. See Checkpoints.
Each checkpoint is a point for fork --at, go and rewind.
The listing shows the first 25 checkpoints. Use --limit for a different page
size. No running machine is needed.
The durable column says whether a checkpoint is synced to shared storage. Use -o json for an object of run_id, hosted and checkpoints instead of a bare list.
1 | machine checkpoints [-h] [--durable] [--since date|age] [--until date|age] [--sort field] |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--durable |
Show only checkpoints synced to shared storage |
--since date|age |
Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--until date|age |
Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--sort field |
Sort field: entry, date, name (default: entry) |
--reverse |
Reverse sort order |
--limit n |
Show at most n checkpoints (default: 25) |
--offset n |
Skip the first n checkpoints |
-w, --watch |
Watch for changes |
Examples:
1 | machine checkpoints run1 list the checkpoints of run1 |
Prints:
1 | text NAME ID VTIME DURABLE ANCESTOR |
See also:
machine marks
List and manage marks. See Marks.
A mark is a second name for a branch. It is unique within its tree. Written %mark, it names the branch anywhere a run is named.
Without a command, marks of every tree are listed. Use list with a run for the marks of that run’s tree. No running machine is needed.
1 | machine marks [-h] command ... |
Examples:
1 | machine marks list list marks |
See also:
machine marks list
List marks. See Marks.
The listing covers every tree, or the tree of one run when a run is named.
1 | machine marks list [-h] [run] |
| Argument | Does |
|---|---|
[run] |
Run name |
Examples:
1 | machine marks list marks of every tree |
See also:
machine marks add
Put a mark on a branch. See Marks.
Marking a branch again with the same word succeeds. A word that another branch in the tree already holds is refused. The mark stays with that branch.
1 | machine marks add [-h] run mark |
| Argument | Does |
|---|---|
run |
Run name |
mark |
The mark to add |
Examples:
1 | machine marks add run1 good put the mark good on run1 |
See also:
machine marks remove
Take a mark off a branch. See Marks.
The command is refused when the branch does not carry the mark.
1 | machine marks remove [-h] run mark |
| Argument | Does |
|---|---|
run |
Run name |
mark |
The mark to remove |
Examples:
1 | machine marks remove run1 good take the mark good off |
See also:
machine branches
Show the branch tree. See The tree.
The tree shows how each branch forked from its parent. Use --running to show
only running branches.
The listing shows the newest 25 branches. Use --limit for a different page
size. Use --depth to keep one level of the tree, 0 for the roots. Naming any
run in a tree, root or branch, limits the listing to that tree.
1 | machine branches [-h] [--running] [--depth n] [--since date|age] [--until date|age] |
| Argument | Does |
|---|---|
[run] |
Show only this run’s tree |
| Option | Does |
|---|---|
--running |
Show only running branches |
--depth n |
Show only branches at this depth from the root, 0 is the root |
--since date|age |
Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--until date|age |
Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--sort field |
Sort field: created, name (default: created) |
--reverse |
Reverse sort order |
--limit n |
Show at most n branches (default: 25) |
--offset n |
Skip the first n branches |
-w, --watch |
Watch for changes |
Examples:
1 | machine branches the branch tree |
Prints:
1 | each run <run> (<id>) [<first entry>, <end>) |
See also:
machine lineage
Show the ancestors of a branch, from the root down. See The tree.
1 | machine lineage [-h] run |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine lineage try2 the ancestors of try2 |
Prints:
1 | each ancestor <run> (<id>) [<first entry>, <end>) <n> checkpoints |
See also:
machine entries
Show run log entries. See Inspecting a run.
An entry is one recorded event of the run, such as a vCPU run, an interrupt or a device input.
Use -n to select entries, and -f to follow new output.
1 | machine entries [-h] [-n range] [-f] run |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
-n, --entries range |
Entries: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end |
-f, --follow |
Follow new output (like tail -f) |
Examples:
1 | machine entries run1 -n -20 the last 20 entries |
Prints:
1 | #<id> the entry, then its fields as name=value |
See also:
machine runs
List runs, newest first. See Inspecting a run.
Options filter by state, label, flag, disk, sync state and age.
The listing shows the newest 25. Use --limit for a different page size. Naming
a run limits the listing to its tree. Use --roots for one row per tree, and
--wide for more columns.
1 | machine runs [-h] [--state [^]state] [--running] [--roots] [--label name=value] |
| Argument | Does |
|---|---|
[root] |
Show one tree, by its root |
| Option | Does |
|---|---|
--state [^]state |
Filter by state: running, paused, recovering, read-only, stopped, killed, terminated, died, failed (repeat = OR; ^ excludes) |
--running |
Show only running runs, the same as --state running |
--roots |
One row per tree instead of every run, with its running N/M |
--label name=value |
Filter by a label (repeatable; matches all) |
--flag name[=value] |
Filter by an option the run was created with, such as daemon or rng; name=0 for one it was not (repeatable; matches all) |
--disk name|id |
Filter by the disk a run was created from |
--parents mode |
Filter by where a run reads the runs it branched from: local for this session, remote for shared storage |
--synced |
Show only runs published to shared storage |
--not-synced |
Show only runs with nothing published |
--since date|age |
Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--until date|age |
Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--sort field |
Sort field: created, name, cpus, memory, rng-seed, limit (default: created) |
--reverse |
Reverse sort order |
--limit n |
Show at most n runs (default: 25) |
--offset n |
Skip the first n runs |
--no-trunc |
Show full run ids |
--wide |
Show the DEPTH, DURABLE and PIN columns |
--size |
Measure and show size |
-w, --watch |
Watch for changes |
Examples:
1 | machine runs the newest runs |
Prints:
1 | text NAME STATE CPUS MEMORY DISK OPTIONS SESSION CREATED ID |
See also:
machine describe
Show the full configuration of a run. See Inspecting a run.
The configuration covers the state, resources, options and lineage of the run.
1 | machine describe [-h] run |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine describe run1 configuration and lineage of run1 |
Prints:
1 | text Name ID State Started Resources Options Lineage |
See also:
machine state
Show the state of a run and of its machine. See Inspecting a run.
The command shows the activity, machine time, entry and resources of the machine.
1 | machine state [-h] run |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine state run1 state, activity and position of run1 |
Prints:
1 | running state activity vtime entry cpus vcpu memory host cpu |
See also:
machine now
Show where a run is. See Inspecting a run.
The entry and the vtime are points for go, rewind, fork --at and detach
--at. Nothing moves.
Machine time does not move for work too short to count. A +N after the time tells apart the entries that share it. 0.2s is the first entry at that time and 0.2s+2 is the third.
A time with +N is a point like any other. It names exactly one entry.
1 | machine now [-h] run |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine now run1 entry, vtime and RIP of run1 |
Prints:
1 | text vtime=<time> entry=<id> vcpu=<n> rip=<address> <event> hash=<hash> |
See also:
machine output
Show the output of a run. See Inspecting a run.
The output is everything the machine printed, in the order it was printed. Each line has a source. The console is one source, and it is the one shown.
The serial ports ttyS1 to ttyS3 are sources too. A program in the guest that
writes to /dev/ttyS1 prints on ttyS1. Use --source to read another source. The
sources subcommand lists the sources that printed.
An output file is a source, named by its path. A program in the guest writes its files under the directory that MACHINE_OUTPUT_DIR names. A file events.jsonl that the service app writes there is /output/keep/app/events.jsonl. The path of a directory is every file under it.
The directory that MACHINE_DROP_OUTPUT_DIR names is for output too large to
store. A file under it has a name and no bytes. A branch forked with
--keep-output stores the file when replayed.
The name all is every source in the order printed, each line after the name of its source. A source that prints ends a line another source left unfinished.
Use --entries to select the output of given entries, -n to select lines, and
-f to follow new output. An entry that printed nothing contributes nothing.
The command prints lines and does not search them. Pipe it to grep to find text, or to less to page through it. Use -n : to hand them every line.
On a terminal each line shows the entries and the source that printed it, and
its number. ENTRIES is the first and the last entry of the line. Into a pipe or
a file the lines are printed alone. Use --wide or --plain to choose.
The entry subcommand gives the one entry that printed a byte. Use it for the exact entry of a match.
1 | machine output [-h] [-n range] [--entries range] [-f] [--wide] [--plain] [--source name] |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
-n, --lines range |
Lines: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end |
--entries range |
Output of entry N, or of entries A:B, both ends included |
-f, --follow |
Follow new output (like tail -f) |
--wide |
Show the entries and the source that printed each line, and its number (default on a terminal) |
--plain |
Print the lines alone (default into a pipe or a file) |
--source name |
Source to read: console, all, ttyS0 to ttyS3, hvc0, debugcon, or the path of an output file or directory under /output (default: console) |
Examples:
1 | machine output run1 the last lines |
Prints:
1 | text ENTRIES SOURCE LINE OUTPUT on a terminal |
See also:
machine output sourcesmachine output bytesmachine output entrymachine replaymachine entriesmachine gomachine debug
machine output entry
Show the entry that printed a byte of the output. See Finding a line.
A line is printed over a range of entries. A byte is printed by one. The command takes the offset of a byte and prints its entry.
The offset counts bytes from 0 over what the console printed. The grep option -b prints the offset of a match. The first number of its output is the operand.
Use --source for a byte another source printed. The offset then counts within
the output of that source.
The entry is a point. Use it with go, fork, rewind and the other commands that take an entry.
1 | machine output run entry [-h] [--source name] offset |
| Argument | Does |
|---|---|
offset |
Byte offset in the output of the source, counted from 0 |
| Option | Does |
|---|---|
--source name |
Source to read: console, all, ttyS0 to ttyS3, hvc0, debugcon, or the path of an output file or directory under /output (default: console) |
Examples:
1 | machine output run1 entry 48213 the entry of byte 48213 |
Prints:
1 | text the entry, alone |
Refuses, and the way past:
1 | an offset past the end machine output run1 -n : |
See also:
machine output sources
List the sources that printed and how much each printed.
A source is a serial port such as ttyS1, the virtio console, or an output file. The command prints a row for each source that printed, in the order each first printed.
The lines of a source are counted in its own output. Use the name with
--source to read that source.
An output file with nothing stored is listed after them with 0 bytes. It is a
file the run drops, or a file that was not written. Use its name with
--keep-output on a fork to store it.
The list is made where the output is kept, so the output is not downloaded. The output of the ancestors of the run is counted.
1 | machine output run sources [-h] |
Examples:
1 | machine output run1 sources each source that printed |
Prints:
1 | text SOURCE BYTES LINES ENTRIES, a row for each source |
See also:
machine output bytes
Write the bytes a source printed, exactly as printed.
The command writes the bytes to standard output and changes none. Use it for an output file that is not text.
With the json format the command prints a row for each write instead of the bytes. A row has the entry that wrote, the source, the offset and the length. The offset counts within the output of the source of the row. A row also names the task that wrote.
The bytes are read where the output is kept. The output of the ancestors of the run comes first.
1 | machine output run bytes [-h] [--source name] |
| Option | Does |
|---|---|
--source name |
Source to read: console, all, ttyS0 to ttyS3, hvc0, debugcon, or the path of an output file or directory under /output (default: console) |
Examples:
1 | machine output run1 bytes --source /output/keep/app/a.bin a file |
Prints:
1 | text the bytes, unchanged |
See also:
machine output search
Find a pattern in the output and the entry of each match. See Finding a line.
The pattern is a regular expression. A match is within a line. Each match is printed with the entry that printed its first byte and the time of that entry.
A match has at least one character, since an entry printed it. A pattern that matches without any text is refused. The pattern 6* is one, and 6+ is not.
The search is done where the output is kept, so a large output is not downloaded. Its lineage is searched, from the first line an ancestor printed.
Use --from and --to to search the output between two points. A match begins
at --from or after it and ends at --to or before it.
Use -f to go on with what is printed next. Use --max to stop after a number of
matches.
The console is searched. Use --source to search another source. With all,
every source is searched and a match is still within the line of one source.
The entry and the time are points. Use either with go, fork, rewind and the other commands that take a point.
1 | machine output run search [-h] [-f] [--max n] [--from point [point ...]] |
| Argument | Does |
|---|---|
regex |
Regular expression to find |
| Option | Does |
|---|---|
-f, --follow |
Go on with new output (like tail -f) |
--max n |
Stop after n matches |
--from point [point ...] |
Point the search begins at: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, parent [n] or root (default: the start of the output) |
--to point [point ...] |
Point the search ends at, as --from takes one (default: the end of the output) |
--source name |
Source to read: console, all, ttyS0 to ttyS3, hvc0, debugcon, or the path of an output file or directory under /output (default: console) |
Examples:
1 | machine output run1 search login each match and its entry |
Prints:
1 | text ENTRY TIME SOURCE LINE:COL TEXT, a row for each match |
Refuses, and the way past:
1 | a pattern that matches no text, as 6* machine output run1 search '6+' |
See also:
machine dump
Dump the state of a machine. See Inspecting a run.
The sections cover vCPU code, registers, stack and page tables, timers,
interrupt controllers, devices, memory and hashes. Use --show to choose
sections. Use --entry to dump the state recorded at an entry.
Without --show, every section the machine has is printed. Use --mem to print
memory alone, or with --show beside other sections. Use -o json for one object
per section.
The hash section is what the entry recorded, before its exit was handled. Memory and the disk are 0 there unless the run hashes them. Use machine hash for the hashes after the entry, with its exit handled.
1 | machine dump [-h] [--entry n] [--show sections] [--mem addr] [--vcpu n] run |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--entry n |
Entry whose recorded state to dump, instead of the state now |
--show sections |
One or more, comma-separated: code, regs, stack, pt, instr, trace, events, timers, debug, xsave, irq, pic, ioapic, devs, mem, disk, hash, entry, io, msrs, fullhash, all |
--mem addr |
Dump guest memory at an address: pADDR physical, vADDR virtual in the page tables of the vCPU |
--vcpu n |
The vCPU whose state and page tables are read (default: 0) |
Examples:
1 | machine dump run1 vCPU state at the current entry |
See also:
machine hash
Show the state hashes of the machine. Each is computed as it stands. See Comparing two machines.
The command prints a hash of memory. It prints a hash of the disk. Each device
has a hash too. The interrupts held by --irq-delay have their own hash,
irq_pending.
Each vCPU has a hash of each part of its state. Each vCPU also has a hash of the whole machine. Each hash compares with another machine’s at the same point, such as a replay’s.
A hash is not the one dump --show hash prints. Dump prints the hashes the
newest entry recorded, before its exit was handled. This command computes them
after that entry, with its exit handled. Memory and the disk are in its hashes
even when the entry hashed neither.
The hashes are computed whether or not the run hashes memory and the disk at each entry. Memory is read from all of RAM.
The vCPU hashes are of each vCPU now on a host kernel that computes them. Elsewhere they are of each vCPU’s last exit. The command then says so.
The command changes nothing. The run records nothing, and the machine stays where it is.
The command waits until every hash is computed, however long reading RAM takes. The machine refuses commands that would change it until then. Use Ctrl-C or machine cancel to stop it.
1 | machine hash [-h] [--vcpu n] run [hashes] |
| Argument | Does |
|---|---|
run |
Run name |
[hashes] |
One or more, comma-separated: vcpu, devices, mem, disk, irq_pending, or a component such as regs or rtc (default: all) |
| Option | Does |
|---|---|
--vcpu n |
Show the hashes of this vCPU alone |
Examples:
1 | machine hash run1 every hash of the machine now |
See also:
machine vcpu
Show what a vCPU is doing and how its last dispatch ended.
A dispatch is one entry into the machine. It runs until its limit or until the machine exits to the host.
1 | machine vcpu [-h] [--vcpu n] run |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--vcpu n |
The vCPU (default: 0) |
Examples:
1 | machine vcpu run1 run state of vCPU 0 |
Prints:
1 | first line entry ready pending iter |
See also:
machine ops
List operations on runs and how they finished. See Inspecting a run.
The listing shows the newest 25 operations in every state. Use --limit for a
different page size. Use --state to show some states.
An operation outlives the command that started it. One interrupted with Ctrl-C is still listed here by its id. The next start of a run finishes its unfinished rewind.
Operations on the account’s storage are listed by machine storage ops.
1 | machine ops [-h] [--op-id id] [--state [^]state] [--type [^]type] [--since date|age] |
| Argument | Does |
|---|---|
[run] |
Show operations on one run |
| Option | Does |
|---|---|
--op-id id |
Show one operation, by its id (ignores --state and --type) |
--state [^]state |
Filter by state: running, unfinished, done, failed (default: all; repeat = OR; ^ excludes) |
--type [^]type |
Filter by type: checkpoint, branch, switch, rewind, start, delete (default: all; repeat = OR; ^ excludes) |
--since date|age |
Since a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--until date|age |
Until a date such as 2026-06-01, or an age such as 7d, 24h or 30m |
--limit n |
Show at most n operations (default: 25) |
--offset n |
Skip the first n operations |
--no-trunc |
Show full run ids |
-w, --watch |
Watch for changes |
Examples:
1 | machine ops the newest operations |
See also:
machine debug
Show debug or error output from a run. See Inspecting a run.
The log subcommand shows the debug output. The err subcommand shows errors and crashes.
1 | machine debug [-h] command ... |
Examples:
1 | machine debug log run1 the log of run1 |
See also:
machine debug log
Show the debug log of a run. See Inspecting a run.
Use -n to select lines, and -f to follow new output.
1 | machine debug log [-h] [-n range] [-f] run |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
-n, --lines range |
Lines: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end |
-f, --follow |
Follow new output (like tail -f) |
Examples:
1 | machine debug log run1 the log of run1 |
See also:
machine debug err
Show the error and crash log of a run. See Inspecting a run.
Use -n to select lines, and -f to follow new output.
1 | machine debug err [-h] [-n range] [-f] run |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
-n, --lines range |
Lines: N for one, A:B for A through B, A: from A on, :B up to B, : for all, -N or -N: for the last N, and -A:-B counted from the end |
-f, --follow |
Follow new output (like tail -f) |
Examples:
1 | machine debug err run1 the error log of run1 |
See also:
machine tasks
List the tasks the guest runs. See Looking at tasks.
A task is what the guest kernel schedules: a thread of a program, or the kernel’s own. Each task has a tid. Its tgid names its thread group, the process it belongs to.
Each row is one task, by its tid, the thread id ps -L shows. Its tgid is the pid
of its process. Use --vcpu to list the tasks of one vCPU. Use --focused to
list the tasks that focus steers.
The subcommands steer the scheduler, so an order of tasks can be tried. The hold, yield and release subcommands control whether a task runs.
The slice subcommand sets how long a task runs. The pin subcommand places it on a vCPU. The cgroups subcommand groups the list. The exits subcommand lists the tasks that left.
1 | machine tasks [-h] [--vcpu n] [--focused] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--vcpu n |
Only tasks on this vCPU |
--focused |
Only the tasks focus is steering |
Examples:
1 | machine tasks run1 the tasks in run1's machine |
Prints:
1 | TID TGID the task, and the tgid it belongs to |
Refuses, and the way past:
1 | a machine not yet up machine run run1 --until tasks |
See also:
machine tasks list
List the tasks.
The listing is the one the command prints with no verb. Use --vcpu to list the
tasks of one vCPU.
1 | machine tasks run list [-h] [--vcpu n] [--focused] |
| Option | Does |
|---|---|
--vcpu n |
Only tasks on this vCPU |
--focused |
Only the tasks focus is steering |
Examples:
1 | machine tasks run1 list every task |
See also:
machine tasks cgroups
List the tasks grouped by cgroup. See Holding threads.
Use --vcpu to list the tasks of one vCPU.
1 | machine tasks run cgroups [-h] [--vcpu n] |
| Option | Does |
|---|---|
--vcpu n |
Only tasks on this vCPU |
Examples:
1 | machine tasks run1 cgroups tasks grouped by cgroup |
See also:
machine tasks exits
List the tasks that exited and how each one ended. See Holding threads.
The list is kept across checkpoints and replays. It is bounded and says how many it dropped.
1 | machine tasks run exits [-h] |
Examples:
1 | machine tasks run1 exits tasks that left, newest last |
Prints:
1 | text TID TGID COMM OUTCOME |
See also:
machine tasks exist
Assert that the machine runs a program, a comm, a tid or a tgid.
The command lists the tasks that match and is refused when none does. It takes
the words inject add takes for --in, except all.
A program is the absolute path of the file it runs, the EXE that tasks shows. A symlink’s path matches no task.
The command reads the task table. It changes nothing in the machine and the run records nothing.
1 | machine tasks run exist [-h] path|comm=name|tid=n|tgid=n |
| Argument | Does |
|---|---|
path|comm=name|tid=n|tgid=n |
Program by absolute path, comm, tid or tgid |
Examples:
1 | machine tasks run1 exist /usr/bin/app is /usr/bin/app running |
See also:
machine tasks hold
Take a task off the runqueue. See Holding threads.
The scheduler cannot pick a held task. A hold takes effect at the next scheduling decision of the vCPU. Holding the last runnable task leaves nothing to run. Release it, then use the irq command with preempt to go on.
1 | machine tasks run hold [-h] tid |
| Argument | Does |
|---|---|
tid |
Task, by its tid |
Examples:
1 | machine tasks run1 hold 56 hold tid 56 |
See also:
machine tasks yield
Put a task behind its peers. See Holding threads.
The task stays runnable. It does nothing when no peer is runnable.
1 | machine tasks run yield [-h] tid |
| Argument | Does |
|---|---|
tid |
Task, by its tid |
Examples:
1 | machine tasks run1 yield 56 put tid 56 behind its peers |
See also:
machine tasks release
Undo a hold or a yield. See Holding threads.
A release takes effect at the next scheduling decision of the vCPU. Use the irq command with preempt to force one.
1 | machine tasks run release [-h] tid |
| Argument | Does |
|---|---|
tid |
Task, by its tid |
Examples:
1 | machine tasks run1 release 56 undo hold and yield |
See also:
machine tasks slice
Set how long a task runs between decisions. See Holding threads.
A decision is a scheduling decision of the vCPU. Give a duration with a unit, such as 10us, 0.6ms or 750000ns. Use default for the kernel’s own. A slice survives release.
1 | machine tasks run slice [-h] tid duration|default |
| Argument | Does |
|---|---|
tid |
Task, by its tid |
duration|default |
Duration with a unit, or default |
Examples:
1 | machine tasks run1 slice 56 600us run tid 56 for 600 us |
See also:
machine tasks pin
Place a task on one or more vCPUs when it next wakes. See Holding threads.
A set of vCPUs is one number, 1, or a range, 1-3, both ends included.
Free means any vCPU. None means no vCPU, which stops the task running.
Pin is a placement policy, not confinement. A task already queued elsewhere keeps that vCPU until it wakes.
1 | machine tasks run pin [-h] tid set|free|none |
| Argument | Does |
|---|---|
tid |
Task, by its tid |
set|free|none |
Set of vCPUs to place the task on, such as 1 or 0-2; free for any, none for no vCPU |
Examples:
1 | machine tasks run1 pin 56 2 place tid 56 on vCPU 2 |
See also:
machine tasks signal
Send a task a signal.
The guest sends the signal at its next entry from user mode, as GDB’s signal command delivers one. The task acts on it when it next runs. The run log keeps the request, so a replay sends it at the same instruction.
1 | machine tasks run signal [-h] tid signal |
| Argument | Does |
|---|---|
tid |
Task, by its tid |
signal |
Signal by name, such as SIGTERM or TERM, or by number |
Examples:
1 | machine tasks run1 signal 56 SIGTERM send tid 56 SIGTERM |
See also:
machine syscalls
List the system calls, by number. See Looking at tasks.
A name filters the list to calls whose name contains that text.
A task shows its syscall histogram instead. Name a task by tid, tgid, absolute
path, cgroup=path or comm=name. The histogram counts calls made while the focus
entry of the task was armed with focus add <path> +syscalls.
1 | machine syscalls [-h] run [name|tid|tgid|path] |
| Argument | Does |
|---|---|
run |
Run name |
[name|tid|tgid|path] |
Filter the list, or a task to show its histogram |
Examples:
1 | machine syscalls run1 the system calls of the machine |
See also:
machine ints
List the interrupt vectors. See Looking at tasks.
One row per IDT vector that the machine has a handler for. The vector is in hex, as the table is indexed. A name filters to handlers whose name contains that text.
A task shows its interrupt histogram instead. Name a task by tid, tgid, absolute
path, cgroup=path or comm=name. The histogram counts interrupts taken while the
focus entry of the task was armed with focus add <path> +ints.
1 | machine ints [-h] run [name|tid|tgid|path] |
| Argument | Does |
|---|---|
run |
Run name |
[name|tid|tgid|path] |
Filter the list, or a task to show its histogram |
Examples:
1 | machine ints run1 the interrupt vectors of the machine |
See also:
machine focus
Show and change which tasks a machine steers. See Focus.
The focus list names the programs the machine steers. A run stops at each context switch of a focused program, and where the program reaches its own code. The tasks subcommands then decide which task runs next.
A program is named by its absolute path, matched exactly. Write cgroup=<path>
to match the cgroup of a task. Write tgid=<n> to name a tgid. The tgid of a
task is shown by tasks.
Write comm=<name> to focus each process with a task of that comm, as inject
add and tasks exist read it. A comm names a busybox applet or a script, which a
path cannot. A task renamed after exec is seen at the next fork, exec or exit.
Use add and remove to change the list. Use clear to empty it.
1 | machine focus [-h] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine focus run1 what is focused |
Prints:
1 | Focus each program or cgroup named, and what is counted for it |
See also:
machine focus add
Steer another program. See Focus.
The entry is a program path, cgroup=<path>, comm=<name> or tgid=<n>. Add
+syscalls or +ints to count its system calls or interrupts.
1 | machine focus run add [-h] path|cgroup=path|comm=name|tgid=n [+syscalls|+ints ...] |
| Argument | Does |
|---|---|
path|cgroup=path|comm=name|tgid=n |
Program path, cgroup, comm, or tgid to match |
[+syscalls|+ints ...] |
Count this program’s syscalls or interrupts |
Examples:
1 | machine focus run1 add /test focus the program /test |
See also:
machine focus remove
Stop steering a program. See Focus.
Name the entry as it was added, by program path, cgroup=<path>, comm=<name>
or tgid=<n>.
1 | machine focus run remove [-h] path|cgroup=path|comm=name|tgid=n |
| Argument | Does |
|---|---|
path|cgroup=path|comm=name|tgid=n |
Program path, cgroup, comm, or tgid to match |
Examples:
1 | machine focus run1 remove /test stop focusing /test |
See also:
machine focus clear
Empty the list of steered programs. See Focus.
1 | machine focus run clear [-h] |
Examples:
1 | machine focus run1 clear focus nothing |
See also:
machine send
Send input to a device of a run.
The command writes bytes to a serial port, the virtio console or the keyboard of a run. A program in the guest reads them from the device, as it reads what a person types.
The text is sent as written, and a newline follows it, which is Enter. Use -n to send no newline. Use -e to read escapes in the text, so that Ctrl-C is \x03.
The text - sends standard input as it is. The command adds no newline to it and reads no escape in it.
At the control prompt the command is /send. The text is the rest of the line after the options, with its quotes. A semicolon inside quotes does not end the command. A pair of quotes around the whole text keeps the spaces at its ends.
The keyboard takes text and types it on a US keyboard. The machine presses and releases a key for each character. It holds Shift or Ctrl where the character needs one.
A device that begins with a slash is the path of a file in the machine. The bytes go at the end of the file. A pipe is such a file. A program that reads its input from a pipe is sent input this way.
A process of the machine writes the file, and the command prints its id. The exec proc subcommand says whether the bytes were written.
The command does not run the machine. It queues the bytes and returns. The device takes them the next time the run executes, and the run records them there. A replay reproduces them.
Use fork to send other input than a run recorded. The branch takes its own.
1 | machine send [-h] [-n] [-e] [--device name] run text |
| Argument | Does |
|---|---|
run |
Run name |
text |
Text to send, or - to send standard input as it is |
| Option | Does |
|---|---|
-n |
Send no newline after the text |
-e |
Read the escapes \n, \r, \t, \b, \e, \0, \ and \xNN in the text |
--device name |
Device to send to: console, ttyS0 to ttyS3, hvc0 or keyboard, or the path of a file in the machine (default: console) |
Examples:
1 | machine send run1 ls type ls and Enter on the console |
Prints:
1 | text Queued <n> bytes for <device> of <run> (<id>) |
Refuses, and the way past:
1 | a machine that is replaying machine fork run1 --name try2 |
See also:
machine exec
Execute a command in the guest.
A command is a program of the machine’s disk and its arguments, written after two dashes. No shell reads it. Use -c to give a line to the shell of the disk, with its pipes and redirections.
The run subcommand executes it and runs the machine until it exits. The start subcommand sends it and returns.
Each command is a process with an id. The run records what the process prints and how it ends, each line at its entry. A replay reproduces them, and a branch has the processes of its parent.
Without a subcommand the command lists the processes of the run. The proc subcommand shows one.
The id is the entry the command was sent at and the process number, 23090p1. In
the machine, its output is in /.machine/output/exec/ID/stdout and stderr. Its
events are in /.machine/output/exec/ID/status. The output source
/output/exec/ID/stdout names the first in output and run --until.
The machine’s disk runs the agent that executes the commands, /sbin/machine-exec. A disk without it executes none.
1 | machine exec [-h] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine exec run1 the processes of run1 |
Prints:
1 | text ID PID STATE ENTRIES COMMAND |
See also:
machine exec run
Execute a command and run the machine until it exits.
The command prints what the command printed, its standard output and its standard error each on its own. The exit status is that of the command.
Use --container to execute it in a container. The command then has the
environment of the container. Use --user to choose the user it runs as there.
Use --net to run a program of the machine on a container’s network, such as
tcpdump on its eth0. The program has the machine’s files and the container’s
interfaces.
The machine runs until the command ends. Other programs of the machine run with it.
A command gets 30 minutes of machine time to end. Use --within to give it
another. A command that has not ended by then stays, and the refusal names its
id.
1 | machine exec run run [-h] [-c line] [--container name] [--user user] [--net name] |
| Argument | Does |
|---|---|
[command ...] |
The command and its arguments, after – |
| Option | Does |
|---|---|
-c line |
A line for the shell of the machine’s disk, in place of a command |
--container name |
Container to execute it in, by its id, its name or its Compose service (default: the machine itself) |
--user user |
User the command runs as in the container, such as root or 1000:1000 |
--net name |
Container whose network a program of the machine runs on, by its id, its name or its Compose service |
--within duration |
Machine time the command gets to end in, such as 10s (default: 30m) |
Examples:
1 | machine exec run1 run -- ls -la / execute ls and print its output |
Prints:
1 | text what the command printed, and its exit status |
Refuses, and the way past:
1 | a machine that is replaying machine fork run1 --name try2 |
See also:
machine exec start
Send a command to execute and return.
The command prints the id of the process. The machine does not run. The process starts the next time the run executes.
Use the proc subcommand with the id to read the process.
Use --container to execute it in a container. Use --net to run a program of
the machine on a container’s network.
1 | machine exec run start [-h] [-c line] [--container name] [--user user] [--net name] |
| Argument | Does |
|---|---|
[command ...] |
The command and its arguments, after – |
| Option | Does |
|---|---|
-c line |
A line for the shell of the machine’s disk, in place of a command |
--container name |
Container to execute it in, by its id, its name or its Compose service (default: the machine itself) |
--user user |
User the command runs as in the container, such as root or 1000:1000 |
--net name |
Container whose network a program of the machine runs on, by its id, its name or its Compose service |
Examples:
1 | machine exec run1 start -- sleep 5 send it and return its id |
Prints:
1 | text Sent <id> to <run> (<id>) |
Refuses, and the way past:
1 | a machine that is replaying machine fork run1 --name try2 |
See also:
machine exec proc
Show a process: its state, its output and its events.
The summary has the command, the pid, the state and how the process ended. A process is pending until the machine runs and the guest takes the request.
The events view lists every line the process printed and every change of its state, each with its entry. The stdout and stderr views print one stream. The exits view prints how the process ended.
In the machine, the streams are /.machine/output/exec/ID/stdout and /.machine/output/exec/ID/stderr.
1 | machine exec run proc [-h] id [{summary,events,stdout,stderr,exits}] |
| Argument | Does |
|---|---|
id |
Process id |
[{summary,events,stdout,stderr,exits}] |
What to show: summary, events, stdout, stderr or exits (default: summary) |
Examples:
1 | machine exec run1 proc 40787p1 its state and how it ended |
Prints:
1 | summary Process Run Command Pid State Entries Stdout Stderr |
Refuses, and the way past:
1 | a process the guest has not taken machine run run1 |
See also:
machine put
Put a file into the guest or a container.
The command copies a file on this computer to a path in the running machine. Use
--container to put it in a container instead. A file of any size is put. The
directories on the way to the path are made.
The file has its name only once all of it is written, so a program never reads part of it. A file already at the path is replaced.
The machine runs while the file goes in. Other programs of the machine run with it. The bytes are a recorded input, so a replay puts the same file.
The file is written to the disk before the command returns. A checkpoint taken after it has the file, and fs reads it there.
1 | machine put [-h] [--container name] [--mode mode] [--owner uid:gid] [--no-pipe] |
| Argument | Does |
|---|---|
run |
Run name |
file |
File on this computer to put, or - for standard input |
path |
Path the file gets in the machine or in the container |
| Option | Does |
|---|---|
--container name |
Container to put the file in, by its id, its name or its Compose service (default: the machine itself) |
--mode mode |
Mode of the file, as octal digits (default: 644) |
--owner uid:gid |
Owner of the file, as numbers (default: root) |
--no-pipe |
Send the file over the serial port instead of the faster data channel |
--within duration |
Machine time allowed each time the command runs the machine, such as 10s (default: 30m) |
Examples:
1 | machine put run1 ./a.conf /etc/a.conf into the machine |
Prints:
1 | text Put <n> bytes at <path> of <run> (<id>) |
Refuses, and the way past:
1 | a machine that is replaying machine fork run1 --name try2 |
See also:
machine get
Get a file from the guest or a container.
The command copies a file of the running machine to a file on this computer. Use
--container to get it from a container instead. A file of any size is copied.
The file is read as it is now, with what programs wrote and the disk does not have yet. The fs command reads the disk at a checkpoint instead.
The machine runs while the file comes out. Other programs of the machine run with it, so a file a program is writing can change on the way.
1 | machine get [-h] [--container name] [--within duration] run path file |
| Argument | Does |
|---|---|
run |
Run name |
path |
Path of the file in the machine or in the container |
file |
File on this computer to write, or - for standard output |
| Option | Does |
|---|---|
--container name |
Container to get the file from, by its id, its name or its Compose service (default: the machine itself) |
--within duration |
Machine time allowed each time the command runs the machine, such as 10s (default: 30m) |
Examples:
1 | machine get run1 /etc/a.conf ./a.conf from the machine |
Prints:
1 | text Got <n> bytes from <path> of <run> (<id>) |
Refuses, and the way past:
1 | a machine that is replaying machine fork run1 --name try2 |
See also:
machine containers
List the containers, or stop, kill, crash, pause or limit one.
The command prints each container with its Compose service, its state and its image. Stopped containers are listed too. A container that no service started has no service.
A verb puts a fault on one container while the others run: a node of a cluster that stops, dies, loses power, freezes or starves. Throttle limits its CPU, and memory its memory. Pids, files and ports run it out of processes, open files and local ports.
A limit holds when the container starts again, until it is lifted. A container
that Compose recreates starts without it. A limit set with --for is set back
after the duration, whichever start of the container runs then.
Every command that takes a container takes its id, its name or its Compose service. An id is the surer way to name one. The machine knows an id as soon as Docker creates the container. It knows a name shortly after the container starts.
A running container is found with the machine stopped. A name not known yet is refused, and its id works. A service stands for its one running container. The start verb asks Docker for a stopped container.
A name is matched first, then a service, then an id. An id takes 12 hex digits or more, as docker ps prints it.
The machine runs until Docker has answered. The list is of the machine as it is then. The command runs processes in the machine, and the run records them, so a replay acts at the same point.
1 | machine containers [-h] [--within duration] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--within duration |
Machine time the command gets to end in, such as 10s (default: 30m) |
Examples:
1 | machine containers run1 the containers of run1 |
Prints:
1 | text SERVICE CONTAINER ID STATE IMAGE ADDRESS LINK |
Refuses, and the way past:
1 | a machine that is replaying machine fork run1 --name try2 |
See also:
machine containers stop
Stop a container.
The container’s processes get SIGTERM, and SIGKILL after the grace period. A program that handles SIGTERM exits on its own first.
The container stays down. Use --restart 5s to start it again after 5 seconds
of machine time. Use --restart 0 to start it at once. Other containers keep
running while it is down.
A container started again keeps its writable layer and its volumes. Use
--fresh to start it from its image instead, as Compose recreates a service.
Its volumes are kept.
1 | machine containers run stop [-h] [--grace duration] [--restart duration|never] [--fresh] |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
| Option | Does |
|---|---|
--grace duration |
Machine time between SIGTERM and SIGKILL, such as 5s (default: 10s) |
--restart duration|never |
Start it again after this machine time, 0 at once (default: never) |
--fresh |
Start it from its image, its writable layer discarded |
Examples:
1 | machine containers run1 stop db db stops |
Refuses, and the way past:
1 | a container that is not running or paused machine containers run1 start db |
See also:
machine containers kill
Kill a container.
Every process of the container gets SIGKILL and no cleanup runs. What the programs wrote stays, since the guest’s kernel writes it out.
The container stays down. Use --restart 5s to start it again after 5 seconds
of machine time. Use --restart 0 to start it at once. Other containers keep
running while it is down.
A container started again keeps its writable layer and its volumes. Use
--fresh to start it from its image instead, as Compose recreates a service.
Its volumes are kept.
1 | machine containers run kill [-h] [--restart duration|never] [--fresh] container |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
| Option | Does |
|---|---|
--restart duration|never |
Start it again after this machine time, 0 at once (default: never) |
--fresh |
Start it from its image, its writable layer discarded |
Examples:
1 | machine containers run1 kill db db dies |
Refuses, and the way past:
1 | a container that is not running or paused machine containers run1 start db |
See also:
machine containers crash
Cut a container as a power cut cuts a node.
The container’s volumes lose the writes they had not synced, and it dies. The others run on. A volume keeps what was synced, and loses the writes still in memory.
A crash cuts the volumes the project’s x-machine names, each a filesystem of its own. Name one by its path in the container to cut it alone. A volume replays its journal when the container starts again.
The container is frozen first, and the cut is when its volumes stop writing to the disk. A write the kernel finishes between the two is kept. A volume that another running container mounts is refused, since the cut would fail its writes too.
The container stays down. Use --restart 5s to start it again after 5 seconds
of machine time. Use --restart 0 to start it at once. Other containers keep
running while it is down.
A container started again keeps its writable layer and its volumes. Use
--fresh to start it from its image instead, as Compose recreates a service.
Its volumes are kept.
1 | machine containers run crash [-h] [--restart duration|never] [--fresh] container [path] |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
[path] |
A volume’s path in the container, to cut it alone (default: every volume) |
| Option | Does |
|---|---|
--restart duration|never |
Start it again after this machine time, 0 at once (default: never) |
--fresh |
Start it from its image, its writable layer discarded |
Examples:
1 | machine containers run1 crash db a power cut of db |
Refuses, and the way past:
1 | a container with no volume of its own machine containers run1 |
See also:
machine containers start
Start a container that is down.
Use --fresh to start it from its image, as Compose recreates a service. Its
writable layer is discarded and its volumes are kept.
1 | machine containers run start [-h] [--fresh] container |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
| Option | Does |
|---|---|
--fresh |
Start it from its image, its writable layer discarded |
Examples:
1 | machine containers run1 start db db runs again |
Refuses, and the way past:
1 | a container that is running machine containers run1 |
See also:
machine containers pause
Freeze a container.
The container’s processes are frozen using the cgroup freezer. Other containers
see it as unresponsive. Use --for 5s to thaw it after 5 seconds of machine
time. Use resume to thaw it now.
1 | machine containers run pause [-h] [--for duration] container |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
| Option | Does |
|---|---|
--for duration |
Thaw it after this machine time, such as 5s |
Examples:
1 | machine containers run1 pause db db freezes |
Refuses, and the way past:
1 | a container that is not running machine containers run1 |
See also:
machine containers resume
Thaw a frozen container.
1 | machine containers run resume [-h] container |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
Examples:
1 | machine containers run1 resume db db thaws |
Refuses, and the way past:
1 | a container that is not paused machine containers run1 |
See also:
machine containers list
List the containers.
The command prints each container with its Compose service, its state and its image. Each running one has its addresses, its links and the limits of its cgroup. The listing is the one the command prints with no verb.
1 | machine containers run list [-h] |
Examples:
1 | machine containers run1 list each container and its limits |
See also:
machine containers throttle
Limit the CPU of a container.
The container’s cgroup gets a CPU quota of the number of CPUs given. A number of 0 sets back the container’s own limit, its Compose file’s or none. The limit holds when the container starts again.
Use --for 30s to restore the previous limit after 30 seconds of machine time.
1 | machine containers run throttle [-h] --cpus n [--for duration] container |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
| Option | Does |
|---|---|
--cpus n |
CPUs it may use, such as 0.1; 0 for the container’s own limit |
--for duration |
Restore the previous limit after this machine time, such as 30s |
Examples:
1 | machine containers run1 throttle db --cpus 0.1 a tenth of a CPU |
Refuses, and the way past:
1 | a container that is not running machine containers run1 |
See also:
machine containers memory
Limit the memory of a container.
The container’s processes may use the memory given and no more. A size of 0 sets back the container’s own limit, its Compose file’s or none. The limit holds when the container starts again.
A limit below what they use makes the guest kernel reclaim memory. What it cannot reclaim, its out-of-memory killer takes: a process of the container is killed.
A soft limit, with --soft, is the memory past which the kernel reclaims and
slows the processes. It kills none of them, and the hard limit stays as it was.
Use --for 30s to restore the previous limit after 30 seconds of machine time.
1 | machine containers run memory [-h] [--soft] [--for duration] container size |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
size |
Memory it may use, such as 256m or 1g; 0 for the container’s own limit |
| Option | Does |
|---|---|
--soft |
Reclaim and slow its processes past the size, and kill none |
--for duration |
Restore the previous limit after this machine time, such as 30s |
Examples:
1 | machine containers run1 memory db 256m db may use 256MB |
See also:
machine containers pids
Limit the processes of a container.
The container’s processes and threads may number the count given and no more. A count of 0 sets back the container’s own limit, its Compose file’s or none. The limit holds when the container starts again.
Past the limit, a fork or a new thread in the container fails with EAGAIN. The processes already running go on.
Use --for 30s to restore the previous limit after 30 seconds of machine time.
1 | machine containers run pids [-h] [--for duration] container count |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
count |
Processes and threads it may have; 0 for the container’s own limit |
| Option | Does |
|---|---|
--for duration |
Restore the previous limit after this machine time, such as 30s |
Examples:
1 | machine containers run1 pids db 50 50 processes and threads |
See also:
machine containers ports
Narrow the ephemeral ports of a container.
A connection the container opens takes its local port from the range given. A range of 0 sets back the container’s own range, its Compose file’s or the kernel’s 32768-60999. The range holds when the container starts again.
Once every port of the range is taken for one destination, a connect to it fails with EADDRNOTAVAIL. Two connections to different destinations can share a port. A program that binds port 0 before it connects gets EADDRINUSE. A port stays taken after its connection closes, in TIME_WAIT for 60 seconds.
The range applies to every process on the container’s network. The command refuses a container on the machine’s network. It also refuses a container that uses another container’s network.
Use --for 30s to restore the previous range after 30 seconds of machine time.
1 | machine containers run ports [-h] [--for duration] container range |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
range |
Ports it may connect from, such as 32768-32770; 0 for the container’s own |
| Option | Does |
|---|---|
--for duration |
Restore the previous limit after this machine time, such as 30s |
Examples:
1 | machine containers run1 ports api 40000-40009 10 ports |
Refuses, and the way past:
1 | a container on the machine's network machine containers run1 |
See also:
machine containers files
Limit the open files of a container.
Each process of the container may hold open the count of files given and no more. A socket or a pipe counts as a file. A count of 0 sets back the container’s own limit.
Past the limit, an open, a socket or an accept fails with EMFILE. A file already open stays open.
The limit is set on the processes that run now, and what they start takes it. A process docker exec starts takes the container’s own limit. The limit holds when the container starts again.
Use --for 30s to restore the previous limit after 30 seconds of machine time.
1 | machine containers run files [-h] [--for duration] container count |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
count |
Files each process may hold open; 0 for the container’s own limit |
| Option | Does |
|---|---|
--for duration |
Restore the previous limit after this machine time, such as 30s |
Examples:
1 | machine containers run1 files api 64 64 open files |
See also:
machine net
Show the faults of each container’s link, or put faults on one.
A link connects a container to a peer. A fault acts on the packets the container sends. The command delays them, loses some, sends some twice, changes them, sends them out of order or limits their rate. It holds them and releases them together.
The command cuts two containers apart, one way or both. It resets their TCP connections. It shows and changes a container’s neighbour table.
A fault is on every packet the container sends, or on the packets a match takes. A match names a peer, a protocol, a port or TCP SYNs. ARP is a protocol a match names. A fault on one match leaves the container’s other links as they are.
The faults are part of the machine. The run records each change, and a replay makes it at the same point. A checkpoint keeps a link’s faults.
Every command that takes a container takes its id, its name or its Compose service, as containers does. The machine runs until the link is set.
1 | machine net [-h] [--within duration] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--within duration |
Machine time the command gets to end in, such as 10s (default: 30m) |
Examples:
1 | machine net run1 each link and its faults |
See also:
machine net delay
Delay the packets a container sends.
Each packet waits the delay before it leaves, give or take the jitter. A delay of 0 takes the delay away. The link keeps its other faults.
Use --to db to put the fault on what the container sends to db alone. Use
--proto, --port and --syn to take some packets alone, such as port 5432. A
packet a match takes gets only the match’s faults, not the whole link’s.
The link draws from a seed, so a replay acts on the same packets. Use --seed
to give one. A link keeps its seed, and a link without one draws a new seed.
Use --for 30s to set back what the command set after 30 seconds of machine
time. The link keeps a change made in between.
Use --log to write each decision of the fault to /output/drop/.machine/net, a
file for each container. The run drops that output unless it keeps it.
1 | machine net run delay [-h] [--jitter duration] [--to container] |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
delay |
Delay of each packet, such as 100ms |
| Option | Does |
|---|---|
--jitter duration |
Each packet’s delay differs by up to this, such as 20ms |
--to container |
The packets to this container alone (default: to every peer) |
--proto {tcp,udp,icmp,arp} |
The packets of this protocol alone |
--port n |
The packets with this source or destination port alone |
--syn |
The TCP SYNs that open a connection alone |
--seed n |
Seed of the draws (default: the link’s seed, or one drawn) |
--for duration |
Set the link back after this machine time, such as 30s |
--log, --no-log |
Write each decision of the fault to the container’s log (default: as the link has it) |
Examples:
1 | machine net run1 delay db 100ms db's packets wait 100ms |
See also:
machine net loss
Lose some of the packets a container sends.
Each packet is lost by chance, the share given. A share of 0% takes the fault away. The link keeps its other faults.
Use --to db to put the fault on what the container sends to db alone. Use
--proto, --port and --syn to take some packets alone, such as port 5432. A
packet a match takes gets only the match’s faults, not the whole link’s.
The link draws from a seed, so a replay acts on the same packets. Use --seed
to give one. A link keeps its seed, and a link without one draws a new seed.
Use --for 30s to set back what the command set after 30 seconds of machine
time. The link keeps a change made in between.
Use --log to write each decision of the fault to /output/drop/.machine/net, a
file for each container. The run drops that output unless it keeps it.
1 | machine net run loss [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] [--syn] |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
share |
Share of the packets, such as 10% |
| Option | Does |
|---|---|
--to container |
The packets to this container alone (default: to every peer) |
--proto {tcp,udp,icmp,arp} |
The packets of this protocol alone |
--port n |
The packets with this source or destination port alone |
--syn |
The TCP SYNs that open a connection alone |
--seed n |
Seed of the draws (default: the link’s seed, or one drawn) |
--for duration |
Set the link back after this machine time, such as 30s |
--log, --no-log |
Write each decision of the fault to the container’s log (default: as the link has it) |
Examples:
1 | machine net run1 loss db 10% a tenth of db's packets lost |
See also:
machine net duplicate
Send some of a container’s packets twice.
Each packet is sent twice by chance, the share given. A share of 0% takes the fault away. The link keeps its other faults.
Use --to db to put the fault on what the container sends to db alone. Use
--proto, --port and --syn to take some packets alone, such as port 5432. A
packet a match takes gets only the match’s faults, not the whole link’s.
The link draws from a seed, so a replay acts on the same packets. Use --seed
to give one. A link keeps its seed, and a link without one draws a new seed.
Use --for 30s to set back what the command set after 30 seconds of machine
time. The link keeps a change made in between.
Use --log to write each decision of the fault to /output/drop/.machine/net, a
file for each container. The run drops that output unless it keeps it.
1 | machine net run duplicate [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
share |
Share of the packets, such as 10% |
| Option | Does |
|---|---|
--to container |
The packets to this container alone (default: to every peer) |
--proto {tcp,udp,icmp,arp} |
The packets of this protocol alone |
--port n |
The packets with this source or destination port alone |
--syn |
The TCP SYNs that open a connection alone |
--seed n |
Seed of the draws (default: the link’s seed, or one drawn) |
--for duration |
Set the link back after this machine time, such as 30s |
--log, --no-log |
Write each decision of the fault to the container’s log (default: as the link has it) |
Examples:
1 | machine net run1 duplicate db 5% one packet in 20 sent twice |
See also:
machine net corrupt
Change a bit of some of the packets a container sends.
Each packet has one bit flipped by chance, the share given. A checksum finds most of them. A share of 0% takes the fault away. The link keeps its other faults.
Use --to db to put the fault on what the container sends to db alone. Use
--proto, --port and --syn to take some packets alone, such as port 5432. A
packet a match takes gets only the match’s faults, not the whole link’s.
The link draws from a seed, so a replay acts on the same packets. Use --seed
to give one. A link keeps its seed, and a link without one draws a new seed.
Use --for 30s to set back what the command set after 30 seconds of machine
time. The link keeps a change made in between.
Use --log to write each decision of the fault to /output/drop/.machine/net, a
file for each container. The run drops that output unless it keeps it.
1 | machine net run corrupt [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
share |
Share of the packets, such as 10% |
| Option | Does |
|---|---|
--to container |
The packets to this container alone (default: to every peer) |
--proto {tcp,udp,icmp,arp} |
The packets of this protocol alone |
--port n |
The packets with this source or destination port alone |
--syn |
The TCP SYNs that open a connection alone |
--seed n |
Seed of the draws (default: the link’s seed, or one drawn) |
--for duration |
Set the link back after this machine time, such as 30s |
--log, --no-log |
Write each decision of the fault to the container’s log (default: as the link has it) |
Examples:
1 | machine net run1 corrupt db 1% one packet in 100 changed |
See also:
machine net reorder
Send some of a container’s packets ahead of the others.
Each packet is sent at once by chance, the share given, while the rest wait the link’s delay. A link to reorder needs a delay. A share of 0% takes the fault away.
Use --to db to put the fault on what the container sends to db alone. Use
--proto, --port and --syn to take some packets alone, such as port 5432. A
packet a match takes gets only the match’s faults, not the whole link’s.
The link draws from a seed, so a replay acts on the same packets. Use --seed
to give one. A link keeps its seed, and a link without one draws a new seed.
Use --for 30s to set back what the command set after 30 seconds of machine
time. The link keeps a change made in between.
Use --log to write each decision of the fault to /output/drop/.machine/net, a
file for each container. The run drops that output unless it keeps it.
1 | machine net run reorder [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
share |
Share of the packets sent ahead, such as 25% |
| Option | Does |
|---|---|
--to container |
The packets to this container alone (default: to every peer) |
--proto {tcp,udp,icmp,arp} |
The packets of this protocol alone |
--port n |
The packets with this source or destination port alone |
--syn |
The TCP SYNs that open a connection alone |
--seed n |
Seed of the draws (default: the link’s seed, or one drawn) |
--for duration |
Set the link back after this machine time, such as 30s |
--log, --no-log |
Write each decision of the fault to the container’s log (default: as the link has it) |
Examples:
1 | machine net run1 reorder db 25% a quarter ahead of the delay |
See also:
machine net rate
Limit how fast a container sends.
The link sends at most the rate given, and the packets past it wait. A rate of 0 takes the limit away. The link keeps its other faults.
Use --to db to put the fault on what the container sends to db alone. Use
--proto, --port and --syn to take some packets alone, such as port 5432. A
packet a match takes gets only the match’s faults, not the whole link’s.
Use --for 30s to set back what the command set after 30 seconds of machine
time. The link keeps a change made in between.
Use --log to write each decision of the fault to /output/drop/.machine/net, a
file for each container. The run drops that output unless it keeps it.
1 | machine net run rate [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] [--syn] |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
rate |
Bits a second, such as 1mbit or 500kbit; 0 for no limit |
| Option | Does |
|---|---|
--to container |
The packets to this container alone (default: to every peer) |
--proto {tcp,udp,icmp,arp} |
The packets of this protocol alone |
--port n |
The packets with this source or destination port alone |
--syn |
The TCP SYNs that open a connection alone |
--for duration |
Set the link back after this machine time, such as 30s |
--log, --no-log |
Write each decision of the fault to the container’s log (default: as the link has it) |
Examples:
1 | machine net run1 rate db 1mbit db sends 1 megabit a second |
See also:
machine net partition
Cut the link between two containers.
Everything each container sends the other is lost, and their other links stay as
they are. Use --one-way to cut what the first sends the second alone. Use
--port 5432 to cut the packets of port 5432 alone.
Use --for 30s to set both directions back together after 30 seconds of machine
time.
1 | machine net run partition [-h] [--one-way] [--proto {tcp,udp,icmp,arp}] [--port n] [--syn] |
| Argument | Does |
|---|---|
container |
One container of the two |
other |
The other container |
| Option | Does |
|---|---|
--one-way |
Cut what the first sends the second, not the reverse |
--proto {tcp,udp,icmp,arp} |
The packets of this protocol alone |
--port n |
The packets with this source or destination port alone |
--syn |
The TCP SYNs that open a connection alone |
--for duration |
Set the link back after this machine time, such as 30s |
Examples:
1 | machine net run1 partition db api nothing between db and api |
See also:
machine net list
List each container’s links and their faults.
The listing is the one the command prints with no verb.
1 | machine net run list [-h] |
Examples:
1 | machine net run1 list each link and its faults |
See also:
machine net hold
Hold the packets a container sends.
Every packet waits in the link until a release, and then they all leave together. The link holds 1000 full-size packets and drops the ones past them.
Without a container the command holds every link at the instruction the machine is stopped at. All networking between the containers stops there, and a replay holds at the same point. A release without a container lets every packet go.
Use --to db to put the fault on what the container sends to db alone. Use
--proto, --port and --syn to take some packets alone, such as port 5432. A
packet a match takes gets only the match’s faults, not the whole link’s.
Use --for 5s with a container to release its packets after 5 seconds of
machine time. The hold of every link ends with a release.
1 | machine net run hold [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] [--syn] |
| Argument | Does |
|---|---|
[container] |
Container, by its id, its name or its service’s (default: every one) |
| Option | Does |
|---|---|
--to container |
The packets to this container alone (default: to every peer) |
--proto {tcp,udp,icmp,arp} |
The packets of this protocol alone |
--port n |
The packets with this source or destination port alone |
--syn |
The TCP SYNs that open a connection alone |
--for duration |
Release the packets after this machine time, such as 5s |
Examples:
1 | machine net run1 hold db until a release |
See also:
machine net release
Release the packets a container’s links hold.
The held packets leave together, in the order they were sent. The link keeps its other faults. Without a container the command releases every link.
1 | machine net run release [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] |
| Argument | Does |
|---|---|
[container] |
Container, by its id, its name or its service’s (default: every one) |
| Option | Does |
|---|---|
--to container |
The packets to this container alone (default: to every peer) |
--proto {tcp,udp,icmp,arp} |
The packets of this protocol alone |
--port n |
The packets with this source or destination port alone |
--syn |
The TCP SYNs that open a connection alone |
Examples:
1 | machine net run1 release db db's held packets leave |
See also:
machine net reset
Reset the TCP connections of a container.
The program on the container’s side sees its socket fail with ECONNABORTED, and
the peer receives a TCP reset. A listening socket stays open. Use --to db and
--port 5432 to reset some connections alone.
1 | machine net run reset [-h] [--to container] [--port n] container |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
| Option | Does |
|---|---|
--to container |
The connections to this container alone (default: every one) |
--port n |
The connections with this local or remote port alone |
Examples:
1 | machine net run1 reset api every connection of api |
See also:
machine net arp
Show or change a container’s neighbour table.
The table maps each peer’s address to the MAC that ARP resolved. The show verb lists it, with the peer each address belongs to and the state of each entry.
The other verbs change the table. They set a wrong MAC, mark an entry stale or delete entries. The reachable verb sets how long an entry stays reachable. A container that starts again begins with an empty table.
1 | machine net run arp [-h] command ... |
Examples:
1 | machine net run1 arp show api api's neighbour table |
See also:
machine net arp show
List a container’s neighbour table.
Each entry shows its interface, the address, the peer it belongs to, the MAC and its state.
1 | machine net run arp show [-h] container |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
Examples:
1 | machine net run1 arp show api api's neighbour table |
See also:
machine net arp set
Map a peer’s address to a MAC.
The entry is permanent, so ARP never corrects it. The container sends the peer’s frames to that MAC until the entry is deleted.
1 | machine net run arp set [-h] container peer mac |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
peer |
The peer, by its id, its name or its service’s |
mac |
The MAC, such as 02:42:ac:12:00:99 |
Examples:
1 | machine net run1 arp set api db 02:42:ac:12:00:99 a wrong MAC |
See also:
machine net arp stale
Mark a peer’s entry stale.
The kernel uses the entry, then confirms it by ARP at the next packet.
1 | machine net run arp stale [-h] container peer |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
peer |
The peer, by its id, its name or its service’s |
Examples:
1 | machine net run1 arp stale api db confirmed by ARP next |
See also:
machine net arp delete
Delete a peer’s entry.
The next packet to the peer resolves its address by ARP again.
1 | machine net run arp delete [-h] container peer |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
peer |
The peer, by its id, its name or its service’s |
Examples:
1 | machine net run1 arp delete api db db resolved again |
See also:
machine net arp flush
Delete every entry of the table.
Every peer is resolved by ARP again at its next packet.
1 | machine net run arp flush [-h] container |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
Examples:
1 | machine net run1 arp flush api every entry deleted |
See also:
machine net arp reachable
Set how long an entry stays reachable.
After ARP confirms an entry, the kernel keeps it reachable for a random time around this one. The entry is then stale.
1 | machine net run arp reachable [-h] container duration |
| Argument | Does |
|---|---|
container |
Container, by its id, its name or its service’s |
duration |
The base reachable time, such as 1s |
Examples:
1 | machine net run1 arp reachable api 1s stale after about 1s |
See also:
machine net heal
Take the faults off a container’s links.
Without a container the command heals every link. Use --to db to heal the
faults on packets to db alone. The command releases the packets a hold keeps. A
packet the link delays already still leaves at its time.
1 | machine net run heal [-h] [--to container] [--proto {tcp,udp,icmp,arp}] [--port n] [--syn] |
| Argument | Does |
|---|---|
[container] |
Container, by its id, its name or its service’s (default: every one) |
| Option | Does |
|---|---|
--to container |
The packets to this container alone (default: to every peer) |
--proto {tcp,udp,icmp,arp} |
The packets of this protocol alone |
--port n |
The packets with this source or destination port alone |
--syn |
The TCP SYNs that open a connection alone |
Examples:
1 | machine net run1 heal db db's links healed |
See also:
machine core
Get a core dump of a process.
The command tells a process to generate a core dump and copies it to your
computer. A debugger reads the core with the program of the process. Use
--container for a process of a container, by the pid it has there.
Taking a core ends the process. The process dies of the signal. Use fork first to keep the run as it is, and take the core on the branch.
A process that handles the signal dumps no core. Use --signal to name another.
The first process of a container dumps none.
The machine runs until the core is written and copied. Other programs of the machine run with it.
1 | machine core [-h] [--container name] [--signal signal] [--within duration] run pid file |
| Argument | Does |
|---|---|
run |
Run name |
pid |
Pid of the process in the machine or in the container |
file |
File on this computer to write the core to |
| Option | Does |
|---|---|
--container name |
Container the process is in, by its id, its name or its Compose service (default: the machine itself) |
--signal signal |
Signal the process dies of, by name or number (default: ABRT) |
--within duration |
Machine time allowed each time the command runs the machine, such as 10s (default: 30m) |
Examples:
1 | machine core run1 412 ./app.core a core of pid 412 |
Prints:
1 | text Core of <pid> of <run> (<id>): <n> bytes in <file> |
Refuses, and the way past:
1 | a machine that is replaying machine fork run1 --name try2 |
See also:
machine profile
Profile what the guest runs.
The machine samples each CPU at every timer interrupt, 250 times a second. Nothing runs in the guest for it. The command prints the functions that ran. Each has the share of samples it ran in and the share it was on the stack of.
Use --from to profile a stretch the run already recorded. The stretch is
replayed on a branch, which is deleted after. The run is not changed and need
not be running. A stretch profiled twice gives the same samples.
Use --pid or --container to show one process or one container. Use
--folded to print every stack with its count, one a line. That text is what
flame graph tools read. Use --flame to draw a flame graph in the terminal.
The names come from the program files on the machine’s disk when the stretch ends. No checkpoint is taken. A function in a file that is not on the disk is shown by the file’s name.
Time in the kernel is one frame for each system call, named like sys_read_[k].
Without --from the machine runs for the time sampled. A machine that is idle
has no samples.
Use --in-guest to sample with a program in the guest, which the exec agent of
the disk runs. It names the functions of the kernel, and programs not yet
written to the disk. The run records that program like any other.
1 | machine profile [-h] [--for duration | --from point [point ...]] [--to point [point ...]] |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--for duration |
Machine time to run and sample for, such as 500ms (default: 1s) |
--from point [point ...] |
Point a recorded stretch begins at: ^checkpoint, #entry, -N for N entries back, a time such as 10ms or -10ms, console~regex, console:N where line N was printed, parent [n] or root. The stretch is replayed on a branch |
--to point [point ...] |
Point the recorded stretch ends at, as --from takes one (default: the run’s last entry) |
--pid pid |
Process to show, by its pid in the machine or in the container (default: every process) |
--container name |
Container to show, by its id, its name or its Compose service (default: the machine itself) |
--in-guest |
Sample with the profiler of the exec agent, a process of the guest: kernel functions by name, at the cost of changing the run |
--folded |
Print the stacks, one a line with its count, for another tool to draw |
--flame |
Draw a flame graph in the terminal |
--top n |
Functions the table shows (default: 20) |
--within duration |
Machine time allowed to find the container, such as 10s (default: 30m) |
Examples:
1 | machine profile run1 where a second of run1 went |
Prints:
1 | text SELF TOTAL FUNCTION |
Refuses, and the way past:
1 | a container the run does not have machine containers run1 |
See also:
machine irq
Inject an interrupt into the vCPUs. See Interrupts.
The command takes a vector, or --nmi alone for an NMI. Use irqs to list the
vector names.
The vector is a number, or a name from irq_vectors.h of the guest kernel. The name preempt is 0xf5, the vector Machine preempts a task with. The name reschedule is 0xfd, the reschedule vector of the guest kernel.
The vector is delivered as given. It reaches the handler that the kernel of the machine installed for it. A vector with no handler is a spurious interrupt. Vectors below 32 are CPU exceptions and are refused.
The interrupt is queued when the command returns. It is delivered and recorded when the machine next runs. A replay delivers it at the same instruction. Step first to choose that instruction.
1 | machine irq [-h] [--vcpu n] [--nmi] run [vector] |
| Argument | Does |
|---|---|
run |
Run name |
[vector] |
Vector: a name such as preempt, or a number 32-255 |
| Option | Does |
|---|---|
--vcpu n |
Which vCPU takes it, or all (default: the last entry’s) |
--nmi |
Deliver as an NMI rather than a fixed interrupt |
Examples:
1 | machine irq run1 0xec inject vector 0xec |
See also:
machine irqs
List the interrupts of a machine. See Holding interrupts.
The command lists every interrupt the machine owes or holds now, then the vectors irq injects.
An owed interrupt was raised by a device or a timer and is injected at the next
dispatch. A held interrupt waits for its due, which run --irq-delay drew. The
state hash irq_pending is of the held ones, and dump --show irq keeps the
first 32 of them.
The command changes nothing. Use ints for the interrupts the guest took, by vector.
1 | machine irqs [-h] run |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine irqs run1 the pending interrupts and the vectors |
See also:
machine inject
Inject faults into the guest’s system calls and kernel.
A fault makes a system call fail with an error, or wait. A fault at a kernel site fails a disk request or drops its completion. It also fails an allocation of the kernel, or a copy to or from a program’s memory. The guest kernel decides at each call, without stopping the machine.
A replay acts on the same calls the same way. The faults are part of the machine. A checkpoint keeps them, and a fork has its parent’s. The run records each change, and a replay makes it at the same point.
Use --in to say whose calls a fault acts on. It is required. A program is the
absolute path of the file it runs, the EXE that tasks shows. It takes the tasks
running that file, and the processes they fork without exec.
A comm is the name the kernel gives a task at exec, and comm=name takes the tasks of that comm. A script runs as its interpreter, so its comm names it. tid=n takes one task, tgid=n the tasks of one tgid, and all every task in the machine.
A fault lets a number of matching calls through first. It then fires on each call by a chance, up to a number of calls or without end. Each fault counts the calls it matches on its own. The first fault that fires on a call acts on it.
A program’s fault can be added before the program starts. Its calls match from
its exec on, and the commands that start it match none. A fault on a tgid
applies to whatever that tgid runs, exec’s setup of it first. Use tasks exist to
check that what --in names is running.
Without a verb the command lists the faults.
1 | machine inject [-h] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine inject run1 the faults |
See also:
machine inject add
Inject a fault into a subsystem of the guest.
The subsystem is syscall, or one of the kernel’s own fault sites: disk, disk-timeout, slab, pages, usercopy and futex-access. The subsystems disk-corrupt, disk-torn and disk-lost are faults of the disk itself. The function subsystem makes a function of a program return a value.
Every subsystem takes the same options, with the same meaning. The disk
subsystems also take --file, --op and --sectors.
A fault fires on every matching call by default. Use --prob to fire on a call
by chance. A --prob of 0 fires on none and counts the matches.
The fault draws from its seed which calls it fires on. A replay draws the same.
Use --seed to draw as another run did.
Use --on and --off to make a fault cycle on the machine’s clock. With --on 2s --off 5s, the fault is active for 2 seconds from when it is added. It then
rests for 5 seconds, and the cycle repeats. A resting fault matches no call and
counts none.
1 | machine inject run add [-h] subsystem ... |
Examples:
1 | machine inject run1 add syscall openat ENOENT --in /app fail |
See also:
machine inject add syscall
Fail calls of system calls with errors, or hold them.
A failed call returns an error without running. Each system call gets its own fault. A failing call returns one error of the set, drawn at random.
The set holds names or numbers. Use valid for the errors the system call’s man page documents, and all for every error Linux names. A -ERRNO in the set takes that error out.
Use --delay to hold each call the fault fires on for a time, drawn from a
span. A fault with a delay and no errors runs the call after the hold. A fault
with both holds the call, then fails it. The hold is on the guest’s own clock,
so a replay holds the same calls for the same time.
Use --pick to take a number of the set at random when the fault is added. Each
system call picks its own. A run then fails with its own few errors, as swarm
testing does.
A dynamic program’s loader opens and reads its libraries after the exec. A fault on openat or read added before the program starts fails the loader. Add such a fault once the program is up.
1 | machine inject run add syscall [-h] [--delay span] [--pick n] --in who [--prob p] |
| Argument | Does |
|---|---|
syscall[,syscall...] |
System calls, by name or number |
[errors] |
Errors: ENOENT, a number, valid or all, comma-separated; -ERRNO takes one out. Without them, --delay is required and the call runs |
| Option | Does |
|---|---|
--delay span |
Hold each call the fault fires on for a time drawn from a span such as 0:20ms, at most 1s |
--pick n |
Pick n of the errors at random when the fault is added (default: all of them) |
--in who |
Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task |
--prob p |
Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1) |
--after n |
Let n matching calls through first (default: 0) |
--times n |
Fire on at most n calls, then let the rest through (default: no end) |
--seed n |
Seed the draws of the fault (default: a random seed) |
--on duration |
Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active) |
--off duration |
Machine time the fault rests in each cycle, such as 5s; needs --on |
Examples:
1 | machine inject run1 add syscall connect EIO --in /a every connect |
See also:
machine inject add function
Make a function of a program return a value without running.
A call the fault fires on returns the value of --return to its caller, and the
function does not run. A malloc that returns 0 is an allocation that failed. The
C library does not set errno, so a caller that reads errno reads what was there.
The function is found by its name in the files a running task maps, a library or
the program itself. The task is one --in names, or the task of --at when
what --in names does not run yet. The fault acts on every task that maps the
same file and that --in names.
The guest kernel puts a breakpoint at the function in the tasks --in names.
Each call there traps into the kernel, which decides as for every fault.
1 | machine inject run add function [-h] [--return value] [--at tid] --in who [--prob p] |
| Argument | Does |
|---|---|
function |
Function, by its name |
| Option | Does |
|---|---|
--return value |
Value it returns, such as 0 or -1; 0 by default |
--at tid |
Task whose files the function is found in, when --in names none running |
--in who |
Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task |
--prob p |
Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1) |
--after n |
Let n matching calls through first (default: 0) |
--times n |
Fire on at most n calls, then let the rest through (default: no end) |
--seed n |
Seed the draws of the fault (default: a random seed) |
--on duration |
Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active) |
--off duration |
Machine time the fault rests in each cycle, such as 5s; needs --on |
Examples:
1 | machine inject run1 add function malloc --in /app malloc fails |
See also:
machine inject add disk
Fail block requests of the guest’s disks with EIO.
A read or write request fails with EIO in the task that submits it. A program
submits its reads, its O_DIRECT writes and the writes its fsync forces. The
kernel’s flusher threads submit buffered writes later, so --in all or a
kworker comm matches those. A request the kernel splits counts once for each
part.
A buffered read is a readahead request, then one more read of the page when the
readahead fails. With --times 1 the kernel’s second read succeeds and the
program reads the file. Use --times 2 or no limit to return EIO to the
program. An O_DIRECT read fails on its first request.
A disk fault matches a request by its operation, its sectors and its file. Use
--file to name the file whose data the request carries. Buffered writes reach
the disk from the kernel’s flusher, so use --in all with --file.
A file matches whether it is opened or mapped before or after the fault is added.
The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.
1 | machine inject run add disk [-h] --in who [--prob p] [--after n] [--times n] [--seed n] |
| Option | Does |
|---|---|
--in who |
Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task |
--prob p |
Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1) |
--after n |
Let n matching calls through first (default: 0) |
--times n |
Fire on at most n calls, then let the rest through (default: no end) |
--seed n |
Seed the draws of the fault (default: a random seed) |
--on duration |
Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active) |
--off duration |
Machine time the fault rests in each cycle, such as 5s; needs --on |
--file pattern |
Requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file) |
--op {read,write} |
Requests of one operation (default: both) |
--sectors lo:hi |
Requests that touch 512-byte sectors lo to hi of the disk (default: the whole disk) |
Examples:
1 | machine inject run1 add disk --in all every block request |
See also:
machine inject add disk-timeout
Drop the completions of block requests.
The request is decided in the task that submits it, and its completion is dropped when the disk answers. The guest’s disk driver has no timeout handler, so the request never completes and the task waiting on it hangs. The tasks that submit a request are as for disk.
A disk fault matches a request by its operation, its sectors and its file. Use
--file to name the file whose data the request carries. Buffered writes reach
the disk from the kernel’s flusher, so use --in all with --file.
A file matches whether it is opened or mapped before or after the fault is added.
The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.
1 | machine inject run add disk-timeout [-h] --in who [--prob p] [--after n] [--times n] |
| Option | Does |
|---|---|
--in who |
Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task |
--prob p |
Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1) |
--after n |
Let n matching calls through first (default: 0) |
--times n |
Fire on at most n calls, then let the rest through (default: no end) |
--seed n |
Seed the draws of the fault (default: a random seed) |
--on duration |
Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active) |
--off duration |
Machine time the fault rests in each cycle, such as 5s; needs --on |
--file pattern |
Requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file) |
--op {read,write} |
Requests of one operation (default: both) |
--sectors lo:hi |
Requests that touch 512-byte sectors lo to hi of the disk (default: the whole disk) |
Examples:
1 | machine inject run1 add disk-timeout --times 1 --in comm=dd dd hangs |
See also:
machine inject add slab
Fail slab allocations of the guest kernel.
The allocation returns NULL, and the call that made it fails with ENOMEM or as the kernel handles it. An allocation the kernel promised, with __GFP_NOFAIL, never fails.
The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.
1 | machine inject run add slab [-h] --in who [--prob p] [--after n] [--times n] [--seed n] |
| Option | Does |
|---|---|
--in who |
Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task |
--prob p |
Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1) |
--after n |
Let n matching calls through first (default: 0) |
--times n |
Fire on at most n calls, then let the rest through (default: no end) |
--seed n |
Seed the draws of the fault (default: a random seed) |
--on duration |
Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active) |
--off duration |
Machine time the fault rests in each cycle, such as 5s; needs --on |
Examples:
1 | machine inject run1 add slab --prob 0.001 --in comm=cat rarely |
See also:
machine inject add pages
Fail page allocations of the guest kernel.
The allocation returns no pages, and the call that made it fails with ENOMEM or as the kernel handles it. An allocation the kernel promised, with __GFP_NOFAIL, never fails.
The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.
1 | machine inject run add pages [-h] --in who [--prob p] [--after n] [--times n] [--seed n] |
| Option | Does |
|---|---|
--in who |
Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task |
--prob p |
Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1) |
--after n |
Let n matching calls through first (default: 0) |
--times n |
Fire on at most n calls, then let the rest through (default: no end) |
--seed n |
Seed the draws of the fault (default: a random seed) |
--on duration |
Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active) |
--off duration |
Machine time the fault rests in each cycle, such as 5s; needs --on |
Examples:
1 | machine inject run1 add pages --after 99 --times 1 --in all the 100th |
See also:
machine inject add usercopy
Fail copies between the kernel and a program’s memory.
The copy fails, and the system call that made it returns EFAULT or a short count. A system call that copies a buffer to or from the program copies through it. A call that reads or writes one value with get_user or put_user does not.
The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.
1 | machine inject run add usercopy [-h] --in who [--prob p] [--after n] [--times n] [--seed n] |
| Option | Does |
|---|---|
--in who |
Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task |
--prob p |
Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1) |
--after n |
Let n matching calls through first (default: 0) |
--times n |
Fire on at most n calls, then let the rest through (default: no end) |
--seed n |
Seed the draws of the fault (default: a random seed) |
--on duration |
Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active) |
--off duration |
Machine time the fault rests in each cycle, such as 5s; needs --on |
Examples:
1 | machine inject run1 add usercopy --times 1 --in tid=41 one copy |
See also:
machine inject add futex-access
Fail a futex’s access to a program’s memory.
A futex operation’s access to the program’s memory fails. The futex call returns EFAULT, or EDEADLK where a priority-inheritance futex checks its owner.
The guest kernel decides at the kernel’s own fault site. It decides after the checks that keep a promised operation from failing. A site takes no errors and no delay, because it fails its own way.
1 | machine inject run add futex-access [-h] --in who [--prob p] [--after n] [--times n] |
| Option | Does |
|---|---|
--in who |
Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task |
--prob p |
Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1) |
--after n |
Let n matching calls through first (default: 0) |
--times n |
Fire on at most n calls, then let the rest through (default: no end) |
--seed n |
Seed the draws of the fault (default: a random seed) |
--on duration |
Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active) |
--off duration |
Machine time the fault rests in each cycle, such as 5s; needs --on |
Examples:
1 | machine inject run1 add futex-access --in tgid=412 every futex call |
See also:
machine inject add disk-corrupt
Change bytes of reads and writes of the guest’s disks.
The disk stores a write with bytes changed, or returns a read with bytes changed. The request succeeds. The changed bytes are drawn from the fault’s seed. A program’s own reads of a written file come from the page cache until the guest drops it.
A disk fault matches a request by its operation, its sectors and its file. Use
--file to name the file whose data the request carries. Buffered writes reach
the disk from the kernel’s flusher, so use --in all with --file.
A file matches whether it is opened or mapped before or after the fault is added.
The guest kernel decides on each request in the task that submits it. The virtual disk then changes what it stores or returns. A replay makes the same decisions and changes the same bytes.
1 | machine inject run add disk-corrupt [-h] --in who [--prob p] [--after n] [--times n] |
| Option | Does |
|---|---|
--in who |
Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task |
--prob p |
Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1) |
--after n |
Let n matching calls through first (default: 0) |
--times n |
Fire on at most n calls, then let the rest through (default: no end) |
--seed n |
Seed the draws of the fault (default: a random seed) |
--on duration |
Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active) |
--off duration |
Machine time the fault rests in each cycle, such as 5s; needs --on |
--file pattern |
Requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file) |
--op {read,write} |
Requests of one operation (default: both) |
--sectors lo:hi |
Requests that touch 512-byte sectors lo to hi of the disk (default: the whole disk) |
--bytes n |
Bytes of the request to change, 1 to 4096 (default: 1) |
Examples:
1 | machine inject run1 add disk-corrupt --file '/db/*/c.bin' --in all parts |
See also:
machine inject add disk-torn
Store part of a write of the guest’s disks.
The disk stores the first sectors of a write and reports success. It stores one sector at least, and fewer than all. The rest keeps what the disk held. A sector is written whole, so a write of one sector does not tear.
A disk fault matches a request by its operation, its sectors and its file. Use
--file to name the file whose data the request carries. Buffered writes reach
the disk from the kernel’s flusher, so use --in all with --file.
A file matches whether it is opened or mapped before or after the fault is added.
The guest kernel decides on each request in the task that submits it. The virtual disk then changes what it stores or returns. A replay makes the same decisions and changes the same bytes.
1 | machine inject run add disk-torn [-h] --in who [--prob p] [--after n] [--times n] |
| Option | Does |
|---|---|
--in who |
Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task |
--prob p |
Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1) |
--after n |
Let n matching calls through first (default: 0) |
--times n |
Fire on at most n calls, then let the rest through (default: no end) |
--seed n |
Seed the draws of the fault (default: a random seed) |
--on duration |
Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active) |
--off duration |
Machine time the fault rests in each cycle, such as 5s; needs --on |
--file pattern |
Requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file) |
--sectors lo:hi |
Requests that touch 512-byte sectors lo to hi of the disk (default: the whole disk) |
Examples:
1 | machine inject run1 add disk-torn --file /db/wal --in all torn writes |
See also:
machine inject add disk-lost
Lose writes of the guest’s disks.
The disk reports a write done and stores none of it. The disk keeps what it held, as a disk that lost its cache does.
A disk fault matches a request by its operation, its sectors and its file. Use
--file to name the file whose data the request carries. Buffered writes reach
the disk from the kernel’s flusher, so use --in all with --file.
A file matches whether it is opened or mapped before or after the fault is added.
The guest kernel decides on each request in the task that submits it. The virtual disk then changes what it stores or returns. A replay makes the same decisions and changes the same bytes.
1 | machine inject run add disk-lost [-h] --in who [--prob p] [--after n] [--times n] |
| Option | Does |
|---|---|
--in who |
Whose calls the fault acts on: a program by the absolute path of its file, comm=name, tid=n, tgid=n, or all for every task |
--prob p |
Chance a matching call fires the fault, 0 to 1, such as 0.1 (default: 1) |
--after n |
Let n matching calls through first (default: 0) |
--times n |
Fire on at most n calls, then let the rest through (default: no end) |
--seed n |
Seed the draws of the fault (default: a random seed) |
--on duration |
Machine time the fault is active in each cycle, such as 2s; needs --off (default: always active) |
--off duration |
Machine time the fault rests in each cycle, such as 5s; needs --on |
--file pattern |
Requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file) |
--sectors lo:hi |
Requests that touch 512-byte sectors lo to hi of the disk (default: the whole disk) |
Examples:
1 | machine inject run1 add disk-lost --file /db/wal --in all lost writes |
See also:
machine inject delete
Delete a fault.
The fault’s calls run again from here.
1 | machine inject run delete [-h] n |
| Argument | Does |
|---|---|
n |
Fault number, as the listing shows it |
Examples:
1 | machine inject run1 delete 2 the second fault |
See also:
machine inject list
List the faults.
The listing is the one the command prints with no verb.
1 | machine inject run list [-h] |
Examples:
1 | machine inject run1 list the faults, by number |
See also:
machine inject clear
Delete every fault.
1 | machine inject run clear [-h] |
Examples:
1 | machine inject run1 clear every fault |
See also:
machine axis
Set the defaults of the draw axes a run uses.
A default is a plan clause the run keeps. Every run, exec and plan draws from the defaults until they are removed. The axes are the ones a plan clause takes, such as mode, vtime-rate, task-sched and irq-delay.
A run option or a plan clause replaces a default for the axes it names. A default with a when condition replaces one with no condition where the condition holds. Of two conditions that hold, the default set last wins.
The defaults are part of the machine. A checkpoint keeps them, and a fork has its parent’s. The run records each change, and a replay makes it at the same point.
Without a verb the command lists the defaults.
1 | machine axis [-h] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine axis run1 the defaults |
See also:
machine axis set
Set a default of the draw axes.
The default is a clause of a plan, written as a plan file writes it. It names one or more axes and their values. A clause that starts with when applies where its predicate holds.
The default replaces the values its axes had under the same condition. A clause that names several axes draws them together.
1 | machine axis run set [-h] ... |
| Argument | Does |
|---|---|
... |
A plan clause, such as when /app:on task-sched hold |
Examples:
1 | machine axis run1 set vtime-rate 10:40 a drawn rate |
See also:
machine axis unset
Remove defaults of the draw axes.
The axes are named as a clause names them, without values. A clause that starts with when removes the defaults under that predicate. Without when, the command removes the defaults that have no condition.
Use --all to remove the axes’ defaults under every condition.
1 | machine axis run unset [-h] [--all] ... |
| Argument | Does |
|---|---|
... |
Axis names after an optional when predicate, such as when /app:on mode |
| Option | Does |
|---|---|
--all |
Remove the axes’ defaults under every condition |
Examples:
1 | machine axis run1 unset mode no condition |
See also:
machine axis clear
Remove every default.
1 | machine axis run clear [-h] |
Examples:
1 | machine axis run1 clear every default |
See also:
machine vtime
Show and change machine time. See Machine time.
Machine time is the guest’s clock. It is derived from what the machine executes, not from the host clock.
Use rate to change how fast machine time passes. Use cost to change what each exit costs on it. Use add to push the clock forward without running the machine. The clock only goes forward.
Use rewind to put the run back at an earlier point.
1 | machine vtime [-h] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine vtime run1 machine time and its rate |
See also:
machine vtime rate
Set how fast machine time passes. See Machine time.
The rate is the nanoseconds of machine time for each conditional branch the
guest retires. A run starts at 10 unless create was given --vtime-rate.
Give the rate as a number or as num/den. Add N=rate to give vCPU N its own rate, so its work costs more or less time than the others’. A rate for every vCPU clears each vCPU’s own.
The run records each change, and a replay makes it at the same point. A run’s
--vtime-rate draws one rate for every vCPU for that run alone.
1 | machine vtime run rate [-h] rate |
| Argument | Does |
|---|---|
rate |
Rate, num/den, and N=rate for vCPU N’s own, such as 10,1=40 |
Examples:
1 | machine vtime run1 rate 20 20 ns for each conditional branch |
See also:
machine vtime add
Push the clock forward without running the machine. See Machine time.
Give a time such as 1s or 250ms. A bare number is nanoseconds.
1 | machine vtime run add [-h] time |
| Argument | Does |
|---|---|
time |
Time to add, such as 1s or 250ms; a bare number is nanoseconds |
Examples:
1 | machine vtime run1 add 10us advance the clock 10 us |
See also:
machine vtime cost
Set what each exit costs on the machine’s clock.
Each exit of the guest advances machine time by a cost. The exits priced are an I/O, MMIO or hypercall exit, an RDTSC and each PAUSE of a spin loop. The rate prices the guest’s computing, and the costs price its waiting. Their ratio decides whether a spin or an I/O completion wins a race.
Give io=, rdtsc= or pause= with a time from 1ns to 1s. A bare number is nanoseconds. A cost not given stays as it is. The RDTSC cost follows the I/O cost until it is given, and rdtsc=io makes it follow again.
The run records the change, and a replay makes it at the same point. A run’s
--vtime-io-cost, --vtime-rdtsc-cost or --vtime-pause-cost draws its own
costs for that run alone.
1 | machine vtime run cost [-h] exit=time [exit=time ...] |
| Argument | Does |
|---|---|
exit=time [exit=time ...] |
An exit and its cost, such as io=2us, rdtsc=1us or pause=50; rdtsc=io follows the I/O cost |
Examples:
1 | machine vtime run1 cost io=20us each I/O exit costs 20 us |
See also:
machine clock
Show and change the guest’s date and each container’s.
The command steps the date and sets it. It makes a container’s date drift, or slews it to a new value. It steps the date to a drawn offset each period, as NTP does. It inserts or deletes a leap second.
The date is what date, time and gettimeofday return. A change moves it for every process in the guest. Elapsed time, the TSC and timers measured in elapsed time do not move. Use vtime to change how fast time runs.
The guest kernel applies a change when the run next executes. The command answers with the date as it stands and the change waiting. The reading after that run names a change the guest kernel refused.
A change is part of the machine. The run records it, and a replay makes it at the same point. A checkpoint keeps the date and a pending undo.
A container’s date is an offset from the guest’s date. Every process of the container reads it, those docker exec starts included. A timer armed after a change agrees with it. A timer armed before keeps its deadline.
Without a verb the command shows the date and each container’s. The command finds a container with the machine stopped and never runs it. It tries a name first, then a service, then an id. An id works from the container’s start, while a name is known shortly after.
A jump or a set takes a running container. A show or a reset also takes a stopped one with an offset. A service stands for its one container of those.
1 | machine clock [-h] [--container name] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
| Option | Does |
|---|---|
--container name |
A container by its id, its name or its Compose service |
Examples:
1 | machine clock run1 the guest's date |
See also:
machine clock jumpmachine clock setmachine clock driftmachine clock jittermachine clock leapmachine containersmachine vtime
machine clock jump
Step the guest’s date forward or back.
The step is the one an NTP step or date -s makes. A timer armed for an absolute date sees the step. A timerfd with TFD_TIMER_CANCEL_ON_SET is woken. Each step adds to the ones before it.
Use --for 30s to undo the change after 30 seconds of machine time. The undo
takes back the command’s own change, and what a drift gave meanwhile stays. A
later change cancels an undo still pending.
Use --slew 30s with --container to move a container’s date over 30 seconds
of machine time. No read in it sees a jump. With --for, the date slews back
over the same time.
Use --container db to change one container’s date alone. The container keeps
its offset from the guest’s date across a restart.
1 | machine clock run jump [-h] [--for duration] [--slew duration] [--container name] offset |
| Argument | Does |
|---|---|
offset |
The step, such as +5s, -2h or 59m50s; no sign is forward |
| Option | Does |
|---|---|
--for duration |
Undo the change after this much machine time, such as 30s |
--slew duration |
Move a container’s date over this machine time, such as 30s |
--container name |
A container by its id, its name or its Compose service |
Examples:
1 | machine clock run1 jump +5s forward |
See also:
machine clock set
Set the guest’s date.
The date is written with its zone, such as 2027-03-28T00:59:50Z. The change is a step, as date -s makes one.
Use --for 30s to undo the change after 30 seconds of machine time. The undo
takes back the command’s own change, and what a drift gave meanwhile stays. A
later change cancels an undo still pending.
Use --slew 30s with --container to move a container’s date over 30 seconds
of machine time. No read in it sees a jump. With --for, the date slews back
over the same time.
Use --container db to change one container’s date alone. The container keeps
its offset from the guest’s date across a restart.
1 | machine clock run set [-h] [--for duration] [--slew duration] [--container name] date |
| Argument | Does |
|---|---|
date |
The date with its zone, such as 2027-03-28T00:59:50Z |
| Option | Does |
|---|---|
--for duration |
Undo the change after this much machine time, such as 30s |
--slew duration |
Move a container’s date over this machine time, such as 30s |
--container name |
A container by its id, its name or its Compose service |
Examples:
1 | machine clock run1 set 2026-12-31T23:59:50Z the year's end |
See also:
machine clock drift
Make a container’s date gain or lose time.
The container’s date gains the rate of each second of machine time, such as +200ppm, 0.72 seconds an hour. No read sees a jump. Its processes read it in the vDSO and in system calls. A timer armed after agrees with it.
Use drift 0 to stop it and keep the offset it reached. A jump, a set or a jitter moves the date under the drift.
The guest’s date has no drift. Use vtime rate to make the whole machine’s time run fast or slow.
Use --for 30s to undo the change after 30 seconds of machine time. The undo
takes back the command’s own change, and what a drift gave meanwhile stays. A
later change cancels an undo still pending.
1 | machine clock run drift [-h] [--for duration] [--container name] rate |
| Argument | Does |
|---|---|
rate |
The rate, such as +200ppm, -1.5ppm or 300ppb; 0 stops it |
| Option | Does |
|---|---|
--for duration |
Undo the change after this much machine time, such as 30s |
--container name |
A container by its id, its name or its Compose service |
Examples:
1 | machine clock run1 drift +200ppm --container db 0.72s an hour |
See also:
machine clock jitter
Step the guest’s date to a drawn offset each period.
Each period of machine time the date is set to a value drawn from the choices. The value is around the date the jitter leaves alone. A choice is a span such as -200ms:200ms or one value such as 1s. A weight in brackets draws a choice more often, as -1s[1],0[8],1s[1] does.
The first step is taken when the guest takes the command, and the next a period later. Each step is a step of the date, as NTP makes one. A jump or a set moves the date under the jitter, which goes on around it. Use jitter off to stop it and return to the date without it.
The draws come from --seed, or from a seed drawn at random and shown. A replay
steps by the same amounts at the same points. A fork has its parent’s jitter and
draws on from where it was.
Use --for 30s to undo the change after 30 seconds of machine time. The undo
takes back the command’s own change, and what a drift gave meanwhile stays. A
later change cancels an undo still pending.
Use --container db to change one container’s date alone. The container keeps
its offset from the guest’s date across a restart.
1 | machine clock run jitter [-h] [--every duration] [--seed n] [--for duration] |
| Argument | Does |
|---|---|
choices |
The choices, such as -200ms:200ms or -1s[1],0[8],1s[1]; off stops it |
| Option | Does |
|---|---|
--every duration |
Machine time between two steps, such as 10m |
--seed n |
The seed the steps are drawn from (default: one drawn at random) |
--for duration |
Undo the change after this much machine time, such as 30s |
--container name |
A container by its id, its name or its Compose service |
Examples:
1 | machine clock run1 jitter -200ms:200ms --every 10m NTP's steps |
See also:
machine clock leap
Insert or delete a second at the next UTC midnight.
The guest kernel inserts 23:59:60 or skips 23:59:59 at the next midnight of the guest’s date. It does so as an NTP daemon’s leap flag asks it to. Then it clears the flag. Use leap off to cancel a leap still pending.
A leap second is the guest kernel’s timekeeping, so it moves every container with the guest’s date. A reset cancels it too.
1 | machine clock run leap [-h] [--container name] {insert,delete,off} |
| Argument | Does |
|---|---|
{insert,delete,off} |
Insert, delete, or off to cancel a leap pending |
| Option | Does |
|---|---|
--container name |
A container by its id, its name or its Compose service |
Examples:
1 | machine clock run1 leap insert 23:59:60 at midnight |
See also:
machine clock reset
Step the guest’s date and every container’s back to machine time.
Machine time is the date the guest kernel had at boot, moved by elapsed time. The reset undoes every step, those of clock and those a process made, and cancels every pending undo.
Use --container db to return one container to the guest’s date. The guest’s
date and the other containers stay as they are.
1 | machine clock run reset [-h] [--container name] |
| Option | Does |
|---|---|
--container name |
A container by its id, its name or its Compose service |
Examples:
1 | machine clock run1 reset the guest and every container |
See also:
machine rng
Show and seed the RNG device of a machine. See Random numbers.
The RNG device gives the guest its random bytes from a seed. A run takes the
seed from create --rng-seed.
Use seed to seed it again here. The run records the seed, and a replay seeds the device at the same point. A guest kernel that reads the seed reseeds its own generator at once. A kernel that does not read the seed picks it up the next time that kernel reseeds.
A run created with --no-rng has no RNG device, and the command refuses it.
1 | machine rng [-h] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine rng run1 the seed and whether the guest reseeds at once |
See also:
machine rng seed
Seed the RNG device again.
The device gives its bytes from the new seed, as from create --rng-seed. The
run records the seed, so a replay seeds the device at the same point.
1 | machine rng run seed [-h] n |
| Argument | Does |
|---|---|
n |
Seed, 0 to 2^64-1 |
Examples:
1 | machine rng run1 seed 7 seed the RNG device from 7 |
See also:
machine regs
Show the registers of a task.
The registers are the user registers of a task of the guest, by its tid. A task on a vCPU has them in the vCPU. A task off every vCPU has them where it saved them. A task in the kernel has the ones it entered with.
Without names the general registers are shown. Use --all to show the x87, SSE
and AVX registers as well. Name registers to show those alone.
NAME=VALUE writes a register. A value is a number or a location as mem takes one, such as main+4. The write is part of the run, and a replay makes it again. The selectors and the x87, SSE and AVX registers are not written.
1 | machine regs [-h] [--all] run tid [register ...] |
| Argument | Does |
|---|---|
run |
Run name |
tid |
Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process |
[register ...] |
Register to show, such as rip or xmm0, or NAME=VALUE to write one (default: the general registers) |
| Option | Does |
|---|---|
--all |
Show the x87, SSE and AVX registers as well |
Examples:
1 | machine regs run1 412 the general registers of tid 412 |
Prints:
1 | text REGISTER VALUE NATURAL, then where they were read |
Refuses, and the way past:
1 | a tid the machine does not have machine tasks run1 |
See also:
machine mem
Show the memory of a tgid.
The memory is the tgid’s own, through its page tables. The task names its tgid. Guest kernel memory is not shown.
A location names an address in the task’s process. Every command that takes a location takes one of these:
1 | 0x401136 an address |
A format after the location shows the memory as GDB’s x command does. A format is a count, a letter and a unit, such as 8xg, 4i or s. The letters are x, d, u, o, t, a, c, f, s, i and z. The units are b, h, w and g.
Each line begins with its address and the symbol the address is in. A read that reaches an address that is not mapped ends there.
LOCATION=VALUE writes there, and the write is part of the run. A quoted string writes its bytes. A number takes the size of the variable it is written to, or of a unit after it. A page the tgid cannot write is not written.
LOCATION^=MASK flips the bits the mask sets in the value there, as ^= does in C. The value is as wide as the variable or the unit after it. The flip is a write and is part of the run.
1 | machine mem [-h] [--find pattern] [--dump file] [--restore file] [--fill pattern] |
| Argument | Does |
|---|---|
run |
Run name |
tid |
Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process |
[location] |
Address to read at: a number, *ADDR, $register, NAME, PATH:NAME or FILE:LINE, moved by +N or -N. LOCATION=VALUE writes there, LOCATION^=MASK flips bits |
[format] |
Count, format letter and unit, as GDB’s x takes them, such as 8xg, 4i or s (default: 8xg) |
| Option | Does |
|---|---|
--find pattern |
Find a pattern in --length bytes: text, or 0x and hex bytes |
--dump file |
Write --length bytes to a file on this computer |
--restore file |
Write the bytes of a file on this computer at the location |
--fill pattern |
Write a pattern over --length bytes: text, or 0x and hex bytes |
--length n |
Bytes a find, a dump or a fill reaches over |
--mapped |
List the pages the tgid maps, with their access and file |
Examples:
1 | machine mem run1 412 '$sp' eight 8-byte words at the stack |
Prints:
1 | text <address> <symbol+offset>: units, as GDB's x |
Refuses, and the way past:
1 | a symbol no file names machine symbols run1 412 --regex name |
See also:
machine backtrace
Show the stack of a task.
Each frame is a function the task is in, the innermost first. The frames are walked by the call frame information of the program files. Code with no call frame information is walked by its frame pointer.
The names come from the program files on the machine’s disk as it stands. A frame of a file the disk does not have is named by its file. A frame of a program built with debug information names its source line.
1 | machine backtrace [-h] [--frame n] run tid |
| Argument | Does |
|---|---|
run |
Run name |
tid |
Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process |
| Option | Does |
|---|---|
--frame n |
Describe one frame, as GDB’s info frame: where it is and what called it |
Examples:
1 | machine backtrace run1 412 the stack of tid 412 |
Prints:
1 | text # ADDRESS FUNCTION SOURCE FILE |
Refuses, and the way past:
1 | a tid the machine does not have machine tasks run1 |
See also:
machine call
Call a function in a task.
The task runs the function from where it stands. The arguments go in the registers the x86-64 ABI passes them in. An argument is an integer or a pointer. The value returned is printed by the function’s return type.
The task’s registers are put back after the call. What the function wrote in memory stays written. The call is part of the run, and a replay makes it again.
A breakpoint the call reaches ends it there. The registers are put back then too, and the call is abandoned.
1 | machine call [-h] run tid function [arguments ...] |
| Argument | Does |
|---|---|
run |
Run name |
tid |
Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process |
function |
Function to call: a name or a location |
[arguments ...] |
Arguments: expressions as print takes them, integers or pointers |
Examples:
1 | machine call run1 412 square 3 square of 3, by its type |
Prints:
1 | text FUNCTION(ARGUMENTS) = VALUE |
Refuses, and the way past:
1 | a tid the machine does not have machine tasks run1 |
See also:
machine return
Return from a task’s innermost function.
The task goes on at the instruction after the call, in its caller. The caller’s stack pointer and the registers the function saved are restored, as GDB’s return does. A value goes in rax.
The registers come from the call frame information of the program files on the machine’s disk. Machine refuses the operation if the function has no call frame information.
The write is part of the run, and a replay makes it again. Memory is not changed.
1 | machine return [-h] run tid [value] |
| Argument | Does |
|---|---|
run |
Run name |
tid |
Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process |
[value] |
Value the function returns in rax: a number or a location |
Examples:
1 | machine return run1 412 leave the function, rax as it is |
Prints:
1 | text Returned from FUNCTION to PLACE, in tid TID |
Refuses, and the way past:
1 | a tid the machine does not have machine tasks run1 |
See also:
machine print
Print the value of an expression.
An expression is a variable of the frame, a global or a $register. A member, an arrow, an index, a star and an ampersand apply to it as in C. The value is printed by its type.
The types and the variables come from the program’s debug information on the
machine’s disk. Use --frame to print in a frame of a caller, as backtrace
numbers them.
A format after the expression prints each number in it, as print/x does. The
letters are x, d, u, o, t, c and z. Use --type to print the type instead, as
whatis and ptype do.
TARGET = VALUE writes VALUE at TARGET and prints TARGET after the write. The write is part of the run, and a replay makes it again.
1 | machine print [-h] [--frame n] [--type] run tid expression [format] |
| Argument | Does |
|---|---|
run |
Run name |
tid |
Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process |
expression |
A variable or $register, with .m, ->m, [i], * and & |
[format] |
Format letter for each number, as print/x takes one: x, d, u, o, t, c or z |
| Option | Does |
|---|---|
--frame n |
Frame of the stack, as backtrace numbers it (default: 0) |
--type |
Print its type instead, as GDB’s whatis and ptype do |
Examples:
1 | machine print run1 412 counter a global or a local |
Prints:
1 | text EXPRESSION = VALUE |
Refuses, and the way past:
1 | a name the frame and the globals do not have machine locals run1 412 |
See also:
machine locals
List the local variables of a frame.
Each local of the function is printed with its value, by its type. Use --args
to list the function’s arguments instead. Use --frame to list a frame of a
caller.
The variables come from the program’s debug information on the machine’s disk. A variable the debug information places where it is not read is listed with why.
1 | machine locals [-h] [--frame n] [--args] run tid |
| Argument | Does |
|---|---|
run |
Run name |
tid |
Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process |
| Option | Does |
|---|---|
--frame n |
Frame of the stack, as backtrace numbers it (default: 0) |
--args |
The function’s arguments instead |
Examples:
1 | machine locals run1 412 the locals of the innermost frame |
Prints:
1 | text NAME = VALUE, a line each |
See also:
machine symbols
Find the symbols of a tgid.
A name finds the address of each symbol of that name. An address finds the symbol it is in. A symbol of one file is written PATH:NAME. A source line written FILE:LINE finds its code.
Use --regex to find each function whose name matches a pattern. The symbols
come from the program files on the machine’s disk.
1 | machine symbols [-h] [--regex pattern] [--variables] [--types] [--scope] [--sources] |
| Argument | Does |
|---|---|
run |
Run name |
tid |
Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process |
[name|address|file:line] |
Symbol to find the address of, address to find the symbol of, or source line to find the code of |
| Option | Does |
|---|---|
--regex pattern |
Find each function whose name the regular expression matches |
--variables |
Find variables with --regex, not functions |
--types |
Find the types the debug information names with --regex, as info types |
--scope |
List the variables in scope at a location, as info scope |
--sources |
Find the source files the line tables name with --regex, as info sources |
--file path |
File to look in with --regex, by its path or its file name (default: every file the tgid maps) |
Examples:
1 | machine symbols run1 412 main where main is |
Prints:
1 | text ADDRESS SIZE KIND NAME SOURCE FILE |
See also:
machine libs
List the program and the libraries a tgid maps.
Each row is a file the tgid maps executable, at its range of addresses. The first row is the program. A file the machine’s disk does not have has no symbols.
1 | machine libs [-h] run tid |
| Argument | Does |
|---|---|
run |
Run name |
tid |
Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process |
Examples:
1 | machine libs run1 412 the program and libraries of 412 |
Prints:
1 | text START END OFFSET FILE |
See also:
machine step
Step a task.
A step is one instruction of the task. Use --line to step a source line. Use
--over to step over a call, so the function and all it calls are one step. Use
--out to run until the function returns.
The machine runs whichever task the guest schedules. A step of one task runs the machine until that task is in its own code, then steps it. Other tasks run meanwhile.
Each step runs the machine, and the run records it, so a replay makes it again. A breakpoint the step reaches ends it. A line step steps out of a function that has no line tables.
A C++ exception that leaves the function ends --over and --out at the catch
that takes it, as GDB does. In a C++ program such a step takes 2 of the run’s 3
debug registers. With fewer free, the step is refused.
A longjmp out of the function ends the step after 30 minutes of machine time. Use cancel to end it sooner.
Use --skip NAME with --line to step out of a function or a source file of
that name. It does as GDB’s skip does. Give it once for each name.
Use --back to go back over the task’s executions on any vCPU, as reverse-stepi
does. An execution is one stretch the task ran, one instruction where the run
stepped. The machine switches onto a new branch there, as go does.
With --line, --over or --out, --back goes back over the task’s own code.
It does as reverse-step, reverse-next, reverse-nexti and reverse-finish do. It
goes back over what the run recorded one instruction at a time. Past that it
stops, and says the history ends there.
1 | machine step [-h] [--line] [--over] [--out] [--back [n]] [--skip name] run tid [count] |
| Argument | Does |
|---|---|
run |
Run name |
tid |
Task of the machine, by its tid, the thread id ps -L shows. Its tgid is the pid of its process |
[count] |
Steps to make (default: 1) |
| Option | Does |
|---|---|
--line |
Step a source line, not an instruction |
--over |
Step over a call, the function it calls and all it calls being one step |
--out |
Run until the function returns to the one that called it |
--back [n] |
Go back n steps over the task’s dispatches, onto a branch |
--skip name |
Step out of a function or a source file of this name in a line step (repeatable) |
Examples:
1 | machine step run1 412 one instruction, as stepi |
Prints:
1 | text Stepped N instructions to WHERE, in tid TID |
See also:
machine points
List the breakpoints, watchpoints and catchpoints.
The three share one table and its numbers. Each row has its number and its kind. Use break, watch or catch to change the row of its kind.
1 | machine points [-h] run |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine points run1 every breakpoint, watchpoint and catchpoint |
Prints:
1 | text # KIND ADDRESS SIZE WHERE TGID HITS WHEN |
See also:
machine break
List and manage breakpoints.
A breakpoint stops the run when a task of its tgid reaches its location. The listing has the run’s breakpoints, each by its number. Use points to list them with the watchpoints and catchpoints.
The run keeps them in its run log. A branch has the ones its parent had where it was forked. A run holds 64 breakpoints. Hardware breakpoints and watchpoints share the 3 debug registers.
A breakpoint stops the tasks of its tgid alone. Another process that maps the same file does not stop. A read of the task’s memory shows the program’s byte, not the breakpoint’s.
A run that reaches a breakpoint ends there and names it. The run arms its
breakpoints while it records and not while it replays. So run --back finds
only the hits the run recorded.
1 | machine break [-h] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine break run1 the breakpoints of run1 |
Prints:
1 | text # KIND ADDRESS SIZE WHERE TGID HITS WHEN |
See also:
machine break list
List the breakpoints.
The listing is the one the command prints with no verb.
1 | machine break run list [-h] |
Examples:
1 | machine break run1 list the breakpoints, by number |
See also:
machine break add
Add a breakpoint.
The breakpoint stops the run before the instruction at its location runs. It stops the tgid of the task alone. Another tgid that runs the same address goes on.
The location is resolved now, from the tgid’s program files. The run log keeps the breakpoint, and a branch has its parent’s.
Use --hardware to put it in one of the 3 debug registers, as GDB’s hbreak
does. A breakpoint in code the program allocated needs one, as does one in code
the program reads.
1 | machine break run add [-h] [--disabled] [--ignore count] [--temporary] |
| Argument | Does |
|---|---|
tid |
Task whose tgid the stop is in |
[location] |
Location, as mem takes one: an address, *ADDR, NAME, PATH:NAME, FILE:LINE or $reg, moved by +N or -N |
| Option | Does |
|---|---|
--disabled |
Add it and do not arm it |
--ignore count |
Pass this many hits where the condition holds (default: 0) |
--temporary |
Delete it at its first hit where the condition holds |
--condition condition |
Count a hit only where this expression holds, as print takes one, such as ‘$rax == 0’ |
--print format |
Print this at a hit and go on. {EXPR} prints an expression as print takes one, {EXPR:x} in a print format letter: ‘n={counter} p={$rdi:x}’ |
--regex pattern |
Add one at each function whose name matches, in place of a location |
--hardware |
Put it in a debug register, as GDB’s hbreak |
Examples:
1 | machine break run1 add 412 main at the start of main |
Prints:
1 | text Breakpoint N at WHERE, in tgid TGID |
See also:
machine break delete
Delete breakpoints.
The run log keeps the delete, so a branch forked before it keeps the breakpoint.
1 | machine break run delete [-h] number [number ...] |
| Argument | Does |
|---|---|
number [number ...] |
Number of a breakpoint |
Examples:
1 | machine break run1 delete 2 delete breakpoint 2 |
See also:
machine break clear
Delete the breakpoints at a location.
Each breakpoint at the address the location names in the tgid of the task is deleted.
1 | machine break run clear [-h] tid location |
| Argument | Does |
|---|---|
tid |
Task whose tgid the breakpoints are in |
location |
Location, as mem takes one |
Examples:
1 | machine break run1 clear 412 main delete the stops at main |
See also:
machine break save
Save the breakpoints as the commands that add them.
Each line is a machine command that adds one breakpoint again. The command keeps its condition, ignore count and flags. A location written by a register is saved as its address.
1 | machine break run save [-h] file |
| Argument | Does |
|---|---|
file |
File to write, or - for the standard output |
Examples:
1 | machine break run1 save stops.sh write the commands to a file |
See also:
machine break hits
List the hits of breakpoints.
A hit is a breakpoint that stopped the run. It has the entry it stopped at, its machine time and the number of the breakpoint. The run log keeps each one, so a branch lists its parent’s hits before it.
Use run --back to switch onto a branch at the newest hit.
1 | machine break run hits [-h] [-n range] |
| Option | Does |
|---|---|
-n, --entries range |
Hits to list, by the entry each stopped at: N, A:B, A:, :B; or -N for the last N hits, -A:-B counted from the end (default: the last 20) |
Examples:
1 | machine break run1 hits the last 20 hits |
Prints:
1 | text ENTRY VTIME # KIND EVENT |
See also:
machine break enable
Enable a breakpoint.
The machine arms an enabled breakpoint and not a disabled one. The run log keeps the change.
1 | machine break run enable [-h] [--once | --count n] number |
| Argument | Does |
|---|---|
number |
Number of the breakpoint |
| Option | Does |
|---|---|
--once |
Disable it again after its next hit, as GDB’s enable once |
--count n |
Disable it again after N hits, as GDB’s enable count |
Examples:
1 | machine break run1 enable 2 arm stop 2 again |
See also:
machine break disable
Disable a breakpoint.
The machine arms an enabled breakpoint and not a disabled one. The run log keeps the change.
1 | machine break run disable [-h] number |
| Argument | Does |
|---|---|
number |
Number of the breakpoint |
Examples:
1 | machine break run1 disable 2 keep stop 2 and do not arm it |
See also:
machine break condition
Set the condition of a breakpoint.
A hit where the condition does not hold does not end the run. Without a condition the breakpoint has none.
1 | machine break run condition [-h] number [condition] |
| Argument | Does |
|---|---|
number |
Number of the breakpoint |
[condition] |
Condition, such as ‘$rax == 0’ or ‘counter > 3’ (default: none) |
Examples:
1 | machine break run1 condition 2 '$rax == 0' where rax is 0 |
See also:
machine break ignore
Pass the next hits of a breakpoint.
Each hit where the condition holds is counted. A counted hit is passed while the ignore count lasts. A count of 0 stops at the next.
1 | machine break run ignore [-h] number count |
| Argument | Does |
|---|---|
number |
Number of the breakpoint |
count |
Hits to pass |
Examples:
1 | machine break run1 ignore 2 5 pass the next 5 hits |
See also:
machine watch
List and manage watchpoints.
A watchpoint stops the run after a task of its tgid writes its bytes, or reads
them with --on access. The listing has the run’s watchpoints, each by its
number. Use points to list them with the breakpoints and catchpoints.
The run keeps them in its run log, as it keeps its breakpoints. A run holds 3
watchpoints and hardware breakpoints together, one in each debug register. A
step over a call takes one of the 3 while it runs. So do step --out and run
--until at= or write=.
1 | machine watch [-h] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine watch run1 add 412 counter a write of counter |
Prints:
1 | text # KIND ADDRESS SIZE WHERE TGID HITS WHEN |
See also:
machine watch list
List the watchpoints.
The listing is the one the command prints with no verb.
1 | machine watch run list [-h] |
Examples:
1 | machine watch run1 list the watchpoints, by number |
See also:
machine watch add
Add a watchpoint.
The watchpoint stops the run after an instruction writes its bytes. Use --on
change to stop only where the bytes change, as GDB’s watch does. Use --on
access to stop at a read as well. It watches the tgid of the task alone.
A variable is watched at its size. So is an expression, as a local, a member or
an element, at the bytes it names when added. A local’s watch stays after its
function returns. Any other location takes --size, of 1, 2, 4 or 8 bytes.
The run log keeps the watchpoint, and a branch has its parent’s.
1 | machine watch run add [-h] [--disabled] [--ignore count] [--temporary] |
| Argument | Does |
|---|---|
tid |
Task whose tgid the stop is in |
location |
Location, as mem takes one: an address, *ADDR, NAME, PATH:NAME, FILE:LINE or $reg, moved by +N or -N |
| Option | Does |
|---|---|
--disabled |
Add it and do not arm it |
--ignore count |
Pass this many hits where the condition holds (default: 0) |
--temporary |
Delete it at its first hit where the condition holds |
--condition condition |
Count a hit only where this expression holds, as print takes one, such as ‘$rax == 0’ |
--print format |
Print this at a hit and go on. {EXPR} prints an expression as print takes one, {EXPR:x} in a print format letter: ‘n={counter} p={$rdi:x}’ |
--size bytes |
Bytes to watch: 1, 2, 4 or 8 (default: the variable’s size) |
--on {write,change,read,access} |
What stops the run: a write, a write that changes the bytes, a read, or any access (default: write) |
Examples:
1 | machine watch run1 add 412 counter a write of the variable |
Prints:
1 | text Watchpoint N at WHERE, in tgid TGID |
See also:
machine watch delete
Delete watchpoints.
The run log keeps the delete, so a branch forked before it keeps the watchpoint.
1 | machine watch run delete [-h] number [number ...] |
| Argument | Does |
|---|---|
number [number ...] |
Number of a watchpoint |
Examples:
1 | machine watch run1 delete 3 delete watchpoint 3 |
See also:
machine watch save
Save the watchpoints as the commands that add them.
Each line is a machine command that adds one watchpoint again. The command keeps its condition, ignore count and flags. A location written by a register is saved as its address.
1 | machine watch run save [-h] file |
| Argument | Does |
|---|---|
file |
File to write, or - for the standard output |
Examples:
1 | machine watch run1 save stops.sh write the commands to a file |
See also:
machine watch hits
List the hits of watchpoints.
A hit is a watchpoint that stopped the run. It has the entry it stopped at, its machine time and the number of the watchpoint. The run log keeps each one, so a branch lists its parent’s hits before it.
Use run --back to switch onto a branch at the newest hit.
1 | machine watch run hits [-h] [-n range] |
| Option | Does |
|---|---|
-n, --entries range |
Hits to list, by the entry each stopped at: N, A:B, A:, :B; or -N for the last N hits, -A:-B counted from the end (default: the last 20) |
Examples:
1 | machine watch run1 hits the last 20 hits |
Prints:
1 | text ENTRY VTIME # KIND EVENT |
See also:
machine watch enable
Enable a watchpoint.
The machine arms an enabled watchpoint and not a disabled one. The run log keeps the change.
1 | machine watch run enable [-h] [--once | --count n] number |
| Argument | Does |
|---|---|
number |
Number of the watchpoint |
| Option | Does |
|---|---|
--once |
Disable it again after its next hit, as GDB’s enable once |
--count n |
Disable it again after N hits, as GDB’s enable count |
Examples:
1 | machine watch run1 enable 2 arm stop 2 again |
See also:
machine watch disable
Disable a watchpoint.
The machine arms an enabled watchpoint and not a disabled one. The run log keeps the change.
1 | machine watch run disable [-h] number |
| Argument | Does |
|---|---|
number |
Number of the watchpoint |
Examples:
1 | machine watch run1 disable 2 keep stop 2 and do not arm it |
See also:
machine watch condition
Set the condition of a watchpoint.
A hit where the condition does not hold does not end the run. Without a condition the watchpoint has none.
1 | machine watch run condition [-h] number [condition] |
| Argument | Does |
|---|---|
number |
Number of the watchpoint |
[condition] |
Condition, such as ‘$rax == 0’ or ‘counter > 3’ (default: none) |
Examples:
1 | machine watch run1 condition 2 '$rax == 0' where rax is 0 |
See also:
machine watch ignore
Pass the next hits of a watchpoint.
Each hit where the condition holds is counted. A counted hit is passed while the ignore count lasts. A count of 0 stops at the next.
1 | machine watch run ignore [-h] number count |
| Argument | Does |
|---|---|
number |
Number of the watchpoint |
count |
Hits to pass |
Examples:
1 | machine watch run1 ignore 2 5 pass the next 5 hits |
See also:
machine catch
List and manage catchpoints.
A catchpoint stops the run at an event of a task rather than at an address. The events are exec, exit, fork, system call, signal, fault, disk request, file mapped and C++ exception. The listing has the run’s catchpoints, each by its number. Use points to list them with the breakpoints and watchpoints.
The run keeps them in its run log, as it keeps its breakpoints. A run holds 8 catchpoints at most.
1 | machine catch [-h] run command ... |
| Argument | Does |
|---|---|
run |
Run name |
Examples:
1 | machine catch run1 the catchpoints of run1 |
Prints:
1 | text # KIND ADDRESS SIZE WHERE TGID HITS WHEN |
See also:
machine catch list
List the catchpoints.
The listing is the one the command prints with no verb.
1 | machine catch run list [-h] |
Examples:
1 | machine catch run1 list the catchpoints, by number |
See also:
machine catch add
Add a catchpoint.
The catchpoint stops the run at an event of a task. The events are exec, exit, fork, vfork, syscall, signal, load, unload, fault and io. The C++ events throw, catch and rethrow are breakpoints in one tgid.
A fault catchpoint stops where an injected fault fires. A syscall fault stops before its delay and its error. The hit names the call and the error, or delayed for a fault with no error. A site fault stops at the site, and the hit names the site.
An io catchpoint stops at a disk request as its task submits it. No disk fault has decided on the request yet, so a fault added at the stop decides on it. The hit names the request’s operation, sectors and tid in the form inject add takes them.
Use --file to stop at the requests of the files a pattern matches, and --op
for reads or writes alone. A file matches whether it is opened or mapped before
or after the catchpoint is added.
A syscall catchpoint names its system calls, and a signal catchpoint its signals, four at most. A fault catchpoint names system calls or the sites of inject add, such as disk. A load, unload or exec catchpoint names a path or a file’s name. Without one, every event of its kind stops the run.
Use --in to stop at the events of one task or one tgid alone. The run log
keeps the catchpoint, and a branch has its parent’s.
Use --in comm=NAME to stop at the events of the tasks of that comm. The comm
is read as tasks exist reads it. An exec catchpoint sees the new program’s comm.
A task renamed after exec is seen by its new comm at the next fork, exec or
exit.
An exit catchpoint sees the comm of the tgid’s leader, its last task to go. A worker of that comm in a tgid whose leader has another does not stop it. A syscall catchpoint with a comm makes the guest report every task’s calls while it is armed.
An exit is the end of a tgid, so an exit catchpoint takes --in tgid=N and not
a tid. A throw, catch or rethrow catchpoint is a breakpoint at the C++ runtime’s
function. It needs --in tgid=N and takes no values. It is one of the run’s 64
breakpoints.
1 | machine catch run add [-h] [--in tid=N|tgid=N|comm=NAME] [--file pattern] |
| Argument | Does |
|---|---|
event |
Event: exec, exit, fork, vfork, syscall, signal, load, unload, fault, io, throw, catch, rethrow |
[value ...] |
System call of a syscall or fault catchpoint, signal, by name or number, or the path of a load, unload or exec |
| Option | Does |
|---|---|
--in tid=N|tgid=N|comm=NAME |
Stop at the events of this task, tgid or comm alone (default: any task) |
--file pattern |
An io catchpoint’s requests of a file’s data: an absolute path, where * matches within a directory and ** across them (default: any file) |
--op {read,write} |
An io catchpoint’s requests of one operation (default: both) |
--disabled |
Add it and do not arm it |
--ignore count |
Pass this many hits where the condition holds (default: 0) |
--temporary |
Delete it at its first hit where the condition holds |
--condition condition |
Count a hit only where this expression holds, as print takes one, such as ‘$rax == 0’ |
--print format |
Print this at a hit and go on. {EXPR} prints an expression as print takes one, {EXPR:x} in a print format letter: ‘n={counter} p={$rdi:x}’ |
Examples:
1 | machine catch run1 add exec any task execs |
Prints:
1 | text Catchpoint N: EVENT, in tgid TGID, in tid TID or in comm NAME |
See also:
machine catch delete
Delete catchpoints.
The run log keeps the delete, so a branch forked before it keeps the catchpoint.
1 | machine catch run delete [-h] number [number ...] |
| Argument | Does |
|---|---|
number [number ...] |
Number of a catchpoint |
Examples:
1 | machine catch run1 delete 4 delete catchpoint 4 |
See also:
machine catch save
Save the catchpoints as the commands that add them.
Each line is a machine command that adds one catchpoint again. The command keeps its condition, ignore count and flags.
1 | machine catch run save [-h] file |
| Argument | Does |
|---|---|
file |
File to write, or - for the standard output |
Examples:
1 | machine catch run1 save stops.sh write the commands to a file |
See also:
machine catch hits
List the hits of catchpoints.
A hit is a catchpoint that stopped the run. It has the entry it stopped at, its machine time and the number of the catchpoint. A catchpoint’s hit also has its event. The run log keeps each one, so a branch lists its parent’s hits before it.
Use run --back to switch onto a branch at the newest hit.
1 | machine catch run hits [-h] [-n range] |
| Option | Does |
|---|---|
-n, --entries range |
Hits to list, by the entry each stopped at: N, A:B, A:, :B; or -N for the last N hits, -A:-B counted from the end (default: the last 20) |
Examples:
1 | machine catch run1 hits the last 20 hits |
Prints:
1 | text ENTRY VTIME # KIND EVENT |
See also:
machine catch enable
Enable a catchpoint.
The machine arms an enabled catchpoint and not a disabled one. The run log keeps the change.
1 | machine catch run enable [-h] [--once | --count n] number |
| Argument | Does |
|---|---|
number |
Number of the catchpoint |
| Option | Does |
|---|---|
--once |
Disable it again after its next hit, as GDB’s enable once |
--count n |
Disable it again after N hits, as GDB’s enable count |
Examples:
1 | machine catch run1 enable 4 arm catchpoint 4 again |
See also:
machine catch disable
Disable a catchpoint.
The machine arms an enabled catchpoint and not a disabled one. The run log keeps the change.
1 | machine catch run disable [-h] number |
| Argument | Does |
|---|---|
number |
Number of the catchpoint |
Examples:
1 | machine catch run1 disable 4 keep catchpoint 4 and do not arm it |
See also:
machine catch condition
Set the condition of a catchpoint.
A hit where the condition does not hold does not end the run. Without a condition the catchpoint has none.
1 | machine catch run condition [-h] number [condition] |
| Argument | Does |
|---|---|
number |
Number of the catchpoint |
[condition] |
Condition, such as ‘$rax == 0’ or ‘counter > 3’ (default: none) |
Examples:
1 | machine catch run1 condition 4 '$rax == 0' where rax is 0 |
See also:
machine catch ignore
Pass the next hits of a catchpoint.
Each hit where the condition holds is counted. A counted hit is passed while the ignore count lasts. A count of 0 stops at the next.
1 | machine catch run ignore [-h] number count |
| Argument | Does |
|---|---|
number |
Number of the catchpoint |
count |
Hits to pass |
Examples:
1 | machine catch run1 ignore 4 5 pass the next 5 hits |
See also:
Python SDK
The SDK does what the client does, from Python. If you can type it in a terminal, you can call it from a program.
Installing
The SDK runs on the SDK Core, a proprietary compiled module licensed under the Client EULA. The SDK Core is not on PyPI. The client’s installer installs it with the client, so install the client first:
1 | curl https://testflows.com/machine/install -fsS | bash |
Then install the SDK from PyPI:
1 | pip3 install testflows.machine |
It needs Python 3.11 or later, on the systems the client runs on: Linux x86_64 and arm64, and Macs with Apple Silicon. If you need another platform, contact us.
Without the SDK Core, import testflows.machine.v1 raises ImportError with
the installer’s command. Running the installer again updates the client and
the SDK Core together.
Hello Machine
1 | import testflows.machine.v1 as machine |
This creates a session, boots a run from the disk hello and prints the last
twenty lines of its console. A run waits for commands, so it moves only when
you call run.run(...); pass daemon=False to have it run to the end by
itself.
Leaving a with block deletes what it created, whether the block ends normally
or with an error. That is how you make sure a session never outlives your
program.
The same names
Every client command has a matching SDK call, and the names follow a simple rule.
| In the client | In the SDK |
|---|---|
a top-level command, machine create --disk app |
a method on the client, client.create(disk="app") |
a group, machine sessions use ci-1 |
an attribute of the client, client.sessions.use("ci-1") |
a command that takes a run, machine fork app --at ^boot |
a method on the run, run.fork(at="^boot") |
a group that takes a run, machine tasks app hold 56 |
an attribute of the run, run.tasks.hold(56) |
| a positional argument | a positional argument |
a flag like --no-wait, -w or -f |
a keyword argument like wait=False, watch=True or follow=True |
So once you know the client, you know most of the SDK. Commands
lists them, and the client’s --help says what each call takes.
Signing in
The SDK signs in with an API key from TESTFLOWS_MACHINE_API_KEY if it is set.
If it isn’t, it uses the login that machine login stored, which it shares
with the client.
A program can sign in on its own too. The one-time code arrives by email.
1 | machine.login.start("me@example.com") |
machine.login.logout() signs out.
Sessions
A client has a current session, like the CLI does. You can set it when you create the client, or later.
1 | client = machine.Client(session="ci-1") # like -s ci-1 |
sessions.use only changes the session for this client. It never changes it for
the machine command. Without a session you can still read your account’s runs:
client.runs(), client.branches() and a run’s describe(), lineage()
and checkpoints() all work.
client.sessions does what machine sessions
does. A session is a context manager, so leaving the with block deletes it:
1 | with client.sessions.create("ci-1", cpus=4, mem=8192) as session: |
| Call | Does |
|---|---|
client.sessions.create(name, cpus=..., mem=..., storage=..., class_=...) |
creates a session and waits until it runs; mem is in MB and storage in GB |
client.sessions(), client.sessions.show(name) |
lists them, or reads one |
client.sessions.keep(name, minutes) |
keeps a session running idle for that long |
client.sessions.rename(name, new_name), client.sessions.delete(name) |
renames or deletes one |
Disks
client.disks builds and manages disks, as machine disks
does. A build takes the same sources: binary, from_, image or compose.
1 | disk = client.disks.build("hello-py", from_="python:3.12-slim", |
image and compose read your local Docker, as --image and --compose do.
Disks says what each source needs.
| Call | Does |
|---|---|
client.disks(), client.disks.show(name) |
lists your disks, or reads one: name, size_bytes, state, labels |
client.disks.rename(name, new_name) |
renames one |
client.disks.delete(name) |
deletes one |
client.preload.disk(name), client.offload.disk(name) |
copies a disk into the session ahead of time, or drops the session’s copy; preload_and_offload.py shows both |
Runs
Here is a typical run, from creating it to branching and replaying.
1 | run = client.create("app", disk="app", rng=True) |
And the calls you will reach for most often:
| Call | Does |
|---|---|
client.create(name, disk=..., cpus=..., ...) |
creates a run and boots it; takes every option machine create does |
client.runs(), client.runs["app"] |
lists runs, or finds one by name |
run.run(until=..., iters=...), run.plan(file) |
drives it |
run.checkpoint(name), run.checkpoints() |
saves a checkpoint, lists them |
run.fork(...), run.go(point), run.switch(branch), run.detach(...) |
branches |
run.replay(), run.play(...), run.diff(...) |
replays and compares |
run.hash() |
the hashes of the machine as it stands, memory and the disk included, as machine hash |
run.dump(show="regs"), run.vcpu() |
the machine’s state, and where one vCPU stopped |
run.rng(), run.rng.seed(7), run.fork(at="^booted", rng_seed=7) |
the RNG device’s seed, seeding it again, and a branch whose guest draws other random numbers |
run.run(irq_delay="blk=1ms", timer_slack="0:20ms"), run.irq.pending() |
hold interrupts a drawn time, draw how late a timer may be, and list the interrupts owed and held |
run.describe(), run.state(), run.now(), run.entries(...) |
reads it |
run.output(...) |
reads what it printed |
run.output.search(pattern, from_=..., to=...), run.output.find(pattern), run.output.entry(offset) |
finds text in what it printed, between two points if you like; each match has the entry that printed it and a place to go to |
run.output.sources(), run.output(source="ttyS1"), run.output.lines(source="all") |
lists the sources that printed, reads one, and reads them all with the source of each line |
run.fs.ls(path, at=...), .cat(...), .cp(...) |
reads its disk at a checkpoint, which at names |
run.send(text, device=...) |
sends input to a device or to a file in the machine, as machine send does; returns a Sent with queued, and process for a file |
run.exec(command), run.exec(command, wait=False), run.process(id), run.processes() |
executes a command in the machine, a list for a program and a str for a shell line, and returns its Process: pid, exit_code, stdout, stderr, events, exits |
run.exec(command, container="db", user="root") |
executes it in a container, named by its Compose service or by its own name, as a user of it |
run.exec(command, net="db") |
runs a program of the machine on db’s network, such as tcpdump on its eth0 |
run.core(pid, dest, container=...) |
gets a core dump of a process into a file; the process dies of it |
run.profile("2s", container=..., pid=...) |
samples what runs for a time and returns a Profile: samples, functions, stacks, and folded text for flame graph tools |
run.profile(from_="^request", to="^reply") |
profiles a stretch the run already recorded, without changing the run |
run.profile("2s", in_guest=True) |
samples with a process in the run, for the functions of the kernel by name |
run.containers(), run.containers.kill("db", restart="5s"), run.containers.crash("db", restart="0") |
lists the containers, the Compose service of each, and its addresses and links; stop, kill, crash, start, pause, resume, throttle, memory, pids, ports and files put a fault on one, as machine containers does |
run.put(source, path, container=..., mode=..., owner=...) |
puts a file of any size into the machine or into a container of it; source is bytes or the path of a file |
run.get(path, dest, container=...) |
gets a file out, as it is now; returns a Got with size, mode, uid, gid, and data when no dest is given |
run.tasks(), run.focus.add(...), run.irq(...), run.vtime.rate(...) |
steers the machine |
run.inject.add("syscall", "read", "valid", in_="/usr/bin/app", pick=1, prob=0.1), run.inject.add("syscall", "read", delay="0:20ms", in_="/usr/bin/app"), run.inject.add("disk", in_="all", prob=0.01), run.inject.add("function", "malloc", ret=0, prob=0.01, in_="/usr/bin/app"), run.inject.add("disk-corrupt", in_="all", file="/var/lib/db/*/data.bin", op="write", bytes=8, times=1), run.inject(), run.inject.delete(2), run.inject.clear() |
injects faults as machine inject does: system calls fail with chosen errors or wait first, a program’s function returns ret without running, and the kernel’s disk, allocation, usercopy and futex-access sites fail, and the disk corrupts, tears or loses data; lists the faults with their counts, and deletes them; in_ takes the words --in takes, delay the span --delay takes, at the tid whose files hold the function, as --at takes it, and file, op, sectors and bytes the disk options |
run.tasks.exist("/usr/bin/app") |
returns the tasks running that program, comm, tid or tgid now, or raises NotFound; it changes nothing |
run.task(tid) |
a guest task to debug, by its tid: registers(), read(address, size), address("main+4"), symbol(address), symbols(regex="^wait"), backtrace(), libraries() |
task.set(rax=0, rip="main+4"), task.write(address, data) |
writes its registers and its tgid’s memory; a replay makes the writes again |
task.print("shape->at.x", frame=1), task.print("counter = 3"), task.locals(args=True) |
an expression in a frame, a write with TARGET = VALUE, the locals or the arguments of a frame |
task.step(), task.step(line=True, over=True), task.finish() |
steps an instruction or a line, over calls, or out of the function |
task.back(), task.back(line=True, over=True) |
goes back over the task’s code onto a branch, as reverse-stepi and reverse-next |
task.call("strlen", "buf"), task.return_(0) |
calls a function in the task, or returns from its innermost one |
task.breakpoint("main"), task.watch("counter", on="access") |
stops its tgid at a location, or at a write or an access of its bytes; run.run(...).hits names the one the run ended at |
run.breakpoints.catch("fork", in_="tgid=1"), run.breakpoints.catch("io", file="/db/*/data.bin", op="write") |
stops at an event of a task: exec, exit, fork, vfork, syscall, signal, load, unload, fault, io, throw, catch, rethrow; in_ takes the words --in takes, tid=N, tgid=N or comm=NAME; file and op are an io catchpoint’s, as --file and --op |
run.breakpoints.list(), .enable(2, False), .condition(2, "n > 3"), .ignore(2, 5), .delete(2), .hits() |
the run’s stops, changed one at a time, and the hits the run log keeps |
run.run(iters=0), run.run(until="tid=412:at=main"), run.back() |
runs to the next hit, or until a task reaches a location; goes back onto a branch at the newest hit |
run.tasks.signal(412, "SIGTERM") |
sends a task a signal; a replay sends it at the same instruction |
run.start(), run.stop(), run.pause(), run.kill(), run.delete() |
lifecycle |
run.start(at="^booted"), run.start(at="^booted", read_only=True), run.start(recover=False) |
replays from a checkpoint to the tip, opens a checkpoint to read, or refuses to replay the tail, as machine start does |
run.fork(from_="^booted", name="b"), run.rewind("^booted") |
a branch forked at the checkpoint before a point and replayed to it; the run put back at a checkpoint |
run.sync(), run.cancel(), run.cancel(soft=True) |
publishes the run to your storage; cancels the drive or checkpoint in progress, at once or, with soft=True, at the end of the dispatch in progress so the run stays repeatable |
client.create(disk="app", instr_margin=100000, dispatch_bound="2s"), run.replay(instr_margin=100000), run.describe().repeats_to |
sets where a recording stops, as Recording that repeats describes; replays with that margin; the first entry where recording the run again stops repeating, None when it repeats to its tip |
run.keep() |
keeps a run created in a with block when the block exits, such as one that just failed, with its tree |
run.marks.add("good"), run.marks.list(), run.marks.remove("good"), client.marks.list() |
names a branch, as Marks describes; the last lists every mark |
run.tasks.hold(56), .release(56), .yield_(56), .slice(56, "10us"), .pin(56, "2") |
steers one thread, as Holding threads describes |
run.syscalls.counts("/usr/bin/app"), run.ints.counts(...), run.vtime.add("5s") |
counts a task’s system calls or interrupts; pushes the clock forward |
run.debug.log(lines=-20), run.debug.err(), run.control.log() |
the run’s debug output, its error log, and the commands issued to it |
Every call returns a record with named fields, such as a RunInfo from
describe() or a list of TaskInfo from tasks(). Call as_json() on one to get
a plain dict.
run.run(...) and run.plan(...) return a RunResult, which says how far the
drive went. An iteration has no fixed length, so iterations alone does not
tell you:
| Field | Is |
|---|---|
iterations |
the iterations that ran |
vtime_elapsed |
how far the machine’s clock moved, in nanoseconds |
reached |
whether until held; False when iters ran out first |
clauses |
each clause the drive drew from, with the iterations it applied to |
entry_id |
the entry the drive ended at |
entries |
where each vCPU stopped |
hits |
the breakpoints, watchpoints and catchpoints the drive ended at; empty when none ended it |
printed |
the lines printed by stops that print, in the order the stops hit |
1 | done = run.run(iters=300, when="/app:on", mode="step") |
until takes the conditions machine run --until does,
including a machine time: run.run(until="vtime=+10ms").
Bounding a drive
within bounds a drive by the machine’s clock, as --within does. With
until, the one that happens first ends the drive, and reached says which. The machine checks the bound itself, so a slow host never ends the drive
early:
1 | done = run.run(until="console~ready", within="30s") |
timeout is different: it is how many seconds of your time the call waits,
and it raises Timeout when they run out. To wait for a run that something
else drives, such as another program or a run stopping in the background, use
run.wait:
1 | run.stop(wait=False) |
for_ takes the predicates machine wait does: ready, stopped,
halted, entry>=N, vtime>=T, console~REGEX, or a task event such as
/app:exit.
Scheduling from Python
Every option of machine run is a keyword. The draws described in
Schedules look like this, on a run whose
program was focused with run.focus.add("/app") before booted, so that its
branches have the focus too:
1 | for seed in range(10): |
The explore_seeds.py example keeps the branches that fail and marks them.
Faults
run.inject adds the faults Fault injection
describes, with the same subsystems and options as keywords. Check the target
first with run.tasks.exist, which raises NotFound when nothing runs it:
1 | run.tasks.exist(f"tgid={pid}") |
A C library opens a file with open or openat, depending on the library,
so the fault takes both. glibc’s open() makes no open call, so the add
issues a UserWarning that names openat; the fault on open is added all
the same. matched counts the calls the fault saw and fired the ones it
failed. A disk fault takes the file and the operation. The kernel’s flusher
sends an ordinary write, so the example syncs it, and the request is dd’s own:
1 | run.inject.add("disk", in_="comm=dd", file="/var/data.bin", op="write") |
The fault_injection.py example runs both and prints what the program saw.
on and off make a fault cycle:
run.inject.add("disk", in_="all", on="2s", off="5s") fails every disk
request for 2 seconds in every 7 of machine time. Each Fault says where it
is: active, and left_ns, the time left in that window.
Container faults
run.containers puts a fault on one service of a Compose project, as
Container faults describes, while the
others keep running. A crash loses what the service had not synced to the
volumes its project names under x-machine:
1 | run.exec("echo synced > /data/kept && sync", container="app") |
stop, kill, start, pause, resume, throttle, memory, pids, ports
and files work the same way.
stop and kill take restart= as crash does, stop takes grace=,
fresh=True starts a new container instead of the old one, pause and
throttle take for_=, throttle takes cpus=, and memory("db", "64m", for_="30s") limits a container’s memory. pids("db", 20),
ports("api", 40000, 40009) and files("api", 64) run a container out of
processes, local ports or open files, and each takes for_= too.
run.containers() lists the containers with the service, name, state and
limits of each. The container_crash.py example builds a
project, crashes one service and reads its volume after the restart.
Clock faults
run.clock changes the guest’s date or one container’s, as
Clock faults describes. Each method returns the date as it
stands, with the change waiting for the run to execute:
1 | run.clock.jump("+59m50s") |
run.clock(container="db") reads one container. ClockInfo.containers lists
each container off the guest’s date with its offset, drift, slew and
jitter, and ClockInfo.jitter and ClockInfo.leap are the guest’s.
The guest kernel takes a change when the run executes, so read the clock after
that run to see whether it was refused. ClockInfo.refused names each change
of the last batch the guest refused, with its errno and the reason:
1 | run.clock.set("2038-01-19T03:13:30Z") |
To find how much skew a system tolerates, fork one checkpoint into branches, one skew each:
1 | for skew_ms in (50, 100, 200, 400, 800): |
Network faults
run.net puts faults on the links between a run’s containers, as
Network faults describes. A fault acts on every packet a
container sends, or on those a match takes. A match is any of to, a peer;
proto, tcp, udp, icmp or arp; port, a source or destination
port; and syn=True, the TCP SYNs that open a connection:
1 | run.net.delay("db", "100ms", jitter="20ms", seed=7) |
delay, loss, duplicate, corrupt, reorder, rate and hold
return a LinkChange, the link before and after; partition returns one
for each link it cuts, and heal and release return the links as they
were before. duplicate, corrupt and reorder take a share as loss does,
and rate takes bits a second, such as "1mbit". A Link gives its delay
and jitter in nanoseconds and its shares in parts per billion, and what the
fault did since it was set: sent, dropped, and queued, the packets it
holds now. A replay delays and drops the same packets.
log=True on a fault writes each of its decisions to the container’s file
under /output/drop/.machine/net, and log=False stops it, as --log and
--no-log do:
1 | run.net.loss("db", "30%", to="nest", seed=7, log=True) |
hold keeps a link’s packets until release, or for for_, and they then
leave together. With no name, hold holds every link where the machine
stopped, until release with no name. reset aborts TCP connections and
returns how many:
1 | run.net.hold("node1", for_="10s") # node1's messages arrive late, in a burst |
run.net.arp shows and changes a container’s neighbour table, and returns
the table as Neighbor entries:
1 | run.net.loss("db", "100%", proto="arp", to="api") # db's ARP to api lost |
Stopping at an event
A catchpoint stops the run at an event of a task, as
machine catch does. The drive that reaches it
returns the hit in hits:
1 | stop = run.breakpoints.catch("syscall", "open", "openat", in_="comm=cat") |
The kinds are exec, exit, fork, vfork, syscall, signal, load,
unload, fault, io, throw, catch and rethrow. An io catchpoint
takes file= and op=, and stops at a file’s disk request before any disk
fault decides on it:
1 | run.breakpoints.catch("io", file="/var/lib/db/*/data.bin", op="write") |
Debugging a task
run.task(tid) is a task to debug while the machine is stopped, by its tid,
as Debugging describes for the command line:
1 | task = run.task(412) |
The debug_task.py example stops at a catchpoint, reads the registers of the
task that hit it, reads the path it was opening from its memory, and prints
its stack.
Points in a run’s history
Calls that take a point, such as fork(at=...), go, rewind,
replay(to=...) and profile(from_=..., to=...), take it as text, "^booted",
"#300", "1.194s" or "-10ms", or as one of three records:
| Record | Is | Works on |
|---|---|---|
Point |
a time: vtime, the machine’s clock in nanoseconds |
every run; each finds its own entry at that time |
Place |
one entry: the run_id that recorded it, entry_id, point and vtime_offset |
the run that recorded it, and branches forked after it |
Checkpoint |
a saved state: its name, and the place it was saved at |
wherever its place works |
start(at=...) and the fs readers take only a checkpoint, by its name or as
a Checkpoint: they open a saved state, and a time or an entry is not one.
A point moves by a duration: an int of nanoseconds, a string such as "10ms",
or a timedelta. Subtract one point from another for the nanoseconds between
them. run.now() returns a Place, and run.place(point) finds the entry a
point names.
1 | booted = run.checkpoint("booted") |
A place is one run’s entry, so once the parent and the branch have both run
past the fork, run.place(here) is refused: the parent’s entry with that id
is another state. Pass here.point to take the same time instead.
Watching and following
A listing takes watch=True and then gives you a new list every time
something changes. A log takes follow=True and gives you new lines as they
come, like tail -f.
1 | for runs in client.runs(states=["running"], watch=True, timeout=60): |
Choosing entries
Anything that reads entries takes the same few spellings.
| You write | Means |
|---|---|
200 or (100, 200) |
that entry, or that range, both ends included |
"100:200", "100:", ":200" |
a range, from 100 on, or up to 200 |
"-20" or "-20:" |
the last 20 entries |
"-20:-2" |
the 20th entry from the end through the 2nd |
slice(100, 200) |
a Python slice, so 100 through 199 |
slice(-20, -1) |
a Python slice from the end, so the same as "-20:-2" |
Be careful with the slice rows. A pair and a string include both ends, the way
Machine’s own messages do. A slice is Python’s, so it stops before the end.
That makes (100, 200) one entry longer than slice(100, 200).
Both ends of a range count from the same end: "-20:5" and slice(-20, 5)
are refused.
lines= selects lines of output in the same grammar: 20 is line 20, -20
the last 20, "-10:-2" the tenth line from the end through the second, and
0 none, which starts a follow at the end.
Cleaning up
A session costs from the moment you create it, and a run keeps its log and checkpoints until you delete it. Both can be used as context managers, as in the first example, and deleting a run takes its branches with it.
If you want to keep a run that found something, don’t wrap it in a with
block. Nothing is deleted until you call delete().
Errors
Every exception the SDK raises is a machine.Error, and each one says what
went wrong.
| Exception | Raised when |
|---|---|
NotSignedIn |
there are no credentials |
Unauthorized |
a key or a sign-in code was rejected |
Forbidden |
your account is not allowed to do that |
NotFound |
nothing matches the name or id |
AlreadyExists |
something with that name exists |
InvalidArgument |
an argument is wrong, or a name matches more than one thing |
NotRunning |
it exists, but it is not running |
Conflict |
the request does not fit the thing’s current state |
NoCapacity |
no host of the session’s class has room for it now; the same request can succeed later, or a smaller session or another class may fit |
Diverged |
play() could not follow the recording; replay() returns a divergence in its result instead |
Timeout |
a wait ran out of time; the work goes on in the service |
ServiceUnavailable |
the service could not be reached |
ApiError |
the service refused the request |
A replay that does not reproduce the recording is a result, not an exception, so check it:
1 | result = branch.replay() |
The exceptions you will usually handle are the ones with a way out. A full class can be retried or sized down, and a run that stopped can be started again:
1 | try: |
The error_handling.py example goes through every exception.
Examples
Small example scripts are in
sdk/python/examples of the TestFlows
Machine repository. Each one creates its own session and
deletes it at the end. Most run as python <script> --disk <name>.
disks.py builds the disk first and takes --binary and --name.
container_crash.py builds its own disk too and takes --name, and deletes
the disk at the end.
| Script | Shows |
|---|---|
hello.py |
creates a run, boots it, reads its console |
disks.py |
builds a disk from a static executable, boots a run from it, renames the disk and deletes it |
checkpoint_and_fork.py |
a checkpoint after boot, and branches from it |
replay_verify.py |
replays a recording and names what diverged |
explore_seeds.py |
one branch per seed, with the failing ones marked and kept |
force_interleaving.py |
holds one thread while another runs past it |
plan_file.py |
drives a run with when and else clauses |
watch_and_follow.py |
watches a listing and follows a console |
syscall_profile.py |
counts a program’s system calls and interrupts |
read_fs.py |
reads and copies files from a checkpoint’s disk |
machine_time.py |
changes the machine’s clock (vtime) rate and pushes it forward |
inject_interrupt.py |
injects an interrupt and replays it |
stop_and_resume.py |
what stop, kill, start and sync keep |
resume_elsewhere.py |
stops a run in one session and starts it in another |
preload_and_offload.py |
preloads a disk onto the session, then drops the session’s copy |
ci_job.py |
a CI job with an API key, a session per job and an exit code |
error_handling.py |
every error and how to recover from it |
fault_injection.py |
fails a program’s opens with a syscall fault and its synced write with a disk fault, and prints what the program saw and the fault counts |
container_crash.py |
builds a Compose project with an x-machine volume, crashes one service, and reads what the volume kept; takes --name for the disk it builds |
debug_task.py |
stops at a catchpoint on open, reads the task’s registers, memory and stack |
License
The SDK is licensed under Apache 2.0, and its source is in
sdk/python of the TestFlows Machine repository.
The SDK Core it runs on is proprietary
and is not part of the SDK. It comes with the client, under the
Client EULA.
ClickHouse is a registered trademark of ClickHouse, Inc. See clickhouse.com.