Substrate is the definitive system call telemetry platform. Every
open(), every read(), every write(). Every signal, every
fork, every byte that crosses the user/kernel boundary — captured, decoded, and rendered in real
time.
Trusted by teams debugging the impossible
Every tool you need to observe, decode, and understand what your processes are really doing at the kernel level.
Trace any command from launch to exit. Every system call, its arguments, return values — printed,
decoded, and symbolically annotated. Just substrate command.
$ substrate ls -la /tmpexecve("/bin/ls", ["ls", "-la", "/tmp"], ...)openat(AT_FDCWD, "/tmp", O_RDONLY) = 3
Attach to any running process by PID. No restarts, no recompilation. Begin observing in real time
with -p PID.
Filter by syscall category — %file, %network, %process,
%memory, %signal — or individual calls. See only what matters.
Follow child processes through fork(), vfork(), and
clone(). The -f flag traces the entire process tree.
Wall clock (-t), microsecond (-tt), or epoch (-ttt)
timestamps. Relative timing with -r. Syscall duration with -T.
See the call stack behind every syscall with -k. Source-level traces with
-kk for debug builds. Know why it happened.
Decode file descriptors to paths (-y), socket details, device numbers, pidfds, and
more with -yy. No more guessing what fd 7 is.
Inject errors, delays, and signals into system calls with --inject. Simulate
failures, test resilience, reproduce race conditions.
Three steps to complete kernel-level observability.
Launch a new command or attach to a running process by PID. Configure syscall filters, set output files, choose your timestamp precision.
substrate -f -e trace=%file,network -o trace.log -p 1234Substrate captures every system call crossing the user/kernel boundary. Arguments are decoded symbolically, structures are dereferenced, file descriptors resolved to paths.
openat(AT_FDCWD, "/etc/passwd", O_RDONLY) = 3read(3</etc/passwd>, "root:x:0:0:..."..., 4096) = 2847
Profile with -c for a statistical summary. Identify bottlenecks, excessive
syscalls, and failure patterns. Stack traces reveal the code paths responsible.
% time calls errors syscall 42.18 891 12 read 31.07 432 0 write
Explore system call categories and discover what Substrate can trace. Click a category to reveal its syscalls.
The -c flag generates a statistical summary of all system calls. Substrate renders it as
a living performance dashboard.
$ substrate -c -f ./application
● LIVE
Real strace output, decoded and annotated in real time.
From single-process debugging to fleet-wide telemetry.
Every system call your process makes to the kernel — open(), read(),
write(), mmap(), fork(), socket(), and
hundreds more. Arguments are decoded symbolically, structures are dereferenced, flags are broken
down into their bitwise components. Signals received by the process are also captured.
Yes — tracing introduces overhead because every syscall must cross through ptrace. However, the
--seccomp-bpf option dramatically reduces overhead by using the kernel's seccomp
facility to filter at the kernel level, only stopping for syscalls you actually want to trace.
You need appropriate permissions. Root can trace any process. For non-root users, ptrace scope
settings (/proc/sys/kernel/yama/ptrace_scope) may restrict attachment to child
processes only. Substrate can be installed setuid for controlled environments.
The -c flag suppresses normal output and instead prints a statistical summary when
the process exits: percentage of time, total time, average time, call count, and error count for
each system call. It's perfect for identifying which syscalls are causing performance
bottlenecks.
The --inject option lets you inject errors (error=ENOENT), return
values (retval=0), signals (signal=SIGSEGV), and delays
(delay_enter=100ms) into specific system calls. You can target specific invocations
with when=3+ to only affect the 3rd and subsequent calls.
-D runs strace as a grandchild of the tracee (reduces visibility). -DD
also puts strace in a separate process group (survives kill signals). -DDD runs in
a completely separate session — true daemonisation for production tracing.
Command #94. The final command. The deepest trace.
Join the Substrate telemetry program.
A mandō portfolio company.