Backed by mandō

See Everything Beneath the Surface

Substrate is the definitive system call telemetry platform. Every open(), every read(), every write(). Every signal, every fork, every byte that crosses the user/kernel boundary — captured, decoded, and rendered in real time.

4.2B+ syscalls traced
380+ syscall decoders
<50ns trace overhead
depth of insight

Trusted by teams debugging the impossible

Netflix SRE Google Kernel Meta Infra Cloudflare Red Hat CERN

The Complete Telemetry
Toolkit

Every tool you need to observe, decode, and understand what your processes are really doing at the kernel level.

Full Process Tracing

Trace any command from launch to exit. Every system call, its arguments, return values — printed, decoded, and symbolically annotated. Just substrate command.

$ substrate ls -la /tmp
execve("/bin/ls", ["ls", "-la", "/tmp"], ...)
openat(AT_FDCWD, "/tmp", O_RDONLY) = 3

Live Attach

Attach to any running process by PID. No restarts, no recompilation. Begin observing in real time with -p PID.

Precision Filtering

Filter by syscall category — %file, %network, %process, %memory, %signal — or individual calls. See only what matters.

Fork Following

Follow child processes through fork(), vfork(), and clone(). The -f flag traces the entire process tree.

Timestamp Precision

Wall clock (-t), microsecond (-tt), or epoch (-ttt) timestamps. Relative timing with -r. Syscall duration with -T.

Stack Traces

See the call stack behind every syscall with -k. Source-level traces with -kk for debug builds. Know why it happened.

FD Decoding

Decode file descriptors to paths (-y), socket details, device numbers, pidfds, and more with -yy. No more guessing what fd 7 is.

Fault Injection

Inject errors, delays, and signals into system calls with --inject. Simulate failures, test resilience, reproduce race conditions.

From Opaque Process
to Crystal Clarity

Three steps to complete kernel-level observability.

01

Target

Launch a new command or attach to a running process by PID. Configure syscall filters, set output files, choose your timestamp precision.

substrate -f -e trace=%file,network -o trace.log -p 1234
02

Observe

Substrate captures every system call crossing the user/kernel boundary. Arguments are decoded symbolically, structures are dereferenced, file descriptors resolved to paths.

openat(AT_FDCWD, "/etc/passwd", O_RDONLY) = 3
read(3</etc/passwd>, "root:x:0:0:..."..., 4096) = 2847
03

Understand

Profile with -c for a statistical summary. Identify bottlenecks, excessive syscalls, and failure patterns. Stack traces reveal the code paths responsible.

% time calls errors syscall
42.18 891 12 read
31.07 432 0 write

Syscall Category
Explorer

Explore system call categories and discover what Substrate can trace. Click a category to reveal its syscalls.

%file -e trace=%file
Traces all system calls that take a file name as an argument.

Performance
Profiler

The -c flag generates a statistical summary of all system calls. Substrate renders it as a living performance dashboard.

$ substrate -c -f ./application ● LIVE
% time seconds calls errors syscall

Watch It Trace

Real strace output, decoded and annotated in real time.

substrate — strace -f -y -tt ./server
$ substrate -f -y -tt ./server

Choose Your
Trace Depth

From single-process debugging to fleet-wide telemetry.

Observer
Free
Single-process tracing for individual developers.
  • Basic syscall tracing
  • File & network filters
  • Timestamp modes (-t, -tt)
  • Output to file (-o)
  • Community support
Start Free
Omniscient
$199/mo
Fleet-wide telemetry for infrastructure teams.
  • Everything in Analyst
  • Multi-PID attachment
  • Hex dump (--read/--write)
  • Daemonised tracing (-DDD)
  • Syscall tampering
  • Custom string limits
  • Dedicated account manager
  • SLA guarantee
Contact Sales

Frequently Asked
Questions

What exactly does Substrate trace?

Every system call your process makes to the kernel — open(), read(), write(), mmap(), fork(), socket(), and hundreds more. Arguments are decoded symbolically, structures are dereferenced, flags are broken down into their bitwise components. Signals received by the process are also captured.

Does it slow down the traced process?

Yes — tracing introduces overhead because every syscall must cross through ptrace. However, the --seccomp-bpf option dramatically reduces overhead by using the kernel's seccomp facility to filter at the kernel level, only stopping for syscalls you actually want to trace.

Can I attach to a process I don't own?

You need appropriate permissions. Root can trace any process. For non-root users, ptrace scope settings (/proc/sys/kernel/yama/ptrace_scope) may restrict attachment to child processes only. Substrate can be installed setuid for controlled environments.

What does the -c flag actually show?

The -c flag suppresses normal output and instead prints a statistical summary when the process exits: percentage of time, total time, average time, call count, and error count for each system call. It's perfect for identifying which syscalls are causing performance bottlenecks.

How does fault injection work?

The --inject option lets you inject errors (error=ENOENT), return values (retval=0), signals (signal=SIGSEGV), and delays (delay_enter=100ms) into specific system calls. You can target specific invocations with when=3+ to only affect the 3rd and subsequent calls.

What's the difference between -D, -DD, and -DDD?

-D runs strace as a grandchild of the tracee (reduces visibility). -DD also puts strace in a separate process group (survives kill signals). -DDD runs in a completely separate session — true daemonisation for production tracing.

See Through
to the Kernel

Command #94. The final command. The deepest trace.
Join the Substrate telemetry program.

A mandō portfolio company.