https://wiki.freebsd.org/DTrace/One-Liners
The following library of DTrace one-liners were last tested on FreeBSD 10.0. In FreeBSD 12.0, dwatch was added to the base.
Version
# Show dtrace version
dtrace -V
# Show dwatch version
dwatch -V
Listing Probes
# List probes and search for string "foo":
dtrace -l | grep foo
# Summarize probes by providers:
dtrace -l | awk '{ print $2 }' | sort | uniq -c | sort -n
# List probes for a particular provider:
dtrace -l -P syscall
Using dwatch:
# List probes and search for string "foo":
dwatch -lr foo
# List unique providers:
dwatch -lP
# List probes for a particular provider:
dwatch -lP syscall
# List unique functions:
dwatch -lf
# List probes for functions ending in read:
dwatch -lf '*read'
# List probe-tuples ending in read using regex:
dwatch -lr 'read$'
# Watch probe traversal for a given command:
dwatch -F 'pid$target:::' -c true
Syscalls
# Trace file opens with process and filename:
dtrace -n 'syscall::open*:entry { printf("%s %s", execname, copyinstr(arg0)); }'
# Count system calls by program name:
dtrace -n 'syscall:::entry { @[execname] = count(); }'
# Count system calls by syscall:
dtrace -n 'syscall:::entry { @[probefunc] = count(); }'
# Count system calls by syscall, for PID 123 only:
dtrace -n 'syscall:::entry /pid == 123/ { @[probefunc] = count(); }'
# Count system calls by syscall, for all processes with a specific program name ("nginx"):
dtrace -n 'syscall:::entry /execname == "nginx"/ { @[probefunc] = count(); }'
# Count system calls by PID and program name:
dtrace -n 'syscall:::entry { @[pid, execname] = count(); }'
# Find the parent of a process calling a syscall
dtrace -n 'syscall::read:entry { printf("execname=%s parent_execname=%s", execname, curthread->td_proc->p_pptr->p_comm); }'
# Summarize requested read() sizes by program name, as power-of-2 distributions (bytes):
dtrace -n 'syscall::read:entry { @[execname] = quantize(arg2); }'
# Summarize returned read() sizes by program name, as power-of-2 distributions (bytes or error):
dtrace -n 'syscall::read:return { @[execname] = quantize(arg1); }'
# Summarize read() latency as a power-of-2 distribution by program name (ns):
dtrace -n 'syscall::read:entry { self->ts = timestamp; } syscall::read:return /self->ts/ {
@[execname, "ns"] = quantize(timestamp - self->ts); self->ts = 0; }'
# Summarize read() latency as a linear distribution (0 to 1000, step 5) by program name (ms):
dtrace -n 'syscall::read:entry { self->ts = timestamp; } syscall::read:return /self->ts/ {
@[execname, "ms"] = lquantize((timestamp - self->ts) / 1000000, 0, 1000, 5); self->ts = 0; }'
# Summarize read() on-CPU duration as a power-of-2 distribution by program name (ns):
dtrace -n 'syscall::read:entry { self->ts = vtimestamp; } syscall::read:return /self->ts/ {
@[execname, "ns"] = quantize(vtimestamp - self->ts); self->ts = 0; }'
# Count read() variants that "nginx" is using (if previous one-liners didn't work):
dtrace -n 'syscall::*read*:entry /execname == "nginx"/ { @[probefunc] = count(); }'
# Summarize returned pread() sizes for "nginx" as distributions (bytes or error):
dtrace -n 'syscall::pread:return /execname == "nginx"/ { @ = quantize(arg1); }'
# Count socket accept() variants by process name:
dtrace -n 'syscall::*accept*:return { @[execname] = count(); }'
# Count socket connect() variants by process name:
dtrace -n 'syscall::*connect*:return { @[execname] = count(); }'
# Summarize returned pread() sizes for "nginx"... and label the output:
dtrace -n 'syscall::pread:return /execname == "nginx"/ { @["rval (bytes)"] = quantize(arg1); }'
Using dwatch:
# Trace file opens with process and filename:
dwatch -X open
# Find the parent of a process calling a syscall
dwatch -R syscall::read:entry
# Watch syscall probe traversal:
dwatch -F syscall
# Watch syscall traversal by ruby processes:
dwatch -k 'ruby*' -F syscall
# Watch syscall traversal by processes containing `daemon' in their name:
dwatch -k '*daemon*' -F syscall
# Watch open and openat syscalls until pid 1234 exits:
dwatch -X open -- -p 1234
Process Tracing
# Trace new processes showing program name (and args if available):
dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'
# Count process-level events:
dtrace -n 'proc::: { @[probename] = count(); }'
# Find the kernel function returning EINVAL for a given process
dtrace -n 'fbt:::return /(int)arg1 == EINVAL && pid == $pid/ { stack(); }' -c ./a.out
Using dwatch:
# Trace new processes showing program name (and args if available):
dwatch exec-success
# Trace new processes showing parent program and command executed:
dwatch execve
# Trace new processes showing mini process-tree for command executed:
dwatch -R execve
# Trace new processes belonging to root super-user:
dwatch -u root execve
# Trace new processes belonging to wheel super-group:
dwatch -g wheel execve
# Trace new processes belonging either to user daemon or nobody:
dwatch -u '1|65534' execve
# Trace new processes belonging either to group daemon or nobody:
dwatch -g '1|65534' execve
# Trace new processes running inside jail named myjail:
dwatch -j myjail execve
# Trace new processes running unjailed (jailed processes ignored):
dwatch -j 0 execve
# Trace new processes forked by pid 1234:
dwatch -p 1234 execve
# Trace new processes forked by either pid 1234 or a child of pid 1234:
dwatch -R -p 1234 execve
# Trace new processes forked by either pid 1234 or 5678:
dwatch -p '1234|5678' execve
# Trace new processes matching either mkdir or rmdir using regex:
dwatch -z '(mk|rm)dir' execve
Signals
Using dwatch:
# Watch signals being passed to kill syscall:
dwatch -X kill
# Watch signals being passed between bash(1) and vi(1):
dwatch -k bash -k vi -X kill
# Display the first process to call kill syscall and then exit:
dwatch -N 1 kill
Profiling
# Count sampled thread names on-CPU at 997 Hertz:
dtrace -n 'profile-997 { @[stringof(curthread->td_name)] = count(); }'
# Count sampled non-idle thread names on-CPU at 997 Hertz:
dtrace -n 'profile-997 /!(curthread->td_flags & 0x20)/ { @[stringof(curthread->td_name)] = count(); }'
# Count sampled on-CPU kernel stacks at 99 Hertz:
dtrace -n 'profile-99 /arg0/ { @[stack()] = count(); }'
# Count sampled process names and on-CP user stacks at 99 Hertz:
dtrace -n 'profile-99 /arg1/ { @[execname, ustack()] = count(); }'
Using dwatch:
# Trace statfs for 5 minutes and then exit:
dwatch -T 5m statfs
Storage I/O
# Count kernel stacks leading to block device I/O:
dtrace -n 'io:::start { @[stack()] = count(); }'
Using dwatch:
# Trace kernel stacks leading to block device I/O:
dwatch -X io-start
Scheduler
# Count kernel stacks leading to a context-switch off-CPU:
dtrace -n 'sched:::off-cpu { @[stack()] = count(); }'
Using dwatch:
# Watch processes entering system CPU scheduler:
dwatch on-cpu
# View the first 100 scheduler preemptions with less:
dwatch -y -N 100 preempt | less -R
IP
# Count IP-level events:
dtrace -n 'ip::: { @[probename] = count(); }'
Using dwatch:
# Trace IP-level events:
dwatch -X ip
UDP
# Count UDP-level events:
dtrace -n 'udp::: { @[probename] = count(); }'
Using dwatch:
# Trace UDP sent messages by remote IP address:
dwatch -X udp-send
# Trace UDP received messages by remote IP address:
dwatch -X udp-receive
TCP
# Count TCP-level events:
dtrace -n 'tcp::: { @[probename] = count(); }'
# Trace TCP accepted connections by remote IP address:
dtrace -n 'tcp:::accept-established { trace(args[3]->tcps_raddr); }'
# Count TCP passive opens by remote IP address:
dtrace -n 'tcp:::accept-established { @[args[3]->tcps_raddr] = count(); }'
# Count TCP active opens by remote IP address:
dtrace -n 'tcp:::connect-established { @[args[3]->tcps_raddr] = count(); }'
# Count TCP sent messages by remote IP address:
dtrace -n 'tcp:::send { @[args[2]->ip_daddr] = count(); }'
# Count TCP received messages by remote IP address:
dtrace -n 'tcp:::receive { @[args[2]->ip_saddr] = count(); }'
# Summarize TCP sent messages by IP payload size, as a power-of-2 distribution:
dtrace -n 'tcp:::send { @[args[2]->ip_daddr] = quantize(args[2]->ip_plength); }'
Using dwatch:
# Trace TCP accepted connections by remote IP address:
dwatch -X tcp-accept-established
# Trace TCP active opens by remote IP address:
dwatch -X tcp-connect-established
# Trace TCP sent messages by remote IP address:
dwatch -X tcp-send
# Trace TCP received messages by remote IP address:
dwatch -X tcp-receive
# Trace TCP activity while given nc command runs:
dwatch -X tcp -- -c "nc -zvw10 google.com 22"
Kernel Locks
# Sum kernel adaptive lock block time by process name (ns):
dtrace -n 'lockstat:::adaptive-block { @[execname] = sum(arg1); }'
# Summarize adaptive lock block time distribution by process name (ns):
dtrace -n 'lockstat:::adaptive-block { @[execname] = quantize(arg1); }'
# Sum kernel adaptive lock block time by kernel stack trace (ns):
dtrace -n 'lockstat:::adaptive-block { @[stack()] = sum(arg1); }'
# Sum kernel adaptive lock block time by lock name (ns):
dtrace -n 'lockstat:::adaptive-block { @[arg0] = sum(arg1); } END { printa("%40a %@16d ns\n", @); }'
# Sum kernel adaptive lock block time by calling function (ns):
dtrace -n 'lockstat:::adaptive-block { @[caller] = sum(arg1); } END { printa("%40a %@16d ns\n", @); }'
Namecache
# Count namecache lookups by program
dtrace -n 'vfs:namecache:lookup: { @missing[execname] = count(); }'
# Count namecache misses by program
dtrace -n 'vfs:namecache:lookup:miss { @missing[execname] = count(); }'
VFS
Using dwatch:
# Trace VFS lookup events by path:
dwatch -X vop_lookup
# Trace VFS lookup (VOP_LOOKUP(9)) paths containing /lib/:
dwatch -X vop_lookup -r /lib/
# Trace VFS create events by path:
dwatch -X vop_create
# Trace VFS remove events by path:
dwatch -X vop_remove
# Trace VFS symlink events by link and path:
dwatch -X vop_symlink
# Trace VFS readdir events by path:
dwatch -X vop_readdir
# Trace VFS mkdir events by path:
dwatch -X vop_mkdir
# Trace VFS rmdir events by path:
dwatch -X vop_rmdir
# Trace VFS rename events by path and destination:
dwatch -X vop_rename
# Trace VFS mknod events by path:
dwatch -X vop_mknod
Raw Kernel Tracing
# Count kernel slab memory allocation by function:
dtrace -n 'fbt::kmem*:entry { @[probefunc] = count(); }'
# Count kernel slab memory allocation by calling function:
dtrace -n 'fbt::kmem*:entry { @[caller] = count(); } END { printa("%40a %@16d\n", @); }'
# Count kernel malloc() by calling function:
dtrace -n 'fbt::malloc:entry { @[caller] = count(); } END { printa("%40a %@16d\n", @); }'
# Count kernel malloc() by kernel stack trace:
dtrace -n 'fbt::malloc:entry { @[stack()] = count(); }'
# Summarize vmem_alloc()s by arena name and size distribution:
dtrace -n 'fbt::vmem_alloc:entry { @[args[0]->vm_name] = quantize(arg1); }'
# Summarize TCP life span in seconds:
dtrace -n 'fbt::tcp_close:entry { @["TCP life span (seconds):"] =
quantize((uint32_t)(`ticks - args[0]->t_starttime) / `hz); }'
Using dwatch:
# Trace processes performing writes with small buffers:
dwatch -t 'arg2<10' -E 'printf("%d",arg2)' write
# Similar to above but ignore dtrace executable:
dwatch -X write -t 'execname != "dtrace" && this->nbytes < 10'