bpftrace provides a high-level abstraction for eBPF-based dynamic tracing, enabling real-time analysis of kernel and user-space behavior without instrumentation overhead or module compilation. Built atop the BPF virtual machine, it offers safe, efficient execution while maintaining an accessible scripting interface.

Programs adopt an awk-inspired syntax consisting of probes, optional predicates, and action blocks:

probe [/filter/] { statements }

Probes attach to instrumentation points including kprobes (dynamic kernel functions), tracepoints (static kernel markers), uprobes (user-space symbols), and USDT (user statically defined tracing). The special BEGIN probe initializes state before event collection, while END performs cleanup and final output.

Predicates evaluate Boolean expressions to conditionally execute actions. For example, /pid == 1234/ restricts processing to a specific process. Action blocks contain the instrumentation logic—variable assignments, map updates, or formatted output.

Variable scoping distinguishes three types. Built-in variables provide immediate execution context: pid and tid identify the process and thread; comm contains the process name; cpu indicates the executing processor; nsecs delivers high-resolution timestamps; kstack captures kernel stack traces; curtask references the current task structure. Argument access varies by probe type: args dereferences tracepoint structures, while arg0 through arg9 access raw kprobe registers. In return probes (kretprobe, uretprobe), retval contains the function result.

User-defined scalar variables, prefixed with $, store intermediate calculations: $ts = nsecs;.

Associative arrays (maps), denoted by @, enable kernel-to-user data transfer and statistical aggregation. These support composite keys and various aggregators:

• @name = count() - event frequency
• @name = sum(value) - arithmetic total
• @name = hist(value) - power-of-2 histogram
• @name = lhist(value, min, max, step) - linear histogram
• @name = min(value), @name = max(value) - range boundaries
• @name = stats(value) - count, average, and total

Built-in functions extend functionality: printf() produces formatted output; str() and buf() safely copy user/kernel memory; kstack() and ustack() generate call stacks; ksym() and usym() resolve addresses to symbols; delete() and clear() manage map lifecycle.

Interactive execution uses one-liners:

bpftrace -e 'kprobe:do_sys_open { printf("%s: %s\n", comm, str(arg1)); }'

Complex analysis requires script files:

#!/usr/bin/bpftrace

BEGIN {
    printf("Tracing network connections for PID %d\n", $1);
}

kprobe:tcp_v4_connect,
kprobe:tcp_v6_connect
/pid == $1/ {
    @conn_attempts[comm] = count();
    @start[tid] = nsecs;
}

kretprobe:tcp_v4_connect,
kretprobe:tcp_v6_connect
/pid == $1 && @start[tid]/ {
    $latency = nsecs - @start[tid];
    @latency_dist = hist($latency);
    delete(@start[tid]);
}

END {
    printf("\nConnection attempts by process:\n");
    print(@conn_attempts);
    printf("\nLatency distribution (ns):\n");
    print(@latency_dist);
}

This instrumentation tracks TCP connection attempts and latencies for a target process, utilizing both counting and histogram aggregations while maintaining per-thread state for duration calculations.