Printk Logging

View current logging levels via /proc/sys/kernel/printk:

$ cat /proc/sys/kernel/printk
1       4       1       3

The four values represent:

• console_loglevel: Minimum priority for console output.
• default_message_loglevel: Default priority when unspecified in printk().
• minimum_console_loglevel: Minimum allowed value for console_loglevel.
• default_console_loglevel: Console level at system boot.

Adjust logging levels:

# echo 1 4 1 7 > /proc/sys/kernel/printk
# cat /proc/sys/kernel/printk
1       4       1       7

If direct modification fails, use dmesg:

$ sudo dmesg -n debug
$ cat /proc/sys/kernel/printk
8       4       1       3

Dynamic Debugging

While printk is useful, it requires kernel recompilation for changes. Dynamic debugging enables runtime control. Enable these kernel options:

CONFIG_DYNAMIC_DEBUG=y
CONFIG_DYNAMIC_DEBUG_CORE=y

Control files reside at /sys/kernel/debug/dynamic_debug/control. Enable or disable prints by file, module, or function.

Oops Analysis

Triggering Oops

Common triggers include:

• BUG() and BUG_ON()
• WARN(x) and WARN_ON(x)
• panic()
• dump_stack()

Stack Trace Interpretation

Without KALLSYMS

If CONFIG_KALLSYMS is disabled, use ksymoops with System.map to decode stack traces.

With KALLSYMS

When CONFIG_KALLSYMS=y, stack traces display decoded function names directly.

SysRq Magic Keys

Enable in kernel configuration:

CONFIG_MAGIC_SYSRQ=y

Usage example to show task states:

echo 1 > /proc/sys/kernel/sysrq
echo t > /proc/sysrq-trigger

If output isn't visible on console, check dmesg. Adjust console logging level if needed.

On physical consoles, use SysRq key combinations (e.g., SysRq+Alt+command). Serial consoles may require BREAK+command sequences, potentially related to CONFIG_KGDB_SERIAL_CONSOLE=y. Non-console environments must use command trigggers.

Kernel Filesystems for Debugging

Procfs

The /proc filesystem exposes kernel and process information. Key directories:

• Process-specific subdirectories (/proc/[pid]/)
• Kernel data files

Process Information

Common files in /proc/[pid]/:

File	Description
cmdline	Command-line arguments
environ	Environment variables (use `cat /proc/pid/environ
fd	File descriptor information
maps	Memory mappings of executable and library files
mem	Process memory
stat	Process status (fields may vary)
statm	Memory status information
status	Human-readable process status
wchan	Kernel function where task is blocked (requires CONFIG_KALLSYMS=y)
pagemap	Page table information
stack	Complete call stack (requires CONFIG_STACKTRACE)
smaps	Extended memory mapping details
smaps_rollup	Summary of smaps information
numa_maps	NUMA memory allocation details
syscall	Currently executing system call
latency	Latency information (enable with echo 1 > /proc/sys/kernel/latencytop)
limits	Resource limits
schedstat	Scheduling statistics
oom_score	OOM killer score
oom_adj	OOM adjustment value
oom_score_adj	OOM score adjustment
coredump_filter	Core dump filter settings
io	I/O statistics
sched	Scheduling information

Note: maps output includes thread stack information corresponding to /proc/[pid]/task/[tid]/ paths.

Key Status Fields

Important fields from /proc/[pid]/status:

Field	Description
Pid	Process ID
PPid	Parent process ID
Threads	Number of threads
TracerPid	PID of process tracing this one
Cpus_allowed_list	CPUs where process may run
voluntary_ctxt_switches	Voluntary context switches
nonvoluntary_ctxt_switches	Involuntary context switches
VmRSS	Resident set size (physical memory used)
RssAnon	Anonymous memory (heap/stack/malloc/COW)
RssFile	File-backed memory pages
RssShmem	Shared memory
SigQ	Signal queue status
SigPnd	Pending signals
SigBlk	Blocked signals
SigIgn	Ignored signals
SigCgt	Caught signals

Signal fields use bitmasks where each bit represents a signal.

Stat Field Examples

Extract specific fields from /proc/[pid]/stat:

# Signal-related information
$ cat /proc/1/stat | awk '{print "pending:",$31,"\nblocked:",$32,"\nsigign:",$33,"\nsigcatch:",$34}'

# Priority information
$ cat /proc/1/stat | awk '{print "pri:",$18,"\nnice:",$19,"\nrtpri:",$40,"\npolicy:",$41}'

Scheduling Statistics

Enable CONFIG_SCHED_DEBUG for detailed scheduling information in /proc/[pid]/sched. Example output includes:

• se.exec_start: Last scheduling timestamp
• se.vruntime: Virtual runtime for CFS scheduler
• se.sum_exec_runtime: Total physical runtime
• nr_migrations: CPU migration count
• Various wait and sleep statistics

I/O Statistics

Example /proc/[pid]/io output:

rchar: 323934931           # Total bytes read (parameter sum, not actual disk reads)
wchar: 323929600           # Total bytes written
syscr: 632687              # read()/pread() call count
syscw: 632675              # write()/pwrite() call count
read_bytes: 0              # Actual disk bytes read
write_bytes: 323932160     # Actual disk bytes written
cancelled_write_bytes: 0   # Write bytes avoided due to truncation

Kernel Data Files

Key files in /proc for kernel information:

File	Description
cmdline	Kernel boot parameters
cpuinfo	CPU information
interrupts	Interrupt usage
iomem	Memory map
ioports	I/O port usage
kmsg	Kernel messages
loadavg	System load averages
meminfo	Memory information
modules	Loaded modules
mounts	Mounted filesystems
stat	System statistics
version	Kernel version

Network Information

Key files in /proc/net/ for IPv4:

File	Description
dev	Network device statistics
snmp	SNMP statistics for IP, ICMP, TCP, UDP
arp	ARP table
route	Routing table
tcp	TCP socket information
udp	UDP socket information
unix	Unix domain sockets

Sysfs

Sysfs exports kernel objects to userspace, typically mounted at /sys. It provides hierarchical representation of devices, drivers, and kernel subsystems.

Debugfs

Debugfs offers a simple way for kernel developers to export debugging information. Mount with:

mount -t debugfs none /sys/kernel/debug

Relayfs

Relayfs provides efficient data transfer from kernel to userspace through user-defined relay channels, useful for large data streams.

Dynamic Tracing Mechanisms

Kprobes

Kprobes allow dynamic insertion of breakpoints into kernel functions. Variants include:

• kprobe: Break on function entry
• kretprobe: Break on function return
• uprobe: User-space function breakpoints
• uretprobe: User-space function return breakpoints

These enable debugging and performance analysis without kernel modification.

Static Tracing

Ftrace

Ftrace provides static tracing through tracepoints declared in kernel code. Key features:

• Function tracing
• Function graph tracing
• Event tracing

Enable function graph tracing:

echo function_graph > /sys/kernel/debug/tracing/current_tracer

Note: Clear set_ftrace_filter before switching from function to function_graph tracer.

Kernel Debuggers

Configuration

Enable debugging options in kernel configuration:

CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y
CONFIG_DEBUG_INFO=y
CONFIG_GDB_SCRIPTS=y

Generate GDB scripts:

make scripts_gdb

KDB Usage

Enter KDB via SysRq:

echo g > /proc/sysrq-trigger

Common KDB commands:

• m$: Memory operations
• go: Continue execution
• r$: Register operations
• b$: Stack operations
• ps: List tasks
• cpu: Switch CPU
• ss: Single step

KGDB/GDB Integration

Configure serial console for KGDB:

echo ttyAMA0 > /sys/module/kgdboc/parameters/kgdboc

Enter KGDB mode:

echo g > /proc/sysrq-trigger

Connect with GDB:

gdb vmlinux
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0

For cross-architecture debugging, build GDB with correct target:

./configure --target=aarch64-linux-gnu

Switching Between Debuggers

• $3#33: Switch to KDB
• $D#44+: Continue execution, exit KGDB

GDB Python Scripts

Load kernel-specific GDB helpers:

(gdb) source vmlinux-gdb.py

Common commands:

• lx-symbols: Load/reload kernel and module symbols
• lx-dmesg: Print kernel log buffer
• lx-ps: List tasks
• p $lx_current().pid: Current task PID (x86_64/arm64)

Debugging Considerations

1. Multithreading: When debugging kernels with GDB, use set detach-on-fork on to avoid suspending all threads when switching.
2. Breakpoint Issues: On ARM64, next command may jump to interrupt handlers. Apply relevant patches if needed.

Kernel Boot Parameters

Key debugging parameters:

• kgdboc: Serial console for debugging
• kgdbwait: Wait for debugger connection at boot
• kgdbcon: Forward printk messages to GDB
• nokaslr: Disable kernel address space layout randomization