Modern Linux Observability (Part 1): eBPF Fundamentals, BTF Raw Tracepoints & The Ring Buffer Revolution

If you are studying software architecture, systems analysis, or cloud infrastructure, you have almost certainly heard the term eBPF (Extended Berkeley Packet Filter) hailed as the biggest revolution in the Linux kernel in the past two decades. Prominent tech organizations use it for high-performance Kubernetes networking (Cilium), dynamic security tracing (Falco), and system-wide observability (Brendan Gregg's performance tools).

Yet, when developers first dive into kernel internals, the learning curve can feel brutal: verifier rejections, kernel page faults, KASLR, and obscure memory barriers.

In this 3-part series, we will demystify the core engineering principles behind building low-overhead, production-grade Linux telemetry systems. We will start from fundamental Operating System concepts that every software developer should know, and progressively build up to high-throughput, real-time observability.

---

1. OS Refresher: The Sacred Boundary Between User and Kernel Space

OS Theory Refresher

CPU Protection Rings: Ring 3 vs Ring 0

Modern x86-64 and ARM processors enforce hardware privilege levels known as Protection Rings. Your standard programs (Java Virtual Machine, Python, Node.js, your web browser) run in User Space (Ring 3). They are strictly sandboxed: they cannot directly talk to the physical network card, read physical disk sectors, or access RAM belonging to another application. The operating system core runs in Kernel Space (Ring 0), which has direct, unrestricted access to the underlying hardware and physical memory.

Real-World Analogy

The Restaurant Customer vs The Kitchen Staff

Think of User Space as the dining room of a restaurant and Kernel Space as the commercial kitchen. As a customer (User Process), you cannot just barge into the kitchen, grab the meat cleaver, and turn on the commercial stove (hardware access). Doing so would risk fires, food contamination, and chaos. Instead, you call the waiter and hand over an order ticket (a System Call or syscall). The kitchen staff (the Kernel) verifies your order, cooks the meal safely, and hands back the result.

Whenever a user process needs to write a file to disk or receive a packet from a network socket, it must perform a System Call (like read(), write(), or epoll_wait()). This triggers a CPU context switch from Ring 3 to Ring 0, which introduces small, but measurable, latency.

Figure 1: Hardware protection rings and the position of the in-kernel eBPF execution engine.

The Old Observability Dilemma: Polling /proc vs Kernel Modules

Historically, if a software engineer wanted to observe how much disk I/O each virtual machine or process was executing, they had two flawed options:

1. Polling /proc or /sys files: User applications periodically read text pseudo-files like /proc/diskstats every few seconds. This is simple, but it only shows coarse aggregates. Transient I/O spikes occurring between scrapes are completely missed.
2. Writing a Linux Kernel Module (LKM): Developers wrote custom C code loaded directly into the kernel. While blazingly fast, a single null-pointer dereference or infinite loop in a kernel module causes a Kernel Panic (the Linux equivalent of the Blue Screen of Death), taking down the entire physical machine.

---

2. Enter eBPF: Programmable, Sandboxed Kernel Execution

eBPF allows developers to write custom C programs that compile down to a specialized bytecode. When loaded into the kernel, this bytecode is not blindly executed. Instead, it is subjected to an exhaustive mathematical safety check by the eBPF Verifier.

OS Theory Refresher

Static Program Verification & The Halting Problem

In theoretical computer science, Turing proved that an algorithm cannot determine whether an arbitrary program will ever halt (finish running). To guarantee that an eBPF program will never freeze or crash the operating system, the eBPF Verifier enforces strict constraints:

• Every pointer dereference must be proven to point to valid memory.
• No unbounded or infinite loops are allowed; all loops must have statically verifiable finite bounds.
• Programs cannot read uninitialized registers or stack variables.
• Memory access bounds must be explicitly checked before slicing data.

Once the verifier approves the bytecode, an in-kernel Just-In-Time (JIT) compiler translates it into native machine code (x86-64 or ARM64) that runs at hardware wire speed!

Figure 2: The eBPF build and validation lifecycle from C code to verified in-kernel execution.

---

3. Choosing the Right Hook: The Power of tp_btf (BTF Raw Tracepoints)

To trace disk I/O, the kernel provides several hook mechanisms. Knowing which one to pick is critical for performance and stability:

Hook Mechanism	How It Works	Overhead	Data Access
Kprobe (kprobe/blk_mq_start_request)	Dynamically patches instructions at any kernel function entry point.	Moderate to High (breakpoint trap). Fragile across kernel refactors.	Full access to function parameters.
Classic Tracepoint (tracepoint/block/block_rq_issue)	Static hooks instrumented into the kernel source code.	Low. Highly stable ABI.	Only provides formatted scalar copies (sector, size, PID). Cannot inspect payload memory!
BTF Raw Tracepoint (tp_btf/block_rq_issue)	Direct pointer invocation using in-kernel BPF Type Format (BTF) metadata.	Ultra-Low (direct function call, no trap, zero argument copying).	Direct typed pointer to kernel structs (struct request *, struct bio *).

System Architecture

Why tp_btf Wins for Low-Latency Telemetry

When an I/O request is issued to a disk device, we need to inspect the operation (Read or Write), the target block sector, and the payload memory without slowing down the disk controller. With tp_btf, the kernel provides typed debug symbols directly via BTF. The probe receives the actual struct request *rq pointer directly in register r1!

Inspecting the Probe Implementation

Here is how an ultra-efficient eBPF probe is structured using tp_btf and libbpf:

#include "vmlinux.h"
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include "subos_monitor_event.h"

char LICENSE[] SEC("license") = "GPL";

/* Hook directly into the block layer request issue point */
SEC("tp_btf/block_rq_issue")
int BPF_PROG(handle_block_rq_issue, struct request* rq) {
    /* Read operation flags safely using CO-RE helper */
    unsigned int op = BPF_CORE_READ(rq, cmd_flags) & 0xFF;
    
    /* Discard flushes, discards, and zone maintenance */
    if (op != REQ_OP_READ && op != REQ_OP_WRITE) {
        return 0;
    }

    /* Reserve space directly in the lockless ring buffer */
    struct block_io_event* event = bpf_ringbuf_reserve(&events_ringbuf, sizeof(*event), 0);
    if (!event) {
        /* Buffer is saturated; increment a per-CPU counter and drop event safely */
        return 0;
    }

    /* Populate event telemetry */
    event->timestamp_ns = bpf_ktime_get_ns();
    event->sector       = BPF_CORE_READ(rq, __sector);
    event->nr_sectors   = BPF_CORE_READ(rq, __data_len) >> 9; // 512-byte blocks
    event->direction    = (op == REQ_OP_WRITE) ? 1 : 0;

    /* Atomically commit event to user-space */
    bpf_ringbuf_submit(event, 0);
    return 0;
}

---

4. The Data Transfer Highway: Perf Buffer vs BPF Ring Buffer

Once the kernel captures an event in Ring 0, how does it send it to our user-space daemon in Ring 3 without stalling the operating system?

OS Theory Refresher

The Multi-Core Memory Problem

Modern servers have dozens of CPU cores (e.g., 32 cores, 64 threads). If all 64 cores simultaneously try to write events into a single queue using traditional locks (mutexes), the CPU spends more time fighting over cache lines and lock ownership (Lock Contention) than doing useful work!

To tackle this, the Linux kernel went through two major generations of communication buffers:

Figure 3: Architectural comparison between Per-CPU Perf Buffers and modern BPF Ring Buffers.

OS Theory Refresher

The Multi-Producer Single-Consumer (MPSC) Pattern

The modern BPF_MAP_TYPE_RINGBUF implements an MPSC queue using lock-free hardware atomics. Multiple CPU cores can simultaneously reserve slices of memory in the buffer. Furthermore, it supports memory-mapped I/O (mmap): the user-space daemon maps the ring buffer directly into its own virtual memory space. When the daemon reads an event, no extra memory copies take place between the kernel and the application!

---

5. Key Takeaways & What's Ahead

Understanding eBPF starts with appreciating the boundary between User Space and Kernel Space:

• Safety without Slowness: The eBPF Verifier mathematically guarantees that custom kernel probes cannot crash or freeze your servers, while the JIT compiler ensures near-zero overhead.
• Precision Instrumentation: BTF-powered raw tracepoints (tp_btf) give typed access to kernel structures (like struct request) right when events occur in the block layer.
• Zero-Copy Data Planes: Modern BPF Ring Buffers replace fragmented per-CPU queues with an efficient, memory-mapped multi-producer pipeline.

In Part 2, we will go deeper into the kernel's memory engine: How does Linux manage memory pages? How does Kernel Address Space Layout Randomization (KASLR) protect against exploits? And how can an eBPF probe accurately identify which Proxmox Virtual Machine or Docker container generated an I/O request using Cgroups v2?

Stay tuned for Part 2 next week! Have questions about eBPF or OS concepts? Drop a comment below!