Modern Linux Observability (Part 2): Peering into Kernel Memory, KASLR & Hypervisor Cgroup Attribution

In Part 1, we explored how eBPF creates a safe sandbox inside the Linux kernel and how BTF-enabled raw tracepoints (tp_btf) allow us to intercept block I/O requests as they are dispatched.

Now comes the harder, fascinating challenge: How do we actually look inside kernel memory without causing a system crash?

If your goal is to analyze data patterns (like calculating byte distribution or entropy) or determine which specific Virtual Machine or Docker container performed an operation, you immediately run into two major operating system defense mechanisms:

1. Virtual Memory Indirection & Page Frames
2. Kernel Address Space Layout Randomization (KASLR)

---

1. OS Refresher: Virtual Memory, Pages & The PFN Illusion

OS Theory Refresher

Virtual Addresses vs Physical RAM

When an application accesses a pointer like 0x7ffee4b2a180, that number is not a real physical wire address on your computer's RAM stick. Modern operating systems use Paging: physical RAM is chopped into fixed-size chunks called Page Frames (typically 4 KiB on x86-64). The CPU's Memory Management Unit (MMU) maintains Page Tables that map virtual addresses to physical pages. In the Linux kernel, every physical 4 KiB page frame in your system is represented by a tracking structure called struct page.

Real-World Analogy

The Hotel Room Key vs The Housekeeping Ledger

When you stay at a large hotel, your room key says "Room 304" (the Virtual Address). You don't need to know the architectural blueprint of the building or which electrical conduit powers your room. However, the hotel housekeeping manager holds an internal ledger (the Page Table). In their ledger, "Room 304" is mapped to "Concrete Floor 3, North Wing, Bay #12" (the Physical Page Frame). If the hotel reorganizes rooms, your room number might point to a different bay tomorrow without you ever noticing.

Figure 1: Virtual address translation to Physical Frame Numbers (PFN) and KASLR randomization.

---

2. The KASLR Mystery: Why Fixed Kernel Addresses Break

OS Theory Refresher

What is KASLR and Why Does It Exist?

Years ago, the Linux kernel always loaded at the exact same virtual memory address on every boot (e.g., 0xffffffff81000000). If a malicious attacker discovered a buffer overflow in any driver, they knew exactly where functions like commit_creds() lived in memory, making exploits easy to execute. To stop this, modern kernels employ KASLR (Kernel Address Space Layout Randomization). Every time you boot your machine, the kernel generates random offsets and relocates all memory segments!

This is great for security, but what happens when our eBPF probe needs to read data from a struct page * in the kernel?

In the Linux block I/O layer, data waiting to be written to disk is not stored in a simple contiguous buffer. Instead, it is organized as a hierarchy:

Figure 2: The Linux kernel Block I/O structure chain: Request ➔ Bio ➔ Bio Vector ➔ Physical Memory Page.

Kernel Pitfall

A struct page * is NOT a Memory Buffer!

A common rookie mistake when reading eBPF tutorials is attempting to run bpf_probe_read_kernel(dst, 64, segment.bv_page). This will read garbage or fail! The bv_page pointer points to the kernel's internal tracking descriptor, not the actual bytes of data!

Translating Page Structures into Virtual Addresses in eBPF

To find the actual bytes, the probe must dynamically calculate the Page Frame Number (PFN) and reconstruct the direct-map virtual address using two kernel symbols exported at runtime:

/* Declare symbols that KASLR randomizes at boot */
extern const void vmemmap_base __ksym;
extern const void page_offset_base __ksym;

/* Inside the eBPF probe */
unsigned long vmemmap = 0;
unsigned long page_offset = 0;

/* Read the live randomized bases */
bpf_probe_read_kernel(&vmemmap, sizeof(vmemmap), &vmemmap_base);
bpf_probe_read_kernel(&page_offset, sizeof(page_offset), &page_offset_base);

/* Calculate the Physical Page Frame Number (PFN) */
unsigned long pfn = ((unsigned long)segment.bv_page - vmemmap) / bpf_core_type_size(struct page);

/* Reconstruct the direct-map kernel virtual address */
void* vaddr = (void*)(page_offset + (pfn << 12) + segment.bv_offset);

/* Now we can safely sample the payload! */
bpf_probe_read_kernel(dst_buffer, sample_len, vaddr);

---

3. The Attribution Dilemma: Why Process IDs (PIDs) Lie

OS Theory Refresher

The Flaw of Process-Level Tracking in Cloud & Virtualization

When observing a hypervisor like Proxmox VE (or a Kubernetes node), measuring disk I/O by PID (Process ID) is notoriously unreliable:

• Thread Sprawl: A single QEMU VM runs dozens of threads (vCPUs, IOThreads, asynchronous disk workers, vhost worker threads). Each thread has a distinct thread ID!
• Delegated I/O: If a VM uses modern Linux io_uring or kernel worker threads (kworker), the actual I/O request is issued by a kernel daemon thread whose PID has nothing to do with the VM!
• PID Wrapping: PIDs are recycled by the operating system, causing metric attribution drift.

The Solution: Hierarchical Cgroups v2

In modern Linux installations, systemd organizes everything into a strict hierarchical tree using Control Groups version 2 (cgroups v2).

In Proxmox VE, every single virtual machine is permanently assigned a dedicated slice in the cgroup filesystem:

/sys/fs/cgroup/qemu.slice/101.scope/
/sys/fs/cgroup/qemu.slice/102.scope/

No matter how many threads QEMU spawns, and regardless of whether the I/O is submitted via an emulator thread or an async worker, all of them belong to the same cgroup scope!

Figure 3: Systemd Cgroup v2 hierarchy showing stable VM scope isolation at Level 2.

Resolving the VMID Inside eBPF

Instead of tracking hundreds of fleeting thread IDs, our probe queries the kernel for the task's ancestor cgroup at level 2:

/* Fast in-kernel VMID lookup */
static __always_inline __u32 vmid_of_current_task(void) {
    /* Level 2 corresponds directly to /qemu.slice/.scope */
    __u64 cgroup_id = bpf_get_current_ancestor_cgroup_id(2);
    
    /* Query the BPF Hash Map populated by userspace */
    __u32* vmid = bpf_map_lookup_elem(&cgroup_to_vmid_map, &cgroup_id);
    
    /* Return the VMID (e.g., 100, 101), or 0 if it's host background traffic */
    return vmid != NULL ? *vmid : 0;
}

In user space, our daemon simply watches the cgroup directory using inotify. Whenever a VM starts or shuts down, the user-space daemon reads its cgroup inode number and inserts it into cgroup_to_vmid_map. The kernel probe does a sub-microsecond O(1) hash lookup, instantly tagging every disk I/O event with its true owner!

---

4. Key Takeaways & What's Next

• Pointers Are Illusions: In virtual memory systems, pointers represent virtual addresses mapped through page tables to 4 KiB physical page frames.
• KASLR Requires Dynamic Resolution: Memory bases shift on every reboot. To sample data inside eBPF, we compute the Page Frame Number (PFN) relative to live kernel symbols (vmemmap_base, page_offset_base).
• Cgroups Are the True Identity in Virtualization: Process IDs are fragile and fragmented across threads. Cgroups v2 provide an unforgeable, hierarchical anchor for attributing system telemetry.

Now that our eBPF probe is extracting clean, properly-attributed block I/O events, thousands of events per second are flowing into our ring buffer.

In Part 3, we will leave the kernel and enter the world of Modern C++20: How do we build an ultra-fast user-space ingestion engine that processes 50,000+ events per second without taking locks, without crashing, and with zero dynamic heap allocations?

Questions about paging, KASLR or cgroups? Leave a comment below, and join us for Part 3!