<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Systems Field Log</title>
  <subtitle>Empirical notes on low-latency Linux systems, storage engines, and consensus protocols.</subtitle>
  <link href="https://blog.somethingfornothing.me/feed.xml" rel="self"/>
  <link href="https://blog.somethingfornothing.me/"/>
  <updated>2026-11-04T12:00:00Z</updated>
  <id>https://blog.somethingfornothing.me/</id>
  <author><name>Marcus Vance</name></author>
  <entry>
    <title>Zero-Copy Socket Receives and Provided Buffer Ring Sizing in Linux 6.12 io_uring</title>
    <link href="https://blog.somethingfornothing.me/posts/io-uring-network-stack.html"/>
    <id>https://blog.somethingfornothing.me/posts/io-uring-network-stack.html</id>
    <updated>2026-11-04T12:00:00Z</updated>
    <summary>Dissecting multishot IORING_OP_RECV with IOSQE_BUFFER_SELECT across 16,000 concurrent RPC connections: when shared kernel-user buffer rings beat edge-triggered epoll, and why dual-tier ring partitioning cut p999 from 4.2 ms to 1.1 ms.</summary>
  </entry>
  <entry>
    <title>Clock Drift Bounds and Linearizable Lease Reads in Multi-AZ Raft Clusters</title>
    <link href="https://blog.somethingfornothing.me/posts/raft-lease-reads.html"/>
    <id>https://blog.somethingfornothing.me/posts/raft-lease-reads.html</id>
    <updated>2026-10-19T12:00:00Z</updated>
    <summary>How monotonic clock uncertainty affects leader lease safety across availability zones, and how combining bounded local leases with chrony dispersion circuit-breakers cut p99 read latency by 87%.</summary>
  </entry>
  <entry>
    <title>FQ Pacing Granularity and BBR Bandwidth Probing on Shallow-Buffer WAN Links</title>
    <link href="https://blog.somethingfornothing.me/posts/tcp-bbr-pacing-notes.html"/>
    <id>https://blog.somethingfornothing.me/posts/tcp-bbr-pacing-notes.html</id>
    <updated>2026-09-27T12:00:00Z</updated>
    <summary>When a 10 Gbps host feeds a policed sub-rate WAN circuit, TSO/GSO micro-bursts during BBR ProbeBW cycles can overflow shallow ToR switch buffers. Using sch_fq EDT timestamps to eliminate policer drops.</summary>
  </entry>
  <entry>
    <title>Debugging P999 Write Stalls in LSM-Tree Storage Engines Under Skewed Zipfian Workloads</title>
    <link href="https://blog.somethingfornothing.me/posts/lsm-compaction-write-stalls.html"/>
    <id>https://blog.somethingfornothing.me/posts/lsm-compaction-write-stalls.html</id>
    <updated>2026-08-14T12:00:00Z</updated>
    <summary>Tracing a recurring 180 ms latency spike every 14 minutes in a 4 TB RocksDB/Pebble cluster: how L0-to-L1 compaction debt, memtable flush backpressure, and NVMe GC interacted.</summary>
  </entry>
  <entry>
    <title>Why Our eBPF Ring Buffer Dropped 4% of Socket Events at 800k eps (And How We Fixed It)</title>
    <link href="https://blog.somethingfornothing.me/posts/ebpf-ringbuf-drop-debugging.html"/>
    <id>https://blog.somethingfornothing.me/posts/ebpf-ringbuf-drop-debugging.html</id>
    <updated>2026-07-02T12:00:00Z</updated>
    <summary>Investigating silent ring buffer reservation failures in high-throughput telemetry: profiling cache-line bouncing on producer_pos and tuning adaptive BPF_RB_NO_WAKEUP watermarks.</summary>
  </entry>
  <entry>
    <title>Measuring NVMe Garbage Collection Tail Latency and Append-Only WAL Alignment</title>
    <link href="https://blog.somethingfornothing.me/posts/nvme-zns-tail-latency.html"/>
    <id>https://blog.somethingfornothing.me/posts/nvme-zns-tail-latency.html</id>
    <updated>2026-05-21T12:00:00Z</updated>
    <summary>Why pre-allocating WAL segments and aligning O_DIRECT appends to 64 KiB Indirection Units (IU) cuts p999 fsync latency by 94% on datacenter NVMe drives.</summary>
  </entry>
  <entry>
    <title>Eliminating False Sharing in Lock-Free MPMC Queues on Zen 4 Multi-CCD Topologies</title>
    <link href="https://blog.somethingfornothing.me/posts/zen4-false-sharing-mpmc.html"/>
    <id>https://blog.somethingfornothing.me/posts/zen4-false-sharing-mpmc.html</id>
    <updated>2026-04-12T12:00:00Z</updated>
    <summary>Using perf c2c HITM metrics to track down a 3.8x throughput regression in a Vyukov bounded MPMC queue across AMD EPYC Core Complex Dies (CCDs).</summary>
  </entry>
  <entry>
    <title>Transparent Huge Pages, jemalloc Arena Fragmentation, and MADV_COLLAPSE in Long-Lived Daemons</title>
    <link href="https://blog.somethingfornothing.me/posts/jemalloc-thp-madv-collapse.html"/>
    <id>https://blog.somethingfornothing.me/posts/jemalloc-thp-madv-collapse.html</id>
    <updated>2026-03-08T12:00:00Z</updated>
    <summary>Balancing dTLB hit rates against RSS bloat in multi-day storage nodes using jemalloc metadata_thp and selective MADV_COLLAPSE.</summary>
  </entry>
  <entry>
    <title>Deterministic Simulation Testing for Consensus State Machines in Rust</title>
    <link href="https://blog.somethingfornothing.me/posts/deterministic-simulation-testing-rust.html"/>
    <id>https://blog.somethingfornothing.me/posts/deterministic-simulation-testing-rust.html</id>
    <updated>2026-01-29T12:00:00Z</updated>
    <summary>Building a single-threaded seed-driven simulator for Raft + WAL storage: reproducing a 1-in-400,000 crash-recovery corruption bug in 11 seconds.</summary>
  </entry>
  <entry>
    <title>Dissecting Linux Page Cache Readahead Thrashing Under Random mmap Lookups</title>
    <link href="https://blog.somethingfornothing.me/posts/mmap-readahead-thrashing.html"/>
    <id>https://blog.somethingfornothing.me/posts/mmap-readahead-thrashing.html</id>
    <updated>2025-12-14T12:00:00Z</updated>
    <summary>How default 128 KiB kernel readahead amplified NVMe read bandwidth by 16x on a memory-mapped B-tree index, and why MADV_RANDOM + explicit MADV_WILLNEED fixed it.</summary>
  </entry>
  <entry>
    <title>XDP and AF_XDP Zero-Copy Packet Steering Across Multi-Queue ConnectX-6 NICs</title>
    <link href="https://blog.somethingfornothing.me/posts/af-xdp-zero-copy-steering.html"/>
    <id>https://blog.somethingfornothing.me/posts/af-xdp-zero-copy-steering.html</id>
    <updated>2025-11-03T12:00:00Z</updated>
    <summary>Using hardware RSS flow steering rules and XDP_REDIRECT to route custom UDP telemetry traffic directly into userspace UMEM rings at 19.4 Mpps per core.</summary>
  </entry>
  <entry>
    <title>SIMD-Accelerated Varint and Delta Decoding with AVX-512 in Columnar Blocks</title>
    <link href="https://blog.somethingfornothing.me/posts/avx512-varint-delta-decoding.html"/>
    <id>https://blog.somethingfornothing.me/posts/avx512-varint-delta-decoding.html</id>
    <updated>2025-09-22T12:00:00Z</updated>
    <summary>Replacing branch-heavy LEB128 varints with StreamVByte + AVX-512 prefix-sum delta decoding to scan 6.8 billion integers per second per core.</summary>
  </entry>
  <entry>
    <title>Understanding ext4 JBD2 Journal Commit Bottlenecks Under Concurrent fdatasync</title>
    <link href="https://blog.somethingfornothing.me/posts/ext4-jbd2-fdatasync-bottlenecks.html"/>
    <id>https://blog.somethingfornothing.me/posts/ext4-jbd2-fdatasync-bottlenecks.html</id>
    <updated>2025-08-11T12:00:00Z</updated>
    <summary>Why un-preallocated appends serialize behind the global jbd2/vda1-8 journal transaction lock, and when XFS or fallocated ext4 nojournal_checksum wins.</summary>
  </entry>
  <entry>
    <title>Anatomy of a Go Runtime Scheduler Latency Spike: Netpoller, Syscall P Hand-offs, and GOMAXPROCS</title>
    <link href="https://blog.somethingfornothing.me/posts/go-scheduler-netpoller-latency.html"/>
    <id>https://blog.somethingfornothing.me/posts/go-scheduler-netpoller-latency.html</id>
    <updated>2025-06-28T12:00:00Z</updated>
    <summary>Using go tool trace and runtime/metrics (/sched/latencies:seconds) to diagnose a 15 ms tail latency anomaly in a containerized Go proxy.</summary>
  </entry>
  <entry>
    <title>Bloom Filter False Positives vs. Ribbon Filters in Memory-Constrained SSTable Caches</title>
    <link href="https://blog.somethingfornothing.me/posts/ribbon-filters-sstable-cache.html"/>
    <id>https://blog.somethingfornothing.me/posts/ribbon-filters-sstable-cache.html</id>
    <updated>2025-05-15T12:00:00Z</updated>
    <summary>Saving 31% of filter RAM in a 12-billion-key LSM store by adopting cache-local Ribbon filters for L2–L6 SSTables while keeping FastLocalBloom in L0/L1.</summary>
  </entry>
  <entry>
    <title>Tracing Ephemeral Port Exhaustion and TIME_WAIT Reuse Anomalies Behind L4 Load Balancers</title>
    <link href="https://blog.somethingfornothing.me/posts/ephemeral-port-time-wait-reuse.html"/>
    <id>https://blog.somethingfornothing.me/posts/ephemeral-port-time-wait-reuse.html</id>
    <updated>2025-04-02T12:00:00Z</updated>
    <summary>Why connect(2) returned EADDRNOTAVAIL despite tcp_tw_reuse=1, and how IP_BIND_ADDRESS_NO_PORT + multi-IP source binding scaled outbound pools to 500k conns.</summary>
  </entry>
  <entry>
    <title>Wait-Free Epoch-Based Reclamation Without Tail Latency Spikes From Stalled Readers</title>
    <link href="https://blog.somethingfornothing.me/posts/epoch-reclamation-stalled-readers.html"/>
    <id>https://blog.somethingfornothing.me/posts/epoch-reclamation-stalled-readers.html</id>
    <updated>2025-02-18T12:00:00Z</updated>
    <summary>Combining QSBR / Crossbeam-style epoch pinning with Hazard Eras to prevent a single preempted reader thread from halting memory reclamation.</summary>
  </entry>
  <entry>
    <title>Measuring PCIe Peer-to-Peer DMA Throughput Between NVMe Drives and RDMA NICs</title>
    <link href="https://blog.somethingfornothing.me/posts/pcie-p2p-dma-nvme-rdma.html"/>
    <id>https://blog.somethingfornothing.me/posts/pcie-p2p-dma-nvme-rdma.html</id>
    <updated>2025-01-09T12:00:00Z</updated>
    <summary>Evaluating Linux pci_p2pdma and CMB (Controller Memory Buffer) transfers across PCIe root complexes vs. dedicated PCIe switches.</summary>
  </entry>
  <entry>
    <title>B-Epsilon Trees vs. Write-Optimized B+Trees for High-Ingestion Secondary Indexes</title>
    <link href="https://blog.somethingfornothing.me/posts/bepsilon-vs-bplustree-indexes.html"/>
    <id>https://blog.somethingfornothing.me/posts/bepsilon-vs-bplustree-indexes.html</id>
    <updated>2024-11-24T12:00:00Z</updated>
    <summary>Benchmarking internal node message buffers (Bε-trees) against WiredTiger-style B+Trees and RocksDB LSM trees on 50/50 read-write secondary index workloads.</summary>
  </entry>
  <entry>
    <title>Profiling Hardware TLB Misses with perf stat and 2MiB Hugetlbfs Backing Pools</title>
    <link href="https://blog.somethingfornothing.me/posts/tlb-misses-hugetlbfs-pools.html"/>
    <id>https://blog.somethingfornothing.me/posts/tlb-misses-hugetlbfs-pools.html</id>
    <updated>2024-10-12T12:00:00Z</updated>
    <summary>Measuring page-table walk cycles on a 48 GiB in-memory hash table using perf dTLB-load-misses and MAP_HUGETLB.</summary>
  </entry>
  <entry>
    <title>Adaptive Concurrency Limits Using Little's Law and Gradient2 Queue Controllers</title>
    <link href="https://blog.somethingfornothing.me/posts/gradient2-adaptive-concurrency-limits.html"/>
    <id>https://blog.somethingfornothing.me/posts/gradient2-adaptive-concurrency-limits.html</id>
    <updated>2024-08-30T12:00:00Z</updated>
    <summary>Preventing retry storms and queue saturation in storage RPC gateways without hardcoded max_inflight magic numbers.</summary>
  </entry>
  <entry>
    <title>Inside Linux cgroup v2 memory.high Throttling and Direct Reclaim Backpressure</title>
    <link href="https://blog.somethingfornothing.me/posts/cgroupv2-memory-high-reclaim.html"/>
    <id>https://blog.somethingfornothing.me/posts/cgroupv2-memory-high-reclaim.html</id>
    <updated>2024-07-14T12:00:00Z</updated>
    <summary>How exceeding memory.high injects proportional schedule_timeout delays into allocating threads before memory.max triggers the OOM killer.</summary>
  </entry>
  <entry>
    <title>CRC32C vs. XXH3 vs. BLAKE3 for Storage Block Checksumming on x86_64 and ARM64</title>
    <link href="https://blog.somethingfornothing.me/posts/storage-block-checksum-benchmarks.html"/>
    <id>https://blog.somethingfornothing.me/posts/storage-block-checksum-benchmarks.html</id>
    <updated>2024-05-26T12:00:00Z</updated>
    <summary>Benchmarking 4 KiB and 64 KiB SSTable block integrity verification speed and bit-error detection properties.</summary>
  </entry>
  <entry>
    <title>Post-Mortem: How a Single Slow Follower Stalled a Quorum Pipeline via TCP Receive Window Zeroing</title>
    <link href="https://blog.somethingfornothing.me/posts/raft-slow-follower-tcp-rwnd.html"/>
    <id>https://blog.somethingfornothing.me/posts/raft-slow-follower-tcp-rwnd.html</id>
    <updated>2024-04-08T12:00:00Z</updated>
    <summary>When a degraded disk on 1 of 3 Raft nodes shouldn't affect quorum commits—but shared gRPC stream head-of-line blocking coupled the leader to the slowest peer.</summary>
  </entry>
</feed>
