CRC32C vs. XXH3 vs. BLAKE3 for Storage Block Checksumming on x86_64 and ARM64

Benchmarking 4 KiB and 64 KiB SSTable block integrity verification speed and bit-error detection properties.

1. Throughput Across 4 KiB and 64 KiB Storage Blocks

Every SSTable block read and WAL frame recovery verifies a checksum to detect silent data corruption (bit rot, torn writes, or misdirected DMA). We compared hardware-accelerated CRC32C (`crc32q` + `pclmulqdq` folding), `XXH3-64` (AVX2/AVX-512), and `BLAKE3` (cryptographic tree hash):

Algorithm4 KiB Block Speed64 KiB Block SpeedOutput WidthCryptographic
CRC32C (SSE4.2 + PCLMULQDQ)19.2 GB/s24.8 GB/s32-bitNo
XXH3-64 (AVX2)26.1 GB/s31.4 GB/s64-bitNo
XXH3-128 (AVX2)24.0 GB/s29.8 GB/s128-bitNo
BLAKE3 (AVX-512)4.1 GB/s6.2 GB/s256-bitYes

For internal storage blocks, `XXH3-64` offers both faster verification than folded CRC32C and a \(2^{32} imes\) lower random collision probability on corrupted multi-megabyte blobs, while `BLAKE3` is ideal for content-addressed snapshot manifests.

← Older Note (2024-04-08)Post-Mortem: How a Single Slow Follower Stalled a Quorum Pipeline via TCP Receive Window ZeroingNewer Note (2024-07-14) →Inside Linux cgroup v2 memory.high Throttling and Direct Reclaim Backpressure

← Back to all notes · View full chronological archive (24)