*Building a crash-resilient storage engine for a custom message broker — from first principles.* --- There's a question that hits every systems engineer building high-throughput infrastructure at some point: > **Your server is processing thousands of messages per second. All of it lives in RAM. The power goes out. What survives?** I faced this exact question while developing **Highway Broker** — a custom message broker I built from scratch. The answer wasn't just "write to disk." It forced me to think carefully about *how* to write to disk without destroying the throughput that made the system worth building in the first place. This is the story of the architecture I designed, the trade-offs I made explicit, and the patterns that made it work. --- ## The Problem: RAM Is Volatile by Design RAM is fast because it's electrically charged. Data exists as long as power flows. The moment it's cut: - All in-flight messages evaporate instantly - Partially written structures are silently corrupted - Any message not yet persisted is permanently gone For a message broker, this breaks the fundamental contract: a producer acknowledged a send, a consumer is expecting delivery. Losing data mid-flight isn't just a performance issue — it's a correctness violation. The naive fix is obvious: write every message synchronously to disk before acknowledging it. But synchronous disk I/O on the hot path is a death sentence for throughput. You've traded one problem for another. The real challenge is: **how do you get durability *and* performance?** --- ## The Architecture: A Two-Stage Write Pipeline The answer I landed on is a staged write pipeline that completely decouples the producer's hot path from disk I/O. Here's the core design, straight from `storage_manager.hpp`: ```cpp // Double-buffered write — pre-allocated, no heap allocation on hot path Buffer buffer_pool_[2]; // Pre-allocated buffers std::atomic active_buffer_; // Current write target (event loop writes here) std::atomic flush_buffer_; // Pending flush (background worker drains this) std::atomic flush_signaled_{false}; ``` Two buffers. Two roles. Never blocking each other. ### Stage 1 — The Active Buffer (Memory Speed) Every incoming message gets appended to the active buffer at memory speed. The `Buffer::append_message` implementation is lock-free on the hot path — no mutex, no syscall, no waiting: ```cpp /** * Append a message to buffer (hot path, O(1), no locks). * If buffer is full, returns false. */ bool append_message(uint64_t offset, const std::vector &payload); ``` Each message is written in a compact binary layout: ``` | uint64 offset | uint32 size | uint32 crc | payload (var) | | 8 bytes | 4 bytes | 4 bytes | size bytes | ``` The `MessageHeader` is exactly 16 bytes — not by accident. Fixed-size headers make forward scanning over the segment file trivially predictable for the CPU prefetcher. Two conditions trigger a flush: - **Buffer reaches 95% capacity** — the `is_full()` threshold avoids exact-size edge cases - **100ms timer elapses** — critical for low-traffic topics that might otherwise sit unflushed indefinitely ### Stage 2 — The Atomic Buffer Swap When flush conditions are met, the active buffer swaps atomically with the flush buffer. This is the key insight of the whole design: ```cpp /** * Acquire buffer that needs flushing. * Swaps active <-> flush atomically. * Returns pointer to buffer containing accumulated messages. */ Buffer *acquire_flush_buffer(); ``` The event loop thread continues writing to the new active buffer *immediately* after the swap. There is no lock held across disk I/O. Zero producer stall. The old data is handed off to the `FlushWorker` background thread. ### Stage 3 — The FlushWorker (Durability Without Blocking) The `FlushWorker` runs in a dedicated thread per topic. It owns the flush path entirely: ```cpp class FlushWorker { public: enum class DurabilityMode { FAST, // Flush every ~100ms (default for telemetry) BALANCED, // Flush when buffer fills SAFE // Per-message (reduces throughput significantly) }; // ... private: void worker_loop(); void flush_buffer(Buffer *buf); void write_to_segment(Buffer *buf, SegmentMetadata &segment); }; ``` The worker loop waits on a condition variable, wakes on `signal()`, drains the flush buffer to disk, then goes back to sleep. Simple. Predictable. Non-blocking to everything else. --- ## Segment Log Files: Append-Only, Offset-Indexed The `FlushWorker` writes to **segment log files** — append-only binary files where each message carries its own offset. Here's the `SegmentMetadata` that tracks each segment's lifecycle: ```cpp struct SegmentMetadata { uint64_t id; // Segment ID (base offset) uint64_t base_offset; // First offset in this segment uint64_t last_offset; // Last offset written uint64_t byte_position; // Current write position uint64_t max_size_bytes; // Rotation threshold (512MB) std::string file_path; // Full path to segment file bool is_active; // Still accepting writes? }; ``` A few constraints are fundamental to the design: - **Only writes** — no updates, no deletes, only appends - **Only reads in special cases** — replay and FETCH_ONE operations - **Rotation when full** — a new segment is created when the active one exceeds 512MB This append-only invariant is what makes everything else fast. There's no random I/O. No file fragmentation. The OS can do large sequential writes, which is the single most efficient I/O pattern on any storage medium. ### The Sparse Index: O(log N) Reads Appending is fast, but finding a specific message offset in a large segment file requires an index. I implemented a **sparse index** — one entry per 1024 messages — that maps logical offsets to file byte positions: ```cpp struct SparseIndexEntry { uint64_t offset; // Message offset (sequence number) uint64_t file_position; // Byte offset in segment file }; class SparseIndex { public: // Binary search: find file position for offset ≤ target std::optional find_file_position(uint64_t target) const; private: static constexpr uint64_t INDEX_INTERVAL = 1024; std::vector entries_; mutable std::shared_mutex mutex_; }; ``` The read path becomes: 1. Binary search on the sparse index → find approximate file position (O(log N)) 2. Sequential scan forward from that position → find exact offset (O(k), bounded to ≤64KB window) 3. Validate CRC32 → confirm integrity 4. Return payload For a segment with 1 million messages, the index has ~976 entries. Binary search finds the right window in ~10 comparisons. Then a short sequential scan to the exact message. In practice, this is fast enough that reads don't compete meaningfully with writes. --- ## Per-Topic Isolation: Independent Storage Managers Each topic in Highway Broker gets its own `SegmentLog`, its own `FlushWorker`, and its own buffer pair. The `StorageManager` acts as the registry and coordinator: ```cpp // Topic registry: topic_name -> SegmentLog std::unordered_map> logs_; mutable std::shared_mutex logs_mutex_; // Flush workers per topic std::unordered_map> flush_workers_; ``` This isolation has real operational consequences: - A slow consumer on `highway/A1/telemetry` cannot starve flushes on `highway/A2/telemetry` - Buffer tuning (size, flush interval, durability mode) is adjustable per topic - A storage failure in one topic is scoped — it cannot corrupt another topic's segments - Retention and compaction run independently per topic The `FlushWorker::DurabilityMode` can be set per topic: a critical control topic might run `SAFE` mode (flush-per-message), while a high-volume telemetry topic runs `FAST` mode (flush every 100ms). Same broker, different guarantees. --- ## Controlled Offset Access: Highway Broker v1.1 The append architecture unlocks a powerful read primitive: **direct offset access**. Because every message has a monotonically increasing offset embedded in its header, you can ask for any message by its offset and retrieve it in O(log N) time. Highway Broker v1.1 exposes this through two new access patterns: ### FETCH_ONE — Stateless Random Read ``` CLIENT BROKER | | |-- FETCH_ONE(topic, offset) -->| | | | [validate offset range] | | [sparse index lookup] | | [sequential scan] | | | |<-- FETCH_RESPONSE(data) ------| | OR OFFSET_NOT_FOUND | ``` Completely stateless. No subscription created. No state stored on the broker. A REST-like primitive on top of a streaming architecture. The `StorageManager` exposes the range validation needed: ```cpp uint64_t get_head_offset(const std::string &topic) const; // Latest written uint64_t get_oldest_offset(const std::string &topic) const; // After retention ``` Invalid offsets return `OFFSET_NOT_FOUND` with the available range — so clients always know how to recover from out-of-range requests. ### SUBSCRIBE_FROM_OFFSET — Catch-Up Then Live The second pattern handles the "I was offline, catch me up" use case: ```cpp enum class SubscriptionMode { PUSH_LIVE, // Default: deliver all new messages CATCHUP_THEN_PUSH // Replay from offset, then switch to PUSH_LIVE }; struct Subscription { Session *session; std::string pattern; QoS qos; SubscriptionMode mode; // Which delivery mode uint64_t current_offset; // Current replay position }; ``` A client subscribes with a `start_offset`. The broker delivers messages from that offset forward. When the subscriber catches up to `head_offset`, the subscription automatically transitions to `PUSH_LIVE` — no client logic required, no reconnect, no re-subscribe. The transition is logged for debugging: ``` Initial: CATCHUP_THEN_PUSH (current_offset = N) ↓ Receive messages N, N+1, N+2, ..., HEAD ↓ offset >= HEAD_OFFSET → switch to PUSH_LIVE ``` This makes Highway Broker useful for IoT telemetry scenarios where a sensor reconnects after hours offline and needs to replay everything it missed before receiving live data. --- ## The Durability Window No buffered write system provides absolute zero data loss under power failure. The architecture I built offers a **well-defined durability window**: messages in the active buffer but not yet flushed can be lost if power fails before the 100ms timer triggers. This is not a bug. It is an explicit, engineered trade-off. In practice, a 100ms window means at most ~100ms of message history is at risk under catastrophic failure. For most telemetry use cases, this is entirely acceptable. For stricter requirements, `DurabilityMode::SAFE` flushes per-message at the cost of throughput. The important thing is that the window is **explicit**, **measurable**, and **tunable** — not a hidden assumption discovered in production. --- ## Recovery on Startup When the broker restarts after a crash, `StorageManager::recover_all()` kicks in: ```cpp /** * Recover segment log from disk. * Validates CRC, truncates at corruption, rebuilds sparse index. */ void recover(); ``` The recovery process: 1. Scan the storage directory for segment files belonging to each topic 2. Read each segment, validating CRC32 on every message header 3. If corruption is detected, truncate the segment at the last valid position 4. Rebuild the sparse index from the valid portion 5. Restore `next_offset_` to continue from the last valid offset Because segments are append-only and each message has an embedded CRC, corruption is always detectable and always localized. The broker never needs to trust partially written messages — it simply discards them and resumes from the last known good state. --- ## Write Path at a Glance ``` Producer │ ▼ [ Active Buffer ] ──── (full / 100ms) ──► [ Flush Buffer ] │ │ │ (atomic swap, zero blocking) Background FlushWorker │ │ ▼ Sequential Write (continues writing) │ [ Segment Log File ] [ Sparse Index ] │ (CRC validated, offset indexed, append-only, recoverable) ``` --- ## Closing Thoughts Building this storage engine from scratch taught me something that's hard to get from reading about Kafka or Pulsar: the patterns aren't arbitrary. They're the result of hard constraints converging on elegant solutions. The double-buffer swap costs almost nothing — an atomic pointer exchange. The sequential append is the single most efficient I/O pattern available. The sparse index compresses 1M message lookups into ~10 comparisons. The 100ms flush window makes the durability trade-off visible and controllable rather than hidden. None of these are exotic tricks. But composed together, they produce a storage layer that is fast, durable, and recoverable — not despite each other, but because of how they interact. That's what systems programming looks like when it's working well: extremely small, well-reasoned decisions that compound into something that feels like it shouldn't be possible at the performance numbers it achieves. --- *Highway Broker is a custom message broker I built for high-throughput IoT telemetry. The full source includes the C++ storage engine, JavaScript and Python client libraries, and protocol implementation.* *If you have questions about any part of the architecture, drop a comment below.*

What Happens When Your Server Loses Power Mid-Processing?

Comments (0)