What Happens When Your Server Loses Power Mid-Processing?

There's a question that hits every systems engineer building high-throughput infrastructure at some point:

Your server is processing thousands of messages per second. All of it lives in RAM. The power goes out. What survives?

I faced this exact question while developing Highway Broker — a custom message broker I built from scratch. The answer wasn't just "write to disk." It forced me to think carefully about how to write to disk without destroying the throughput that made the system worth building in the first place.

This is the story of the architecture I designed, the trade-offs I made explicit, and the patterns that made it work.

The Problem: RAM Is Volatile by Design

RAM is fast because it's electrically charged. Data exists as long as power flows. The moment it's cut:

• All in-flight messages evaporate instantly
• Partially written structures are silently corrupted
• Any message not yet persisted is permanently gone

For a message broker, this breaks the fundamental contract: a producer acknowledged a send, a consumer is expecting delivery. Losing data mid-flight isn't just a performance issue — it's a correctness violation.

The naive fix is obvious: write every message synchronously to disk before acknowledging it. But synchronous disk I/O on the hot path is a death sentence for throughput. You've traded one problem for another.

The real challenge is: how do you get durability and performance?

The Architecture: A Two-Stage Write Pipeline

The answer I landed on is a staged write pipeline that completely decouples the producer's hot path from disk I/O. Here's the core design, straight from storage_manager.hpp:

// Double-buffered write — pre-allocated, no heap allocation on hot path
Buffer buffer_pool_[2];                // Pre-allocated buffers
std::atomic<Buffer *> active_buffer_;  // Current write target (event loop writes here)
std::atomic<Buffer *> flush_buffer_;   // Pending flush (background worker drains this)
std::atomic<bool> flush_signaled_{false};

Two buffers. Two roles. Never blocking each other.

Stage 1 — The Active Buffer (Memory Speed)

Every incoming message gets appended to the active buffer at memory speed. The Buffer::append_message implementation is lock-free on the hot path — no mutex, no syscall, no waiting:

/**
 * Append a message to buffer (hot path, O(1), no locks).
 * If buffer is full, returns false.
 */
bool append_message(uint64_t offset, const std::vector<uint8_t> &payload);

Each message is written in a compact binary layout:

| uint64 offset | uint32 size | uint32 crc | payload (var) |
| 8 bytes       | 4 bytes     | 4 bytes    | size bytes    |

The MessageHeader is exactly 16 bytes — not by accident. Fixed-size headers make forward scanning over the segment file trivially predictable for the CPU prefetcher.

Two conditions trigger a flush:

• Buffer reaches 95% capacity — the is_full() threshold avoids exact-size edge cases
• 100ms timer elapses — critical for low-traffic topics that might otherwise sit unflushed indefinitely

Stage 2 — The Atomic Buffer Swap

When flush conditions are met, the active buffer swaps atomically with the flush buffer. This is the key insight of the whole design:

/**
 * Acquire buffer that needs flushing.
 * Swaps active <-> flush atomically.
 * Returns pointer to buffer containing accumulated messages.
 */
Buffer *acquire_flush_buffer();

The event loop thread continues writing to the new active buffer immediately after the swap. There is no lock held across disk I/O. Zero producer stall. The old data is handed off to the FlushWorker background thread.

Stage 3 — The FlushWorker (Durability Without Blocking)

The FlushWorker runs in a dedicated thread per topic. It owns the flush path entirely:

class FlushWorker {
public:
  enum class DurabilityMode {
    FAST,      // Flush every ~100ms (default for telemetry)
    BALANCED,  // Flush when buffer fills
    SAFE       // Per-message (reduces throughput significantly)
  };
  // ...
private:
  void worker_loop();
  void flush_buffer(Buffer *buf);
  void write_to_segment(Buffer *buf, SegmentMetadata &segment);
};

The worker loop waits on a condition variable, wakes on signal(), drains the flush buffer to disk, then goes back to sleep. Simple. Predictable. Non-blocking to everything else.

Segment Log Files: Append-Only, Offset-Indexed

The FlushWorker writes to segment log files — append-only binary files where each message carries its own offset. Here's the SegmentMetadata that tracks each segment's lifecycle:

struct SegmentMetadata {
  uint64_t id;              // Segment ID (base offset)
  uint64_t base_offset;     // First offset in this segment
  uint64_t last_offset;     // Last offset written
  uint64_t byte_position;   // Current write position
  uint64_t max_size_bytes;  // Rotation threshold (512MB)
  std::string file_path;    // Full path to segment file
  bool is_active;           // Still accepting writes?
};

A few constraints are fundamental to the design:

• Only writes — no updates, no deletes, only appends
• Only reads in special cases — replay and FETCH_ONE operations
• Rotation when full — a new segment is created when the active one exceeds 512MB

This append-only invariant is what makes everything else fast. There's no random I/O. No file fragmentation. The OS can do large sequential writes, which is the single most efficient I/O pattern on any storage medium.

The Sparse Index: O(log N) Reads

Appending is fast, but finding a specific message offset in a large segment file requires an index. I implemented a sparse index — one entry per 1024 messages — that maps logical offsets to file byte positions:

struct SparseIndexEntry {
  uint64_t offset;         // Message offset (sequence number)
  uint64_t file_position;  // Byte offset in segment file
};

class SparseIndex {
public:
  // Binary search: find file position for offset ≤ target
  std::optional<uint64_t> find_file_position(uint64_t target) const;
private:
  static constexpr uint64_t INDEX_INTERVAL = 1024;
  std::vector<SparseIndexEntry> entries_;
  mutable std::shared_mutex mutex_;
};

The read path becomes:

1. Binary search on the sparse index → find approximate file position (O(log N))
2. Sequential scan forward from that position → find exact offset (O(k), bounded to ≤64KB window)
3. Validate CRC32 → confirm integrity
4. Return payload

For a segment with 1 million messages, the index has ~976 entries. Binary search finds the right window in ~10 comparisons. Then a short sequential scan to the exact message. In practice, this is fast enough that reads don't compete meaningfully with writes.

Per-Topic Isolation: Independent Storage Managers

Each topic in Highway Broker gets its own SegmentLog, its own FlushWorker, and its own buffer pair. The StorageManager acts as the registry and coordinator:

// Topic registry: topic_name -> SegmentLog
std::unordered_map<std::string, std::unique_ptr<SegmentLog>> logs_;
mutable std::shared_mutex logs_mutex_;

// Flush workers per topic
std::unordered_map<std::string, std::unique_ptr<FlushWorker>> flush_workers_;

This isolation has real operational consequences:

• A slow consumer on highway/A1/telemetry cannot starve flushes on highway/A2/telemetry
• Buffer tuning (size, flush interval, durability mode) is adjustable per topic
• A storage failure in one topic is scoped — it cannot corrupt another topic's segments
• Retention and compaction run independently per topic

The FlushWorker::DurabilityMode can be set per topic: a critical control topic might run SAFE mode (flush-per-message), while a high-volume telemetry topic runs FAST mode (flush every 100ms). Same broker, different guarantees.

Controlled Offset Access: Highway Broker v1.1

The append architecture unlocks a powerful read primitive: direct offset access. Because every message has a monotonically increasing offset embedded in its header, you can ask for any message by its offset and retrieve it in O(log N) time.

Highway Broker v1.1 exposes this through two new access patterns:

FETCH_ONE — Stateless Random Read

CLIENT                          BROKER
  |                               |
  |-- FETCH_ONE(topic, offset) -->|
  |                               |
  |     [validate offset range]  |
  |     [sparse index lookup]    |
  |     [sequential scan]        |
  |                               |
  |<-- FETCH_RESPONSE(data) ------|
  |    OR OFFSET_NOT_FOUND        |

Completely stateless. No subscription created. No state stored on the broker. A REST-like primitive on top of a streaming architecture.

The StorageManager exposes the range validation needed:

uint64_t get_head_offset(const std::string &topic) const;    // Latest written
uint64_t get_oldest_offset(const std::string &topic) const;  // After retention

Invalid offsets return OFFSET_NOT_FOUND with the available range — so clients always know how to recover from out-of-range requests.

SUBSCRIBE_FROM_OFFSET — Catch-Up Then Live

The second pattern handles the "I was offline, catch me up" use case:

enum class SubscriptionMode {
  PUSH_LIVE,           // Default: deliver all new messages
  CATCHUP_THEN_PUSH    // Replay from offset, then switch to PUSH_LIVE
};

struct Subscription {
  Session *session;
  std::string pattern;
  QoS qos;
  SubscriptionMode mode;      // Which delivery mode
  uint64_t current_offset;    // Current replay position
};

A client subscribes with a start_offset. The broker delivers messages from that offset forward. When the subscriber catches up to head_offset, the subscription automatically transitions to PUSH_LIVE — no client logic required, no reconnect, no re-subscribe. The transition is logged for debugging:

Initial: CATCHUP_THEN_PUSH (current_offset = N)
  ↓
Receive messages N, N+1, N+2, ..., HEAD
  ↓
offset >= HEAD_OFFSET → switch to PUSH_LIVE

This makes Highway Broker useful for IoT telemetry scenarios where a sensor reconnects after hours offline and needs to replay everything it missed before receiving live data.

The Durability Window

No buffered write system provides absolute zero data loss under power failure. The architecture I built offers a well-defined durability window: messages in the active buffer but not yet flushed can be lost if power fails before the 100ms timer triggers.

This is not a bug. It is an explicit, engineered trade-off.

In practice, a 100ms window means at most ~100ms of message history is at risk under catastrophic failure. For most telemetry use cases, this is entirely acceptable. For stricter requirements, DurabilityMode::SAFE flushes per-message at the cost of throughput.

The important thing is that the window is explicit, measurable, and tunable — not a hidden assumption discovered in production.

Recovery on Startup

When the broker restarts after a crash, StorageManager::recover_all() kicks in:

/**
 * Recover segment log from disk.
 * Validates CRC, truncates at corruption, rebuilds sparse index.
 */
void recover();

The recovery process:

1. Scan the storage directory for segment files belonging to each topic
2. Read each segment, validating CRC32 on every message header
3. If corruption is detected, truncate the segment at the last valid position
4. Rebuild the sparse index from the valid portion
5. Restore next_offset_ to continue from the last valid offset

Because segments are append-only and each message has an embedded CRC, corruption is always detectable and always localized. The broker never needs to trust partially written messages — it simply discards them and resumes from the last known good state.

Write Path at a Glance

Producer
   │
   ▼
[ Active Buffer ]  ──── (full / 100ms) ──►  [ Flush Buffer ]
   │                                                │
   │  (atomic swap, zero blocking)          Background FlushWorker
   │                                                │
   ▼                                         Sequential Write
(continues writing)                                 │
                                            [ Segment Log File ]
                                            [ Sparse Index ]
                                                    │
                                         (CRC validated, offset indexed,
                                          append-only, recoverable)

Closing Thoughts

Building this storage engine from scratch taught me something that's hard to get from reading about Kafka or Pulsar: the patterns aren't arbitrary. They're the result of hard constraints converging on elegant solutions.

The double-buffer swap costs almost nothing — an atomic pointer exchange. The sequential append is the single most efficient I/O pattern available. The sparse index compresses 1M message lookups into ~10 comparisons. The 100ms flush window makes the durability trade-off visible and controllable rather than hidden.

None of these are exotic tricks. But composed together, they produce a storage layer that is fast, durable, and recoverable — not despite each other, but because of how they interact.

That's what systems programming looks like when it's working well: extremely small, well-reasoned decisions that compound into something that feels like it shouldn't be possible at the performance numbers it achieves.

Highway Broker is a custom message broker I built for high-throughput IoT telemetry. The full source includes the C++ storage engine, JavaScript and Python client libraries, and protocol implementation.

If you have questions about any part of the architecture, drop a comment below.