All projects
Overview
A message broker for real-time telemetry in the spirit of Kafka, written from scratch in C++17 with Boost.Asio. Producers publish to topics, consumers subscribe with wildcard patterns, and messages are persisted to a log so they survive restarts.
The goal was to understand what makes brokers fast and durable by building the pieces myself: the wire protocol, the storage engine and the asynchronous I/O loop.
What I built
- Architected a Kafka-like C++ message broker with persistent log storage and wildcard topic routing.
- Designed a double-buffered async I/O engine with sparse indexing for O(log N) offset lookup.
- Implemented a custom TCP protocol with QoS levels and crash recovery through log replay.
Design and architecture
- Storage: an append-only persistent log with a sparse index, so finding a message by offset is O(log N).
- I/O: a double-buffered asynchronous engine built on Boost.Asio, built with CMake and Conan.
- Protocol: a custom TCP protocol with QoS levels, and crash recovery by replaying the log.