Overview
DetLog is a C++20 replicated command log with a deterministic failure simulator. I built it as a small, inspectable implementation of Raft’s safety protocol: not a production database, but a system where every vote, write, acknowledgement, crash, and recovery step can be examined and reproduced.
The project includes a pure consensus core, a checksummed write-ahead log, a real TCP node runtime, idempotent client sessions, seeded fault campaigns, and a published evidence corpus. The central question behind all of it was simple: can I make distributed-systems correctness observable instead of merely asserting it?
The Problem
Raft is often introduced through its clean high-level rules: elect a leader, replicate entries to a majority, and apply committed commands in order. The difficult bugs live below that description. A vote response sent before its term is durable, a success acknowledgement released before a flush, or a timer lost during a role transition can break safety or strand a healthy quorum.
Ordinary integration tests make these failures hard to investigate. Thread timing changes between runs, network faults are difficult to reproduce, and the first visible symptom can occur long after the incorrect event. I wanted a test environment where the same seed and action list always recreate the same history—and where the real protocol bytes still pass through production decoders.
Technical Approach
A Pure Consensus Core
Each node owns a single RaftNode state machine. The core receives one protocol, timer, client, or storage-completion event and returns effects. It never reads a clock, opens a socket, performs file I/O, sleeps, or mutates shared state.
protocol / timer / client / storage completion
│
▼
RaftNode
│
┌───────────────┼───────────────┐
▼ ▼ ▼
durable batches messages/timers client repliesThat boundary made the same consensus implementation usable in two very different environments: a deterministic virtual-time simulator and a bounded real TCP runtime. It also made persistence ordering explicit. Actions that depend on stable state remain pending until the matching storage completion arrives.
A node therefore cannot grant a vote, acknowledge an appended entry, start an election based on a new self-vote, or return a successful client result before the required durability barrier succeeds.
Durable Recovery and Ambiguous Retries
Every node stores a versioned, checksummed WAL bound to its cluster and node identity. A frame can carry a hard-state update, a logical suffix deletion, replacement entries, and a monotonic commit watermark. Recovery replays complete frames to reconstruct logical state.
An incomplete final frame is treated as an unacknowledged torn tail and can be removed. A complete frame with a bad checksum, malformed interior data, identity mismatch, or semantic regression fails closed.
Client commands use a 128-bit session ID and monotonically increasing sequence number. The replicated session table remembers the most recent command digest and result, so retrying an ambiguous request returns the cached result without appending or applying it twice. Reusing a sequence number for different data is rejected deterministically.
Deterministic Failure Simulation
The simulator runs on integer virtual time with a pinned pseudorandom generator. Network injection changes delivery—not in-memory C++ objects—so messages still travel through the same codec used by TCP.
Named scenarios cover leader crashes, symmetric and asymmetric partitions, ambiguous retries, torn WAL tails, slow followers, slow disks, and saturation. The broader action generator adds message loss, delay, duplication, reordering, process restarts, and queue pressure. A failing randomized history reports its seed, complete action list, and a deletion-minimized action list that preserves the failure.
build/debug/detlog-sim \
--seed 42 \
--nodes 3 \
--scenario leader-crash \
--trace trace.jsonlThe trace records role and term changes, storage operations, timer generations, network delivery, commits, and recovery as structured JSONL. A saved trace becomes a durable debugging artifact rather than a transient test failure.
Testing Correctness Under Failure
DetLog tests the same rules from several independent angles:
- codec and WAL unit tests cover bounds, checksums, logical truncation, and recovery at every final-frame cut;
- table-driven Raft tests expose each message and storage completion;
- a separate serial reference model enables differential testing;
- named failure histories run invariant checks after every event and injected fault;
- parser fuzz targets exercise protocol frames, WAL bytes, and serialized scenarios;
- real loopback tests cover TCP framing, storage workers, crash/restart, and bounded group commit; and
- CI runs a 2,000-seed deterministic property campaign on every published commit.
This approach found a liveness bug where a stale candidate could force the leader to step down, receive a correct vote denial, and leave the live quorum with no election timer. The first repair introduced another starvation path under repeated stale-candidate traffic. Reducing the failure to a minimal deterministic history made it possible to fix both timer-ownership cases and preserve them as focused regressions. No committed-data safety property was violated, but availability could halt permanently.
Results
The published benchmark corpus contains 432 cluster simulator/TCP runs, 324 runtime-fsync runs, and 108 WAL append/reopen runs. All 151,200 cluster and runtime operations completed successfully, and every WAL run recovered the requested entry count and commit index.
Across 108 real TCP leader-crash and partition trials, the pooled nearest-rank P99 time to resume successful commits was 1.453 seconds. This includes replacement-leader election, replication, WAL durability, application, and the client response.
Bounded group commit demonstrated a useful trade-off. With five closed-loop clients, grouping improved geometric-mean throughput by 13.4% to 14.8% while issuing about 18% fewer flushes. In leader-crash workloads, however, median time to the first successful post-failure commit was 9.4% to 10.7% slower. Better healthy-load batching did not automatically mean better recovery latency.
These measurements compare DetLog configurations against each other on one recorded workstation. They are descriptive engineering evidence—not a production database comparison or a claim of broad statistical confidence. Unsafe no-flush measurements are labelled separately and never used to support crash-safety claims.
What I Learned
The most important lesson was that consensus safety is as much about suppressing premature effects as it is about choosing the right state transition. A correct in-memory decision is still wrong if its acknowledgement escapes before the state supporting it is durable.
The stale-candidate bug also changed how I think about liveness tests. Individual rules can each be correct while their composition leaves no future event capable of making progress. Deterministic simulation made that absence visible: the failure was not corrupted state, but a timer that would never fire.
Finally, reproducibility has to extend beyond the test runner. Pinning seeds was only the beginning. Useful evidence required preserving scenario configuration, action lists, minimized failures, environment captures, raw benchmark streams, derived figures, and checksums. That work turned DetLog from a protocol exercise into an experiment I could audit.
Project Scope
DetLog uses static three- or five-node clusters. It does not implement dynamic membership, snapshots or compaction, Byzantine tolerance, authentication, leader leases, or a production-ready operational surface. Keeping that boundary explicit let the project stay focused on durable Raft behavior, deterministic failure semantics, and evidence-backed recovery.