Distributed KV Store (Raft)

What it is

• A key-value store written in Go, built from scratch around the Raft consensus algorithm so multiple nodes agree on a single ordered log of writes.
• Supports both a single-node mode (write-ahead log + in-memory store, no consensus overhead) and a 3-node clustered mode with leader election, log replication, snapshots, and crash recovery.
• Clients can talk to it over a plain-text TCP protocol or an HTTP/JSON API, and there's a small web dashboard for watching cluster state live.

Use cases

• A learning-oriented reference implementation for how Raft leader election, log replication, and snapshotting actually work end to end, not just in theory.
• A base to build on for anything that needs a small, self-hosted, fault-tolerant key-value store without pulling in etcd or Consul.
• Testing failover behavior directly: kill the leader mid-cluster and watch a new one get elected, or restart a node and watch it catch up from a snapshot.

Tech stack

• Language: Go
• Consensus: Custom Raft implementation (leader election, log replication, snapshots, follower catch-up)
• Storage: Write-ahead log (WAL) + in-memory store for single-node mode; bbolt for persistence in cluster mode
• CLI: Cobra (kvctl)
• API: Plain-text TCP protocol, HTTP/JSON API, embedded HTML dashboard

Project structure

main.go                     Entrypoint, starts a node in single-node or cluster mode based on flags
cmd/kvctl/                  Command-line client, opens a TCP connection and sends GET/PUT/DELETE commands
config/cluster.json         Cluster topology, node IDs and their Raft/KV/HTTP addresses
internal/raft/              Core Raft implementation:
  node.go                     Node state machine and lifecycle
  election.go                 Leader election / heartbeats
  replication.go               Log replication to followers
  log.go                       In-memory Raft log
  apply.go                     Applies committed entries to the store
  snapshot.go                  Snapshotting and follower catch-up after restarts
  persister.go                 Durable persistence of Raft state (bbolt)
  rpc.go, transport.go         Inter-node RPC transport
internal/server/            TCP server implementing the text protocol (PUT/GET/DELETE), routes writes through Raft
                             when clustered, otherwise straight to the WAL + store
internal/httpapi/           HTTP/JSON API (/keys/{key}, /status, /metrics) and the embedded live dashboard
  web/dashboard.html          Single-page dashboard showing node status, metrics, and recent ops
internal/store/             In-memory key-value store
internal/wal/                Write-ahead log for single-node durability
internal/config/            Cluster config loading
scripts/cluster_test.sh     Smoke test, builds binaries, starts a 3-node cluster, exercises failover and restarts
diagrams/                   Architecture diagram (index.html)

Setup

Build the server and CLI:

go build -o bin/kvstore .
go build -o bin/kvctl ./cmd/kvctl

Cluster mode (3 nodes)

Start each node in its own terminal:

./bin/kvstore --id node1 --config config/cluster.json --data data
./bin/kvstore --id node2 --config config/cluster.json --data data
./bin/kvstore --id node3 --config config/cluster.json --data data

Talk to the cluster with kvctl against any node's kv_addr from config/cluster.json, or open the dashboard at the node's http_addr (e.g. http://localhost:9001/).

Run the smoke test to see leader election and failover in action:

bash scripts/cluster_test.sh

Single-node mode

go run . --single --data data

Uses the WAL + in-memory store directly, no Raft involved.

Requirements

• Go 1.21+