EchoLog

EchoLog is a Java 21 distributed log broker I built to understand what happens below a producer and consumer client. It uses a custom TCP protocol, Java NIO selectors, append only files, ZooKeeper, and three Docker brokers.

I made it to learn about socket reads, durable records, partition ordering, replica catch up, leader failure, and consumer offsets.

What it does

• Stores records in append only topic partitions with offsets.
• Uses CRC32C and record lengths to recover after an incomplete write.
• Handles native TCP frames with a Java NIO selector.
• Supports topic creation, produce, fetch, consumer groups, and offset commits.
• Uses ZooKeeper for broker membership, controller election, and partition data.
• Replicates with follower pull requests and replica acknowledgements.
• Tracks ISR and a high watermark for committed reads and acks=all writes.
• Uses leader epochs to reject old writes after failover.
• Exposes Prometheus metrics.

Architecture

flowchart LR
    P[Producer] --> L[Partition leader]
    L --> LOG[Append only log]
    F1[Follower] -->|pull| L
    F2[Follower] -->|pull| L
    F1 -->|ack progress| L
    F2 -->|ack progress| L
    L --> ZK[ZooKeeper metadata]
    ZK --> C[Controller]
    C -->|ISR and high watermark| ZK
    ZK --> L
    ZK --> F1
    ZK --> F2
    L -->|committed response| P

Loading

For acks=all, writing to the leader is not enough. EchoLog waits for follower progress and a controller high watermark checkpoint. Consumers only read offsets below that checkpoint.

Run it

Docker is required.

cd ~/echolog
./scripts/test-java21.sh

Start everything:

docker compose up --build

• Broker ports: 19092, 29092, 39092
• Metrics ports: 19404, 29404, 39404
• Prometheus: http://127.0.0.1:9090
• Grafana: http://127.0.0.1:3000

Clean local state:

docker compose down --remove-orphans -v

Failure demo

./scripts/fault-drill.sh

The script creates one topic with minISR=2, writes ten acks=all records, kills the leader, waits for a new leader and epoch, checks that an old epoch is rejected, starts the failed broker again, and waits for it to catch up.

A good run ends with:

Failover proof complete: committed sequence preserved, leader changed,
epoch advanced, stale epoch rejected, and restarted follower caught up.

Metrics screenshot

Prometheus scraped all three broker metric endpoints in the Docker network.

What I learned

• TCP is a stream. One socket read is not always one request.
• Ordering is per partition, not global.
• Only replicas that are currently in sync should count for a durable acknowledgement.
• If HWM is 10, offsets 0 to 9 can be read. Offset 10 cannot.
• Leader election by itself does not stop split brain. Epoch fencing rejects requests from an old leader.
• On recovery, keep the committed part and remove the uncommitted tail before pulling from the new leader.

Why I started this

I wanted to understand why logs show up in storage, replication, and event streaming. These articles gave me the idea to try a small version myself:

• Jay Kreps, The Log. This made append only logs make sense to me.
• Martin Kleppmann, Using logs to build a solid data infrastructure. This helped me understand ordering, recovery, and partial failures.
• Martin Kleppmann, Kafka, Samza, and the Unix philosophy of distributed data. This made the producer and consumer side feel less abstract.

I used the Kafka design docs and ZooKeeper guide when I needed implementation details. EchoLog only takes ideas from them. It is not a Kafka clone.

Limits

EchoLog is a learning project, not a production broker. It does not have Kafka protocol compatibility, transactions, idempotent producers, exactly once behavior, compaction, ACLs, TLS/SASL, compression, quotas, or a Raft/KRaft metadata quorum.

The benchmark is only a local workload result. It does not prove feature parity or production performance.