Advanced Software Engineering Concepts

Hands-on projects built to learn distributed systems and backend engineering concepts properly — by implementing them, breaking them on purpose, and measuring what happens.

Each project is self-contained, runs locally with docker compose, and comes with its own documentation.

#	Project	Core concepts
1	Webhook Kafka Delivery System	event streaming, at-least-once delivery, idempotency, circuit breakers, transactional outbox, centralised observability
2	GoLang Fundamentals	the Go language ground-up: types, structs, interfaces, errors, packages, goroutines, channels, testing

---

1. Webhook Kafka Delivery System

A production-shaped webhook delivery service — the kind of infrastructure Stripe, GitHub and Shopify run internally. Customers register a URL; when an event occurs it is delivered to them reliably: HMAC-signed, retried with exponential backoff, and dead-lettered when it never succeeds.

Receiving a webhook is one HTTP endpoint. Sending them reliably is a genuine distributed-systems problem, which is what makes it a good vehicle for these concepts.

Stack

FastAPI · Redpanda (Kafka) · SQLite + Alembic · Redis · Splunk · Docker Compose

Eleven containers: an API, three Kafka consumers, an outbox relay, a subscriber simulator, and the supporting infrastructure.

Architecture

   POST /v1/events
         │
         ▼
   ┌───────────┐   idempotency    ┌───────┐
   │  FastAPI  │─────────────────▶│ Redis │
   │    API    │                  └───────┘
   └─────┬─────┘
         │ ONE transaction: event row + outbox rows
         ▼
   ┌───────────────────┐
   │ SQLite            │   the API never talks to Kafka
   │  events + outbox  │
   └─────────┬─────────┘
             │ polled by
             ▼
   ┌───────────────────┐
   │  relay            │   the only publisher
   └─────────┬─────────┘
             │ produce (key = subscription_id)
             ▼
   ┌──────────────────────────────────┐
   │  Redpanda    webhook.events      │
   └───┬──────────────────────────┬───┘
       │                          │
  group: delivery-workers   group: analytics    ◀── independent offsets
       │                          │
       ▼                          ▼
  ┌──────────┐            ┌─────────────┐
  │ Delivery │            │  Analytics  │
  │  Worker  │            │  Consumer   │
  └────┬─────┘            └─────────────┘
       │ HMAC-signed POST
       ▼
  ┌──────────────┐
  │  subscriber  │
  └──────────────┘
       │ on failure
       ├──▶ webhook.retries ──▶ backoff + jitter ──▶ retry worker
       └──▶ webhook.dlq      (after max attempts)

  every service ──── structured JSON logs ────▶ Splunk HEC

Concepts, and where each one lives

Concept	Implementation
Event streaming, partitions, consumer groups	infra/kafka.py, workers/run.py
At-least-once delivery (manual offset commits)	workers/run.py
Consumer group replay and lag	workers/analytics.py
Idempotency keys	api/routes.py
Exponential backoff with full jitter	domain/retry_policy.py
Dead-letter queue	workers/deliverer.py
Circuit breaker (Redis-backed, shared)	infra/cache.py
HMAC-SHA256 request signing	domain/signing.py
Transactional outbox (dual-write problem)	workers/outbox_relay.py
Structured logging, non-blocking log shipping	observability/splunk_hec.py
Correlation IDs across processes	observability/context.py, middleware.py
Schema migrations	migrations/

Results that were actually measured

Not design intentions — outputs from running the system.

Consumer group independence and replay. The analytics consumer was stopped, fell 24 messages behind, restarted and caught up. It was then rewound to offset zero and replayed all 83 historical events — while the delivery workers logged zero deliveries. No subscriber received a duplicate.

Ingest survives a total broker outage. With Redpanda stopped, 10/10 events were accepted (HTTP 202) and held in the outbox. On recovery all 30 pending messages published automatically and all 10 events reached a subscriber. Nothing was lost — the payoff of the transactional outbox.

The circuit breaker's value, quantified. For a subscriber whose URL could not resolve: 4 events × 5 max attempts = 20 delivery slots. Only 5 became real HTTP attempts; the breaker short-circuited the other 15 without opening a socket.

Jitter, visible in the logs. Five deliveries failed in the same millisecond and scheduled retries at 0.21s, 0.58s, 0.24s, 1.17s, 1.03s. Plain exponential backoff would have fired all five at exactly 2.00s.

Latency is bimodal, and averages lie. p50 = 8ms, p95 = 1710ms, mean = 356ms — a number that describes neither group.

Running it

cd Webhook-Kafka-Delivery-system
cp .env.example .env
docker compose up -d

Splunk takes 1–3 minutes on first boot.

URL
http://localhost:8000	Splunk — dashboards and log search
http://localhost:8090	Redpanda Console — topics, partitions, consumer group lag
http://localhost:8080/docs	API

Then generate traffic:

curl -X POST http://localhost:8080/v1/subscriptions \
  -H 'Content-Type: application/json' \
  -d '{"name":"acme-corp","target_url":"http://subscriber-sim:9000/hook",
       "event_types":["*"],"secret":"dev-shared-secret-for-the-simulator"}'

python scripts/loadgen.py --events 100

Documentation

• Project README — setup, layout, and every bug hit while building it
• Kafka concepts — queue vs log, offsets, and three re-runnable experiments
• SPL cheatsheet — validated Splunk queries

---

2. GoLang Fundamentals

A ground-up tour of the Go language, built as the foundation for the concurrent, networked backend work the rest of this repo depends on. Go was designed at Google to make large-scale, concurrent software that is fast to compile, fast to run, and simple to read — this track walks that design one concept at a time.

Unlike the projects above, this is a learning module, not a system to run and break. Each topic is an isolated, runnable program in its own folder, built up in order.

Layout

One Go module at the root (go.mod), one package main per numbered folder. Run any topic from the GoLang/ directory with go run ./<folder>.

#	Topic	What it covers
01	variables	declarations, type inference, :=, constants, unused-variable errors
02	functions	parameters, return types, multiple return values
03	control_flow	if with short statements, for as the only loop, switch (no fallthrough)
04	slice_maps	arrays vs slices, append, range, maps, the comma-ok idiom
05	structs	custom types, methods, value vs pointer receivers, capitalization = visibility
06	pointers	& and *, modifying through a pointer, nil, no pointer arithmetic
07	interfaces	method-set contracts, implicit satisfaction, polymorphism
08	errors	errors as values, the error interface, if err != nil
09	packages	package layout, import paths from the module name, exported vs unexported
10	goroutines	the go keyword, sync.WaitGroup, the M:N scheduler
11	channels	send/receive, blocking, buffered channels, close + range, deadlocks
12	stdlib	strings, strconv, math, sort, time
13	testing	_test.go convention, testing.T, table-driven tests

Running it

cd GoLang
go run ./05_structs        # run any single topic
go test ./...              # run every test in the module

The recurring lesson

The through-line is Go's deliberate simplicity: it often lacks a feature other languages have — no while, no exceptions, no implements keyword, no classes — and almost always that absence is the design, not an oversight. Concurrency (goroutines + channels) is where this pays off, and it feeds directly into the distributed-systems work above.

---

Notes

Every project here is built to be run and broken, not just read. Failure modes are triggered deliberately — a database lock storm, a dead subscriber, a consumer falling behind, a broker outage — because the failure is where the concept actually lives.

Production gaps are documented honestly in each project's README rather than quietly ignored.