Bookstore-Microservices

A distributed online bookstore built as three Flask microservices that talk to each other over HTTP.

A frontend gateway sits in front of two catalog replicas and two order replicas. It spreads requests across replicas with round-robin load balancing, caches single-book lookups in an in-memory LRU cache, and fails over to the peer replica when one goes down. The backends keep the cache honest by pushing invalidations to the gateway before they write, and keep each other in step by replaying every write on their peer.

Five processes, four SQLite databases, no external infrastructure.

---

Architecture

                              ┌──────────┐
                              │  Client  │
                              └────┬─────┘
                                   │ HTTP
                                   ▼
   ┌───────────────────────────────────────────────────────────┐
   │                  Frontend gateway  :5000                  │
   │                                                           │
   │      round-robin balancer   +   LRU cache (by book id)    │
   └──┬─────────────────────┬────────────────────────────▲─────┘
      │ GET /search         │ POST /purchase/<id>        │
      │                     │                            │ DELETE /cache/<id>
      │                     │                            │ "drop this book,
      │                     │                            │  I'm about to write"
      ▼                     ▼                            │
 ┌─────────────┐      ┌─────────────┐                    │
 │  Catalog 1  │◄─────│   Order 1   │────────────────────┤
 │    :5001    │      │    :5002    │                    │
 │ catalog1.db │      │  orders1.db │                    │
 └──────▲──────┘      └──────▲──────┘   PUT /update/<id> │
        │                    │          (decrement stock)│
        │ POST /sync         │ POST /sync                │
        │ (replay write      │ (replay write             │
        ▼  on peer)          ▼  on peer)                 │
 ┌─────────────┐      ┌─────────────┐                    │
 │  Catalog 2  │◄─────│   Order 2   │────────────────────┘
 │    :5003    │      │    :5004    │
 │ catalog2.db │      │  orders2.db │
 └─────────────┘      └─────────────┘

The arrow worth noticing is the one pointing backwards, from the backends up to the gateway. Caches usually go stale because the thing holding the data has no idea the source changed. Here the write path is responsible for saying so: before any replica commits a change to a book, it sends DELETE /cache/<id> to the gateway. Invalidate first, then write — so a read racing the write can never re-cache a value that outlives it.

Each backend replica is the same application started with a different env file. Replica pairs are cross-wired: order 1 reads stock from catalog 1, order 2 from catalog 2.

---

Features

• Round-robin load balancing — the gateway alternates replicas per request via itertools.cycle, with no shared state to coordinate.
• Automatic failover — a ConnectionError against one replica is retried against its peer, so a single replica dying is invisible to clients.
• LRU cache with server-push invalidation — single-book lookups are served from an OrderedDict behind a lock, evicting least-recently-used past CACHE_LIMIT. Backends invalidate before writing rather than letting entries expire.
• Peer-to-peer replication — every write is replayed on the peer replica through an internal /sync endpoint that applies changes without re-propagating, so there is no loop.
• Two-tier auth — X-Admin-Key guards catalog and order administration, X-Internal-Key guards service-to-service endpoints that clients must never call directly.
• Uniform response envelope — every endpoint returns {"success": true, "message": ..., "data": ...} or {"success": false, "error": ...}.

---

API

Clients only ever talk to the gateway on port 5000. The backend tables document what the gateway forwards to, and are useful for inspecting a single replica directly.

Frontend gateway — :5000

Method	Path	Auth	Purpose
GET	/search	—	Search books; supports id, topic, title, min_price, max_price
POST	/purchase/<id>	—	Buy one copy of a book
POST	/books	X-Admin-Key	Add a book
PUT	/books/<id>	X-Admin-Key	Update a book's title, topic, price, or quantity
DELETE	/books/<id>	X-Admin-Key	Delete a book
GET	/orders	X-Admin-Key	List all orders, newest first
GET	/orders/<id>	X-Admin-Key	Fetch one order
GET	/cache	—	Inspect cache size and cached book ids
DELETE	/cache	—	Clear the whole cache
DELETE	/cache/<id>	—	Invalidate one book (called by backends before writes)
GET	/health	—	Per-replica up/down/timeout status

Catalog service — :5001, :5003

Method	Path	Auth	Purpose
GET	/search	—	Query books by id, topic, title, or price range
POST	/books	X-Admin-Key	Add a book; rejects duplicate titles
PUT	/books/<id>	X-Admin-Key	Update one or more fields
DELETE	/books/<id>	X-Admin-Key	Remove a book
PUT	/update/<id>	X-Internal-Key	Adjust stock by a signed delta; refuses to go negative
POST	/sync	X-Internal-Key	Apply a peer replica's write locally

PUT /update/<id> takes a delta rather than an absolute value — -1 for a purchase, +5 for a restock — so the caller controls direction and concurrent adjustments compose instead of clobbering each other.

Order service — :5002, :5004

Method	Path	Auth	Purpose
POST	/purchase/<id>	—	Verify stock, decrement it in the catalog, record the order
GET	/orders	X-Admin-Key	List all orders, newest first
GET	/orders/<id>	X-Admin-Key	Fetch one order
POST	/sync	X-Internal-Key	Apply a peer replica's write locally

Examples

# Search by topic
curl "http://127.0.0.1:5000/search?topic=distributed%20systems"

# Fetch one book — this response gets cached
curl "http://127.0.0.1:5000/search?id=2"

# Buy it — invalidates the cache entry and decrements stock on both catalog replicas
curl -X POST "http://127.0.0.1:5000/purchase/2"

# Update a price (admin)
curl -X PUT "http://127.0.0.1:5000/books/2" \
     -H "X-Admin-Key: $ADMIN_KEY" \
     -H "Content-Type: application/json" \
     -d '{"price": 55.0}'

# Check every replica
curl "http://127.0.0.1:5000/health"

---

Setup

With Docker

docker compose up --build

That builds all five containers from one Dockerfile and puts them on a shared network where they address each other by service name. The gateway is on http://127.0.0.1:5000. Set your own keys first if you like:

export ADMIN_KEY=your-admin-key
export INTERNAL_KEY=your-internal-key
docker compose up --build

Without Docker

Requires Python 3.10+.

pip install -r requirements.txt

Copy each env template and fill in your keys — ADMIN_KEY and INTERNAL_KEY must match across all five files:

cp catalog_server/.env.replica1.example catalog_server/.env.replica1
cp catalog_server/.env.replica2.example catalog_server/.env.replica2
cp order_server/.env.replica1.example   order_server/.env.replica1
cp order_server/.env.replica2.example   order_server/.env.replica2
cp frontend_server/.env.example         frontend_server/.env

Then start five processes, one per terminal. DOTENV_FILE selects which env file a backend loads:

# Terminal 1 — catalog replica 1
cd catalog_server;  $env:DOTENV_FILE=".env.replica1"; python app.py

# Terminal 2 — catalog replica 2
cd catalog_server;  $env:DOTENV_FILE=".env.replica2"; python app.py

# Terminal 3 — order replica 1
cd order_server;    $env:DOTENV_FILE=".env.replica1"; python app.py

# Terminal 4 — order replica 2
cd order_server;    $env:DOTENV_FILE=".env.replica2"; python app.py

# Terminal 5 — gateway
cd frontend_server; python app.py

On macOS or Linux, use DOTENV_FILE=.env.replica1 python app.py instead.

Each catalog replica creates and seeds its own SQLite database on first start. Confirm everything is wired up with curl http://127.0.0.1:5000/health.

Ports

Service	Port	Database
Frontend gateway	5000	—
Catalog replica 1	5001	catalog1.db
Order replica 1	5002	orders1.db
Catalog replica 2	5003	catalog2.db
Order replica 2	5004	orders2.db

Configuration

Variable	Applies to	Meaning
PORT	all	Port to listen on
DEBUG	all	Werkzeug debug mode and auto-reloader
DOTENV_FILE	backends	Which env file to load (default .env)
DB_NAME	backends	SQLite filename, one per replica
REPLICA_URL	backends	Peer replica to sync writes to; empty disables replication
CATALOG_URL	order	Catalog replica this order replica reads stock from
FRONTEND_URL	backends	Where to send cache invalidations
CATALOG_URL_1/2, ORDER_URL_1/2	frontend	Replica pool for the load balancer
CACHE_LIMIT	frontend	Max cached books before LRU eviction (default 10)
ADMIN_KEY	all	Shared secret for admin endpoints
INTERNAL_KEY	all	Shared secret for service-to-service endpoints

---

Performance

Measured with python docs/measure.py against all five services on one machine (Windows 11, Python 3.14, Werkzeug development server, DEBUG=False). N=100 per read scenario, N=10 for purchases.

Scenarios 1 and 2 issue the identical request — GET /search?id=2. The only difference is that scenario 2 clears the cache before every call, so the delta isolates the cache and nothing else.

Scenario	Avg	Min	Max	p95
id lookup — cache hit	2.31 ms	2.07 ms	3.23 ms	2.65 ms
id lookup — cache miss	7.60 ms	5.56 ms	43.36 ms	19.53 ms
topic search (never cached)	7.26 ms	5.67 ms	43.78 ms	18.89 ms
purchase (invalidate + write + sync)	42.28 ms	8.39 ms	52.05 ms	52.05 ms

Cache speedup: 3.3×. A hit skips the gateway→replica round trip and the SQLite query entirely, returning straight from the OrderedDict. Hits are also far more predictable: the p95 of a hit (2.65 ms) is better than the minimum of a miss (5.56 ms), because a hit never touches a network socket.

Purchases cost roughly 18× a cached read, and that price is the consistency model showing up on the bill. One purchase fans out into a stock read, a cache invalidation, a stock decrement on the catalog (which invalidates and syncs again on its own), a local order write, and an order sync to the peer — six HTTP calls before the client hears back. Writes are rare compared to reads here, which is exactly the trade this design assumes.

Two notes on reading these numbers. First, they come from Werkzeug's development server, which is single-process and not built for load; a production WSGI server would be faster across the board, though the ratios are the interesting part and would hold. Second, the services address each other as 127.0.0.1 rather than localhost on purpose — localhost resolves to IPv6 ::1 first on Windows, and since the dev server binds IPv4 only, every single call ate a failed connect and a fallback. That one substitution took cache hits from roughly 2000 ms to 2.31 ms. If you fork this and see absurd loopback latency, check that first.

---

Design decisions

Only id lookups are cached. Topic, title, and price-range queries return lists and are always forwarded. Caching a list means any write to any book in it invalidates the whole entry, and working out which cached lists contain a mutated book is bookkeeping that costs more than the hit rate is worth. Restricting the cache to id → book keeps invalidation exact: one write, one DELETE /cache/<id>.

Invalidate before writing, not after. The ordering is the whole guarantee. Invalidating after a commit leaves a window where a concurrent read can fetch the old value and cache it after the invalidation has already swept past — a stale entry that survives indefinitely. Doing it first closes that window: any read racing the write either sees the pre-write value and caches it before the invalidation lands, or misses and fetches fresh.

Replication is synchronous but best-effort. After a local commit, the replica POSTs the same change to its peer and waits. If the peer is down or slow, the sync is logged as a warning and abandoned rather than rolling back the local write — availability over strict consistency. The /sync handler deliberately never re-propagates, which is what stops two peers from syncing each other forever.

Round-robin over anything smarter. Alternating replicas needs no shared state, no coordination, and no health tracking, which suits two homogeneous replicas on one machine. Least-connections or latency-weighted routing would adapt better to uneven load, but needs state the gateway would then have to maintain and reason about.

SQLite over a real database. Four independent SQLite files make replication observable — you can open each replica's database and watch them converge. Nothing in the service code assumes SQLite; the trade is documented below.

---

Known limitations

These are real and worth knowing before building on this.

• Replicas can diverge. If a sync times out, the peer never gets the write and the two databases disagree until a later overlapping write happens to correct them. There is no retry queue, no write-ahead log, and no automatic reconciliation. A WAL with retries is the standard fix.
• A failed invalidation leaves stale data. invalidate_cache tries once. If the gateway is unreachable at that moment, the stale entry survives until the next write to that book or until LRU evicts it.
• The cache dies with the gateway process. It is in-memory and process-local, so a restart empties it and a second gateway instance would have a completely separate one. A shared cache service would fix both.
• The balancer only notices a replica is down by failing to reach it. There is no background health checking, so a replica that is slow but alive keeps getting half the traffic. /health reports status but does not feed back into routing.
• SQLite serializes writers. One writer at a time per file, so concurrent purchases against one replica queue up. Correctness is unaffected; throughput is not.
• X-Internal-Key is a shared static secret. It travels in a plaintext header with no rotation, expiry, or replay protection. Fine on a private network, insufficient on a public one — mutual TLS or signed tokens would be the real answer.
• Development server only. Everything runs under Werkzeug. Put it behind Gunicorn or uWSGI before pointing real traffic at it.

---

Project layout

frontend_server/     gateway — load balancer, LRU cache, no database
catalog_server/      book catalog — search, admin writes, stock adjustments
order_server/        purchases and order history
docs/measure.py      performance measurement script
Dockerfile           one image definition, parameterized by service
docker-compose.yml   all five services on a shared network

Backends share a layout: app.py (factory and startup), config.py (env-driven settings), models.py (schema), routes.py (endpoints).

---

License

MIT — see LICENSE.