AsyncServer

A from-scratch, single-binary HTTP server for Linux written in C++17 — no frameworks, no libraries beyond the standard library and the Linux syscall API. It is a non-blocking, event-driven reactor built on edge-triggered epoll, backed by an optional worker thread pool so CPU-bound requests scale across cores. Built in layers, each isolating one systems concept.

Tech: C++17 · Linux · epoll (edge-triggered) · POSIX sockets · Thread pool · EPOLLONESHOT · CMake · ApacheBench

---

Why this exists

Most "web server" projects wrap an existing framework. This one implements the machinery underneath a framework: the accept loop, the event loop, non-blocking I/O with partial reads/writes, connection lifecycle, an HTTP request/response model, and a thread pool with backpressure — then benchmarks it to show the thread pool scaling on CPU-bound work and shedding load gracefully under overload.

The guiding rule throughout: every axis of change becomes an interface; every OS resource becomes an RAII object.

---

Architecture

The server is a single-threaded reactor (one thread owns epoll_wait, accept, and event dispatch) that hands the actual per-request work (recv → parse → run handler → send) to a pool of worker threads. Coordination between the reactor and the workers is done with EPOLLONESHOT (explained below).

            ┌─────────────────────────── reactor thread ──────────────────────────┐
listener ─► │ epoll_wait ─► accept new conns / dispatch ready client fds           │
            └──────────────┬───────────────────────────────────────────────────────┘
                           │ submit([fd]{ handle_client_data(fd); })
                           ▼
                   ┌───────────────┐   task queue (mutex + condition_variable)
                   │  ThreadPool   │   workers: recv → parse → handler → send
                   └───────────────┘

Layers (each maps to a file/interface)

Concern	File	What it isolates
A raw file descriptor	net/socket.hpp	RAII over an fd, move-only (Rule of Five) — the fd is closed exactly once, automatically.
A live connection	net/client_connection.hpp	has-a Socket + input/output buffers + peer IP. Move-only for free.
HTTP model	http/http_request.hpp, http/http_response.hpp	Request-line + query-string parse; response computes its own Content-Length.
Routing	http/handler.hpp	Strategy pattern: abstract Handler, concrete Hello/Time/Big/Compute/NotFound; map<path, unique_ptr<Handler>>. Adding a route never touches the server.
Event mechanism	io/io_multiplexer.hpp	Strategy over the OS event API (add/remove/modify/wait).
poll backend	io/poll_multiplexer.hpp	level-triggered poll() (reactor-only).
epoll backend	io/epoll_multiplexer.hpp	edge-triggered epoll, with EPOLLONESHOT support.
The reactor	server/tcp_server.hpp/.cpp	listener, accept loop, clients_ map, routing, read-drain, write-buffering, load shedding.
Concurrency	concurrency/thread_pool.hpp/.cpp	fixed workers draining a bounded task queue; generic function<void()>.

Swapping the whole event backend from poll to epoll is a one-line change in main.cpp — that's the payoff of the IoMultiplexer interface (Dependency Inversion).

Key systems details

• Edge-triggered epoll. epoll only notifies once per readiness change, so every read loops recv() until EAGAIN (read-drain) and the listener loops accept() until EAGAIN. Miss this and you lose events.
• Non-blocking writes. A response that doesn't fit the socket buffer is parked in a per-connection output buffer; the server arms EPOLLOUT and finishes the write when the socket is writable again, closing only once fully drained.
• EPOLLONESHOT for reactor↔worker handoff. When a client fd is handed to a worker, epoll mutes that fd so no second worker can grab the same connection concurrently. The worker's last action is to either re-arm the fd (modify) or drop it — never touch the connection after that, because the instant it's re-armed another worker may own it.
• Two mutexes, held narrowly. One guards the clients_ map, one guards the epoll one-shot bookkeeping. Neither is ever held across a recv/send syscall.
• RAII everywhere. Sockets close on scope exit; the thread pool joins all workers in its destructor.

---

Endpoints

Route	Handler	Purpose
GET /	HelloHandler	trivial "Hello, World!" — the low-overhead baseline.
GET /time	TimeHandler	current time, formatted thread-safely (localtime_r + strftime).
GET /big	BigHandler	5 MB body — exercises the partial-write / EPOLLOUT path.
GET /compute?n=N	ComputeHandler	CPU-bound: counts primes ≤ N by trial division (deterministic, allocation-free hot loop). The benchmark workload.
anything else	NotFoundHandler	404.

Query strings are parsed into a param(key) map; routing matches on the path before the ?.

---

Build & run

Linux only (epoll). On Windows, use WSL.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
./build/server                 # port 3490, workers = hardware_concurrency(), unbounded queue
./build/server 3490 8          # port 3490, 8 workers
./build/server 3490 8 64       # port 3490, 8 workers, bounded task queue of 64 (enables load shedding)

./server <port> <workers> [max_queue]. max_queue = 0 means unbounded.

Quick check:

curl localhost:3490/                 # Hello, World!
curl "localhost:3490/compute?n=100"  # primes<=100: 25

Build in Release for any benchmarking. A debug -O0 build makes the compute loop ~10× slower and the numbers meaningless.

---

Benchmarking

The headline deliverable. Everything below was measured on a CPU-pinned Linux host with ApacheBench (ab) as the load generator. Harness lives in tests/bench/.

How to read the metrics

Before the numbers, what each column actually measures:

• Concurrency (C) — how many requests are in flight at the same time. The "pressure" dial. ab -c C.
• Workers — threads in the server's pool pulling work off the queue. The thing we're proving scales. Set by the server's launch argument.
• Throughput (RPS) — completed requests ÷ wall-clock seconds. Raw capacity: how much work the server finishes per second.
• Latency p50 / p90 / p99 — sort every request's duration; p50 is the median (typical experience), p99 is the tail (the worst 1% — what users feel at scale, because a page making many backend calls almost always hits the tail at least once). Throughput can look healthy while p99 is terrible, which is exactly why both are reported together.
• Goodput — requests successfully served per second (2xx only), as opposed to throughput which counts every response including errors. The two are equal normally and diverge under overload — that gap is the load-shedding story.

Two structural facts the data illustrates:

• Little's Law (in-flight = throughput × latency): once the server saturates, throughput plateaus and extra concurrency can only make latency rise — the queue just gets longer. You see this directly: past saturation, RPS is flat while latency doubles with every doubling of C.
• Scaling efficiency: perfect scaling means N workers = N× the 1-worker throughput. Reality falls short, and why it falls short depends on the machine — on this host it's the CPU topology (see the results below), not lock contention. Reporting the efficiency and correctly attributing the loss is the point.

Methodology (why the numbers are trustworthy)

1. Release build (-O3) — a debug build's numbers are noise.
2. Workload calibration — n chosen so one /compute request is ~10–15 ms: fast enough for throughput, slow enough to stay genuinely CPU-bound.
3. Hot-path logging removed — std::cout is lock-synchronized, so per-request logging makes workers contend on stdout and distorts scaling.
4. CPU pinning — server pinned to one CPU set, ab to another (taskset), so the load generator can't steal the cores it's supposed to be measuring.
5. No HTTP keep-alive — the server closes after each response, so ab runs without -k: every request is its own connection, matching the model.

Results — throughput scaling on /compute

Peak requests/sec as the worker pool grows (higher is better):

Workers	Peak RPS	Speedup vs. 1 worker	Efficiency
1	89	1.0×	100%
2	169	1.9×	95%
4	314	3.5×	88%
8	551	6.2×	77%

Near-linear to ~4 workers, then diminishing returns. The knee is explained by the CPU, not the software: this host is an Intel Core Ultra 7 155U — a hybrid part with only 2 performance (P) cores (4 threads via SMT) plus 8 efficiency (E) cores and 2 low-power E-cores. Workers 1–4 land on fast P-threads and scale near-linearly; workers 5–8 spill onto slower E-cores (and shared SMT siblings), each adding less than a full core's throughput — hence 6.2×, not 8×.

A note on the queue mutex: the ThreadPool uses a single mutex-guarded task queue, which is a scaling limit in principle (Amdahl's serial section). But at this scale it is not the bottleneck — the queue is locked only a few hundred times/sec for microseconds, far too little to explain a 23% loss. Isolating the mutex as the true limit would require many more homogeneous cores (so cores stop being the ceiling) or a much cheaper handler (so the submit/pop rate is high enough to make the lock hot). This benchmark doesn't claim to have done that.

Results — tail latency under load (8 workers, /compute)

Concurrency	RPS	p50	p99
8	397	20 ms	25 ms
64	542	117 ms	124 ms
256	541	459 ms	1484 ms

Throughput saturates ~540 RPS; beyond that, concurrency turns straight into latency (Little's Law).

Results — goodput & load shedding (8 workers, bounded queue = 64)

Offered load pushed far past capacity. When the queue is full, further requests get an immediate 503 Service Unavailable instead of piling up:

Concurrency	Offered (RPS)	Goodput (RPS)	Shed (503)	p99
8	133	133	0%	81 ms
64	6136	~331	~95%	118 ms
256	4638	~139	~97%	115 ms

The server is offered 6,000+ RPS — over 10× its capacity — and instead of collapsing (unbounded queue → memory growth → everything times out → zero goodput), it sheds load and keeps p99 bounded near ~118 ms. That is backpressure: reject excess work early so accepted work stays fast. This is what real load balancers and rate limiters do.

Baseline — trivial endpoint /

Concurrency Level:      100
Requests per second:    12852.53 [#/sec] (mean)
Failed requests:        0

12.8k RPS on near-zero work is a control: it isolates the reactor→pool dispatch overhead from the handler cost. It also shows the honest flip side — on trivial work the thread pool is roughly net-neutral (syscall overhead dominates, not the handler); the pool only earns its keep on the CPU-bound endpoint.

Reproducing

# start a Release server, then:
N=<calibrated> bash tests/bench/sweep.sh      # full worker sweep → results.csv + results_goodput.csv

See tests/bench/README.md for the harness details and tunables (WORKERS_LIST, SERVER_CPUS, AB_CPUS, REQUESTS).

---

Tests

Not wired into CMake — run on demand (see tests/README.md).

• Unit — tests/unit/: HTTP request parsing, thread-pool basics, bounded-queue rejection.
• Integration — tests/integration/smoke.sh: starts the server, curls every route.
• Bench — tests/bench/: the throughput / latency / goodput harness above.

g++ -std=c++17 -pthread -Iinclude tests/unit/test_pool.cpp \
    src/concurrency/thread_pool.cpp -o test_pool && ./test_pool

---

Known limitations / future work

• Single task-queue mutex is a scaling limit in principle (one lock every submit/pop). It isn't the bottleneck on the benchmark host — the CPU topology is (see the scaling results) — but it would bite on a many-core homogeneous machine or a higher task-submission rate. Per-worker queues with work-stealing would relax it.
• No HTTP keep-alive — one connection per request. Adding it needs explicit EPOLLOUT disarm handling; it's the most instructive next layer.
• No request-body / Content-Length buffering — parameters go via the query string, so no real POST yet.
• Single reactor thread — one thread owns all epoll_wait + accept + dispatch. On CPU-bound work the workers are the limit so this is invisible, but on cheap/trivial requests (or very high connection churn) that one thread saturates a single core and becomes the ceiling — no amount of extra workers helps, because they're all fed by one thread. This is the real reason the pool is net-neutral on the trivial / baseline. The fix is a reactor-per-core design: N reactor threads, each with its own listening socket via SO_REUSEPORT (the kernel hashes connections across them), so accepts fan out over cores with no shared listen socket and no thundering herd. This is how nginx/envoy/HAProxy scale accepts. EPOLLEXCLUSIVE is a lighter alternative (shared socket, kernel wakes only one waiter per event).
• No deadline-based shedding — the bounded queue sheds at enqueue (cheap, early), but a request already queued is never dropped even if it's waited past any reasonable client timeout. A per-request deadline check at dequeue would let workers skip already-doomed work under sustained overload.

Hardening gaps

Built as a systems-design study, not a hardened edge server. The known holes, roughly by severity:

• Algorithmic DoS — /compute?n=... has no upper bound on n; one request can peg a core for minutes. Needs an n cap + 400 above it.
• No read/idle timeout (Slowloris) — incomplete connections stay armed forever and exhaust file descriptors. Needs per-connection deadlines (timerfd / timing wheel) + a max-connection cap.
• SIGPIPE — MSG_NOSIGNAL is only on the shed path; any other send() to a dead peer can kill the process. Ignore SIGPIPE process-wide at startup.
• No graceful shutdown — no SIGINT/SIGTERM handler; drain-then-join is needed for a clean exit.
• Concurrency correctness is reasoned, not proven — wants a -fsanitize=thread run under load before claiming race-free.
• EMFILE on accept — no reserve-fd trick, so the edge-triggered listener can spin under fd exhaustion.

See docs/DESIGN.md for the narrative why behind each design decision.