FailureAtlas

A Taxonomy of Failure Modes in Multi-Provider LLM Serving Infrastructure

Multi-provider LLM gateways (e.g., LiteLLM, Portkey, OpenRouter) have become critical production infrastructure. Yet, the specific failure modes that emerge at the architectural seam between application code and upstream LLM providers remain scattered across issue trackers and post-mortems. FailureAtlas is an open, extensible taxonomy and evidence-grounded catalog of these failure modes.

Paper: Failure Atlas Paper

The Core Finding: Silent Failures Dominate

The principal finding of FailureAtlas is that the most operationally severe failures in multi-provider LLM architectures are silent.

Traditional failures (e.g., timeouts, deadlocks) trigger HTTP 5xx errors and are caught by standard APM tools like Datadog or Prometheus (categorised as Loud). However, LLM infrastructure frequently experiences failures that return HTTP 200 OK while quietly corrupting the semantic payload such as context bleeding across tenant boundaries or tool-call index collisions (categorised as Silent). These require semantic observability to detect.

---

Repository Structure

The repository is organized to maintain a clean separation between the live dataset, reproduction code, and academic reporting.

failure-atlas/
├── catalog/          # The core dataset: Structured YAML files for all documented failures

├── reproductions/    # Executable reproduction scripts (e.g., Race Condition, Retry Storm)
├── paper/            # LaTeX source code and figures for the FailureAtlas manuscript
├── scripts/          # Utility scripts for rendering the grid and fetching issues
└── notes/            # Raw research notes and scraped Markdown records

---

The Taxonomy

FailureAtlas organises infrastructure failures along two orthogonal axes:

1. Layer (Origin of Failure):
   • Network/Transport: Connection pools, socket timeouts, sync blocking async.
   • Streaming/Protocol: SSE chunking, malformed JSON streams.
   • State/Session: KV-cache, conversation history mutation, concurrency races.
   • Model Behavior: Instruction-following degradation, output distribution drift.
   • Governance/Cost: Retry storms, rate-limit deadlocks, silent model substitution.
2. Detectability:
   • Loud: Caught by traditional HTTP-level infrastructure monitoring.
   • Silent: Invisible to traditional APM; requires semantic evaluation to detect.

---

Running Reproductions

Where possible, catalog entries are accompanied by standalone reproduction harnesses. These allow practitioners to safely simulate the exact conditions under which a gateway proxy or client library fails.

For example, to simulate the high-concurrency race condition:

cd reproductions/race-condition
python repro.py

(Check the README.md inside reproductions/ for setup instructions depending on the specific failure mode).

---

Contributing to the Catalog

FailureAtlas is designed to be an extensible, community-curated repository similar to CVE databases or awesome-lists. The rapid evolution of stateful LLM proxies guarantees that new failure modes will continually emerge.

If you have discovered or debugged an interesting infrastructure-level failure in an LLM application, we welcome your contribution!

How to add an entry

1. Copy an existing YAML schema from catalog/.
2. Ensure the failure is mechanistically explainable and backed by evidence (a GitHub issue, post-mortem URL, or first-hand stress test).
3. Submit a Pull Request.

---

Built for rigorous engineering in the generative AI era.