RouteWise

Latency–Cost Optimization for Multi-Provider LLM Routing
_{Learn from recent outcomes. Route each request. Hedge selectively.}

Developed and maintained by the Harvard MadSys Lab at Harvard SEAS.

Documentation · English API · 中文 API

RouteWise is a dependency-free Python library for cost-aware, latency-optimized routing across multiple LLM API providers. Applications supply provider prices, dispatch the returned attempt, and report outcomes so RouteWise can learn from them.

The main branch contains the public library, its documentation, examples, tests, and release tooling. The EuroSys paper artifact is maintained separately on the eurosys27-ae branch so that its simulator, experiments, data preparation, and reproduction environment remain self-contained.

Package name: The PyPI project routewise is an unaffiliated, incompatible project. Install the llm-routewise distribution and import the llm_routewise package for HarvardMadSys RouteWise.

News

• August 2026: Our paper, RouteWise: Latency–Cost Optimization for Multi-Provider LLM Routing, has been accepted to EuroSys 2027! 🎉

Installation

RouteWise requires Python 3.10 or later. The published wheel has no runtime dependencies.

python -m pip install llm-routewise

For repository development:

git clone https://github.com/HarvardMadSys/RouteWise.git
cd RouteWise
uv sync --locked

Quickstart

import llm_routewise as rw

router = rw.Router(
    [
        rw.Provider("fast", price_in=3.0, price_out=15.0, price_cached=0.30),
        rw.Provider("cheap", price_in=0.15, price_out=0.60),
    ],
    alpha=0.25,  # Cost budget: 0 = cheapest; 1 = full range for latency optimization.
)

decision = router.route(
    input_tokens=800,
    estimated_cached_tokens=600,  # Prompt prefix you expect to hit cache.
)
response = call_your_provider(decision.provider)
decision.completed(
    ttft_ms=response.ttft_ms,
    output_tokens=response.output_tokens,
    cached_tokens=response.cached_tokens,
)

If your application already predicts completion length, pass that point estimate with the request. Omit it to use RouteWise's internal online estimate.

predicted_tokens = predict_output_tokens(prompt)
decision = router.route(
    input_tokens=800,
    estimated_output_tokens=predicted_tokens,
)

The estimate affects route and hedge cost calculations only; it is not actual usage. On completion, report the adopted attempt's actual output_tokens and cached_tokens (or an explicit cost_usd) for billing. Positive actual output tokens also update RouteWise's internal estimator.

Router computes decisions but performs no network I/O and does not read API keys. Your application owns provider clients, credentials, and dispatch. Read the English API reference or 中文 API 参考 for the full contract.

Two offline examples use only the public API: a single decision, and a full request lifecycle showing dispatch, failure reporting, and how outcomes change routing. Run them with:

uv run python examples/basic.py
uv run python examples/simple_router.py

Repository Development

Install the development tools and run the complete library checks:

uv sync --locked
uv run ruff check .
uv run pytest -q
uv run python examples/basic.py
uv run python examples/simple_router.py
uv run python -m build --wheel
uv run python scripts/check_wheel.py dist/*.whl

Maintainer docs live in docs/maintainers/.

Citation

If you use RouteWise in your research, please cite our paper:

@inproceedings{tian2027routewise,
  title     = {{RouteWise}: Latency--Cost Optimization for Multi-Provider LLM Routing},
  author    = {Muxin Tian and Haoran Ni and Yiyan Zhai and Yangsun Park and Juncheng Yang},
  booktitle = {Proceedings of the 22nd European Conference on Computer Systems (EuroSys '27)},
  year      = {2027}
}

License

MIT. See the license.