Background
_estimate_cost (src/evaluation/metrics.py) looks up a model's per-token
rate in _PRICE_PER_1M after normalizing the model id through
_normalize_model. For StirrupAgentRunner's own default model,
watsonx/meta-llama/llama-4-maverick-17b-128e-instruct-fp8
(src/agent/stirrup_agent/runner.py), the normalized id never matches the
existing llama-4-maverick table entry, so _estimate_cost returns None
and any report built on it (including aggregate_ops's
est_input_cost_usd_total/est_output_cost_usd_total, added in #518) is
silently missing a dollar figure for runs using the default model.
src/evaluation/tests/test_metrics.py only exercises single-segment ids
(granite-13b, one date-suffix case), so this gap isn't caught by the
existing suite.
Steps to Reproduce
import sys
sys.path.insert(0, "src")
from evaluation import metrics
default_model = "watsonx/meta-llama/llama-4-maverick-17b-128e-instruct-fp8"
print("normalized:", metrics._normalize_model(default_model))
print("cost:", metrics._estimate_cost(default_model, 1000, 1000))
print("control (bare table key):", metrics._estimate_cost("llama-4-maverick", 1000, 1000))
Observed
normalized: llama-4-maverick-17b-128e-instruct-fp8
cost: None
control (bare table key): 0.00112
Expected
llama-4-maverick-17b-128e-instruct-fp8 resolves to the existing
llama-4-maverick entry, same as the control case.
Environment
- Repo HEAD:
3a7bd23c7421f6ec3c0f215f018d7a8c2bcd8e37
- Python 3.12.14,
python:3.12-slim Docker image, pydantic from PyPI
Reproduced end to end in a clean container against current main, running
the real module (not a mock). Happy to send a PR if useful.
Background
_estimate_cost(src/evaluation/metrics.py) looks up a model's per-tokenrate in
_PRICE_PER_1Mafter normalizing the model id through_normalize_model. ForStirrupAgentRunner's own default model,watsonx/meta-llama/llama-4-maverick-17b-128e-instruct-fp8(
src/agent/stirrup_agent/runner.py), the normalized id never matches theexisting
llama-4-mavericktable entry, so_estimate_costreturnsNoneand any report built on it (including
aggregate_ops'sest_input_cost_usd_total/est_output_cost_usd_total, added in #518) issilently missing a dollar figure for runs using the default model.
src/evaluation/tests/test_metrics.pyonly exercises single-segment ids(
granite-13b, one date-suffix case), so this gap isn't caught by theexisting suite.
Steps to Reproduce
Observed
Expected
llama-4-maverick-17b-128e-instruct-fp8resolves to the existingllama-4-maverickentry, same as the control case.Environment
3a7bd23c7421f6ec3c0f215f018d7a8c2bcd8e37python:3.12-slimDocker image,pydanticfrom PyPIReproduced end to end in a clean container against current
main, runningthe real module (not a mock). Happy to send a PR if useful.