Skip to content

StirrupAgentRunner's default model id never matches a price-table key, so est_cost silently returns None #522

Description

@AmirF194

Background

_estimate_cost (src/evaluation/metrics.py) looks up a model's per-token
rate in _PRICE_PER_1M after normalizing the model id through
_normalize_model. For StirrupAgentRunner's own default model,
watsonx/meta-llama/llama-4-maverick-17b-128e-instruct-fp8
(src/agent/stirrup_agent/runner.py), the normalized id never matches the
existing llama-4-maverick table entry, so _estimate_cost returns None
and any report built on it (including aggregate_ops's
est_input_cost_usd_total/est_output_cost_usd_total, added in #518) is
silently missing a dollar figure for runs using the default model.

src/evaluation/tests/test_metrics.py only exercises single-segment ids
(granite-13b, one date-suffix case), so this gap isn't caught by the
existing suite.

Steps to Reproduce

import sys
sys.path.insert(0, "src")
from evaluation import metrics

default_model = "watsonx/meta-llama/llama-4-maverick-17b-128e-instruct-fp8"
print("normalized:", metrics._normalize_model(default_model))
print("cost:", metrics._estimate_cost(default_model, 1000, 1000))
print("control (bare table key):", metrics._estimate_cost("llama-4-maverick", 1000, 1000))

Observed

normalized: llama-4-maverick-17b-128e-instruct-fp8
cost: None
control (bare table key): 0.00112

Expected

llama-4-maverick-17b-128e-instruct-fp8 resolves to the existing
llama-4-maverick entry, same as the control case.

Environment

  • Repo HEAD: 3a7bd23c7421f6ec3c0f215f018d7a8c2bcd8e37
  • Python 3.12.14, python:3.12-slim Docker image, pydantic from PyPI

Reproduced end to end in a clean container against current main, running
the real module (not a mock). Happy to send a PR if useful.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions