Skip to content

fix(demos): abort the scenario load on a failed write, and pull through client 1.13.1 - #1318

Merged
jfrench9 merged 2 commits into
mainfrom
bugfix/demo-abort-on-dropped-events
Aug 31, 2026
Merged

fix(demos): abort the scenario load on a failed write, and pull through client 1.13.1#1318
jfrench9 merged 2 commits into
mainfrom
bugfix/demo-abort-on-dropped-events

Conversation

@jfrench9

Copy link
Copy Markdown
Member

Summary

A showcase demo run lost 25 of 383 business events to 429 Rate limit exceeded for extensions write operations and still exited green, then closed 14 historical months and materialized the graph over the gap. Two independent problems, both fixed here.

The scenario load issues ~500 extensions writes in ~108s against a 300/min budget — 92% of budget with zero headroom, so ordinary jitter converted straight into dropped rows. That half is fixed in the SDK (robosystems-client 1.13.1, pulled through below): a 429 is raised by a request dependency before the endpoint handler runs, so the rejected call had no effect and is safe to replay.

The other half is that the runner treated an API failure as a data skip. Even with retries in place, a load that does fail should not go on to close periods over an incomplete ledger.

Changes

examples/_scenario/runner.py — the event and agent loops now raise instead of counting failures into skipped.

A dropped event is not just a missing row. bill_payment posts two linked events, so a failure between them strands an AP balance with no discharge; a dropped invoice leaves its later payment with no obligation to discharge. A dropped agent is quieter still — agent_lookup.get() returns None and every event for that counterparty posts with no agent_id, silently removing the REA linkage the episode exists to demonstrate. Both now abort, naming how far the load got.

The genuine data skip in the event loop — a transaction whose lines have no CoA element — stays non-fatal, but is counted and reported separately. Folding it together with API failures under "Skipped N events (missing elements or errors)" is what made this invisible in the first place.

pyproject.toml / uv.lock — dev client floor 1.12.2 → 1.13.1, and locked. This is the half that reaches the demos: the demo workflow clones this repo at ROBOSYSTEMS_REF and runs uv sync in that checkout, so the episode runner's SDK comes from this lockfile rather than the demo repo's. Floor and lock only.

Testing

  • just test14,218 passed, 42 skipped against the new client.
  • just test-code — ruff, format and basedpyright clean.
  • Verified against the published 1.13.1 wheel rather than the source: LedgerClient._get_client() returns a RetryingClient, the GraphQL client carries the retry budget, and a Retry-After: 60 yields a 0.57s backoff rather than a 60s sleep (the limiter is a sliding window, so the header reports the whole window).
  • The failing episode has not been re-run end-to-end — that needs a demo workflow dispatch against a fresh graph.

Related

Not included

examples/roboledger_demo/main.py (the local just demo-roboledger path) has the identical swallow in its own event and agent loops. Left out to keep this scoped to the runner the demo workflow actually uses — happy to fold it in if you'd rather fix both at once.

A showcase run that lost 25 of 383 events to rate limiting still exited
green, then closed 14 historical months and materialized the graph over
the gap. Both loops swallowed the failure into a counter and carried on.

A dropped event is not just a missing row. bill_payment posts two linked
events, so a failure between them strands an AP balance with no
discharge; a dropped invoice leaves its later payment with no obligation
to discharge. A dropped agent is quieter still — agent_lookup.get()
returns None and every event for that counterparty posts with no
agent_id, silently removing the REA linkage the episode exists to show.

Both now raise, naming how far the load got. The genuine data skip in
the event loop — a transaction whose lines have no CoA element — stays
non-fatal but is counted separately, since folding it together with API
failures under "missing elements or errors" is what made this invisible.
robosystems-client 1.13.1 replays a rate-limited request instead of
surfacing the 429. The rejection is raised before the endpoint handler
runs, so the call had no effect and the replay is safe even for a POST
with no idempotency key.

This is the half that reaches the showcase demos: the demo workflow
clones this repo at ROBOSYSTEMS_REF and runs `uv sync` in the checkout,
so the episode runner's SDK comes from this lockfile. The scenario load
issues ~500 extensions writes in ~108s against a 300/min budget, which
is where the dropped events came from.

Floor and lock only.
@jfrench9
jfrench9 merged commit 72711aa into main Aug 31, 2026
7 checks passed
@jfrench9
jfrench9 deleted the bugfix/demo-abort-on-dropped-events branch August 31, 2026 22:05
jfrench9 added a commit that referenced this pull request Aug 31, 2026
The second copy of the defect #1318 fixed in the scenario runner. This
is the `just demo-roboledger` path, which loads Cascade Advisory through
its own duplicate of the event and agent loops — both of which swallowed
a failed write into a counter and carried on.

The consequences are the ones #1318 spelled out. `bill_payment` posts
two linked events, so a failure between them strands an AP balance with
no discharge; a dropped invoice leaves its later payment with no
obligation to discharge; a dropped agent is quieter still, because
`agent_lookup.get()` then returns None and every event for that
counterparty posts with no `agent_id`. The run would then close periods
and materialize over the gap and still exit 0.

Locally this never fires — `RATE_LIMIT_ENABLED=false`, so nothing is
rejected — but the demo also runs against a deployed API via
DEMO_API_URL, which is exactly where the showcase episode lost 25 of 383
events while reporting success.

The genuine data skip (a transaction whose lines have no CoA element)
stays non-fatal and is now counted separately; folding it together with
API failures under "missing elements or errors" is what made the
original invisible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant