fix(demos): abort the scenario load on a failed write, and pull through client 1.13.1 - #1318
Merged
Merged
Conversation
A showcase run that lost 25 of 383 events to rate limiting still exited green, then closed 14 historical months and materialized the graph over the gap. Both loops swallowed the failure into a counter and carried on. A dropped event is not just a missing row. bill_payment posts two linked events, so a failure between them strands an AP balance with no discharge; a dropped invoice leaves its later payment with no obligation to discharge. A dropped agent is quieter still — agent_lookup.get() returns None and every event for that counterparty posts with no agent_id, silently removing the REA linkage the episode exists to show. Both now raise, naming how far the load got. The genuine data skip in the event loop — a transaction whose lines have no CoA element — stays non-fatal but is counted separately, since folding it together with API failures under "missing elements or errors" is what made this invisible.
robosystems-client 1.13.1 replays a rate-limited request instead of surfacing the 429. The rejection is raised before the endpoint handler runs, so the call had no effect and the replay is safe even for a POST with no idempotency key. This is the half that reaches the showcase demos: the demo workflow clones this repo at ROBOSYSTEMS_REF and runs `uv sync` in the checkout, so the episode runner's SDK comes from this lockfile. The scenario load issues ~500 extensions writes in ~108s against a 300/min budget, which is where the dropped events came from. Floor and lock only.
jfrench9
added a commit
that referenced
this pull request
Aug 31, 2026
The second copy of the defect #1318 fixed in the scenario runner. This is the `just demo-roboledger` path, which loads Cascade Advisory through its own duplicate of the event and agent loops — both of which swallowed a failed write into a counter and carried on. The consequences are the ones #1318 spelled out. `bill_payment` posts two linked events, so a failure between them strands an AP balance with no discharge; a dropped invoice leaves its later payment with no obligation to discharge; a dropped agent is quieter still, because `agent_lookup.get()` then returns None and every event for that counterparty posts with no `agent_id`. The run would then close periods and materialize over the gap and still exit 0. Locally this never fires — `RATE_LIMIT_ENABLED=false`, so nothing is rejected — but the demo also runs against a deployed API via DEMO_API_URL, which is exactly where the showcase episode lost 25 of 383 events while reporting success. The genuine data skip (a transaction whose lines have no CoA element) stays non-fatal and is now counted separately; folding it together with API failures under "missing elements or errors" is what made the original invisible.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A showcase demo run lost 25 of 383 business events to
429 Rate limit exceeded for extensions write operationsand still exited green, then closed 14 historical months and materialized the graph over the gap. Two independent problems, both fixed here.The scenario load issues ~500 extensions writes in ~108s against a 300/min budget — 92% of budget with zero headroom, so ordinary jitter converted straight into dropped rows. That half is fixed in the SDK (robosystems-client 1.13.1, pulled through below): a 429 is raised by a request dependency before the endpoint handler runs, so the rejected call had no effect and is safe to replay.
The other half is that the runner treated an API failure as a data skip. Even with retries in place, a load that does fail should not go on to close periods over an incomplete ledger.
Changes
examples/_scenario/runner.py— the event and agent loops now raise instead of counting failures intoskipped.A dropped event is not just a missing row.
bill_paymentposts two linked events, so a failure between them strands an AP balance with no discharge; a dropped invoice leaves its later payment with no obligation to discharge. A dropped agent is quieter still —agent_lookup.get()returnsNoneand every event for that counterparty posts with noagent_id, silently removing the REA linkage the episode exists to demonstrate. Both now abort, naming how far the load got.The genuine data skip in the event loop — a transaction whose lines have no CoA element — stays non-fatal, but is counted and reported separately. Folding it together with API failures under
"Skipped N events (missing elements or errors)"is what made this invisible in the first place.pyproject.toml/uv.lock— dev client floor 1.12.2 → 1.13.1, and locked. This is the half that reaches the demos: the demo workflow clones this repo atROBOSYSTEMS_REFand runsuv syncin that checkout, so the episode runner's SDK comes from this lockfile rather than the demo repo's. Floor and lock only.Testing
just test— 14,218 passed, 42 skipped against the new client.just test-code— ruff, format and basedpyright clean.LedgerClient._get_client()returns aRetryingClient, the GraphQL client carries the retry budget, and aRetry-After: 60yields a 0.57s backoff rather than a 60s sleep (the limiter is a sliding window, so the header reports the whole window).Related
chore/client-1-13-1.Not included
examples/roboledger_demo/main.py(the localjust demo-roboledgerpath) has the identical swallow in its own event and agent loops. Left out to keep this scoped to the runner the demo workflow actually uses — happy to fold it in if you'd rather fix both at once.