Make the live-frame probe survive a transient source-read fault

[hernanwilkinson]

Three flakes, one mechanism

Three one-off failures in one week, all in frame-sensitive tests — WarningLocationTestCase (flagged in #594), FrameEqualityTestCase (#623), TracebackTestCase (#626) — all ValueError: call stack is not deep enough shapes, none reproducible: every one passed in isolation and on every re-run.

The mechanism turned out to be half-documented at the site already. ___isGeneratedPythonMethod___ tells compiled Python apart from Grail's own runtime by probing the method source for the ___curPos___ marker, and an earlier fix stopped a failed probe from being cached — its comment even records that injecting a single false reproduces the exact failure shape. What remained was the failure's effect on the current walk: a transient fault answered false once, the frame silently dropped, the chain came up short, and the next walk re-probed fine. That is precisely a once-per-week flake — and the source string is the one read in the whole walk that goes back to the repository, the read that can fault under four concurrent shard workers. Everything else reads the in-memory method object.

Three changes, each doing one thing

1. A fast path that cannot fault. The method's own debugInfo (argsAndTemps — in memory, off the stack triple) lists a module-level def's ___curPos___ as a method temp: conclusive without touching source. Not a replacement — a def whose body compiles into an inner block declares the marker block-side, invisible to method-level debugInfo (_py_warnings: 11 of 46 methods), so a miss falls through. I checked argsAndTemps' kernel implementation to confirm it decodes in-method data rather than parsing source.
2. The source probe retries once. The page read that just faulted is the likeliest read to succeed a moment later.
3. A double fault leaves a breadcrumb (#GrailPyProbeFailures) so a run that still flakes says why — and stays uncached, as before: a real false is a property of the method, a failed probe is a property of the moment.

Testing something you cannot schedule

A real page fault cannot be provoked on demand, so #GrailPyProbeFailCount is a test seam injecting that many consecutive simulated faults, consumed inside the probe's own failure path. LiveFrameProbeResilienceTestCase (4/4) proves:

• one injected fault → answer still true, no breadcrumb, seam consumed — the retry did the work;
• two faults → false for that walk only, breadcrumb left, and the very next probe answers true (failure never cached);
• the temps fast path never reaches the source probe at all (faults injected, counter untouched);
• both baseline classifications hold.

Verification

probe answers	unchanged, 46/46 vs source-truth on _py_warnings (the block-temp-heavy module)
flake-family classes	WarningLocation + FrameEquality + Traceback looped ×5, all clean
curated suite	0 regressions
SUnit (4.0)	5658 + 127 + 22 run, all passed
SUnit (3.7.5)	2818 + 2840 across CI's shard groups, all passed
fixture gate	2600 OK, 33 XFAIL

Honest limits: the fault itself was never observed directly — the case rests on the site's own prior measurement ("injecting a single false" reproduces the shape), on the source read being the only repository-bound read in the walk, and on the failure signature matching all three sightings. If a one-off still appears, the breadcrumb now turns it from a footnote into a diagnosis.

🤖 Generated with Claude Code

[hernanwilkinson]

Verifying the stacked #656-to-be (the function-type-name PR) produced the A/B this PR's "honest limits" section said was missing:

its install recompiles Object.gs, and in the post-install window TracebackTestCase>>testLiveFramesAndGetframe failed 8-of-8 deterministically without this fix, passed 8-of-8 with it borrowed in, and passed without it once the cache warmed. So the fault is better described as a deterministic cold-read fault immediately after install.sh — first access faults the page in, the retry reads it resident — than as a rare page fault.

Two follow-ons, both in the stacked PR: the same recipe applied to the locals reader (___liveFrameContentsByLevel___ let one flaky level truncate every level after it — found by this PR's own evidence-carrying assertion, 'the raising frame reported no locals'), and an honest residual: a probabilistic cold-window mode survives the retries, observed once each on 4.0 and 3.7.5, never twice, never warm. The breadcrumbs are doing their job.