Rebuild rl/: restore the proven shaped reward, lean campaign machinery - #36
Merged
Conversation
Synthesis of the five-way study (old private library, current rl/ tree, paper commitments, launch-night archaeology, dataset sealing). Decides: port the proven iou_n/v3-b shaped reward onto the sealed depth-v1 curriculum, one training path in track_a over common, two matched runs (L0-SFT->RL and base-RL L0->L4), delete the studies/ceremony apparatus. Commitment: Train the paper's two Qwen arms with the ported v3-b shaped reward (iou_n) on the sealed depth-v1 curriculum, one lean path in rl/track_a; the studies/approval ceremony and Inkling machinery are removed rather than maintained. Type: decision Scope: rl/ Caused: five-way study of PixCell-running/rl, rl/, arxiv/, git history, dataset/ (spec: docs/superpowers/specs/2026-07-28-rl-clean-rebuild-design.md)
The zero point of the shaped reward: what a solid rectangle filling the target's calibrated ink bbox scores, plus the boundary chamfer of that rectangle and the footprint diagonal. Ported from the private library's reward geometry; computed on the evaluator's own calibration and cached per target_image_sha256 (the dataset is frozen). Honors: 7314334
Symmetric mean boundary distance (um, mean-axis scale) between the candidate render and the target, recorded as a diagnostic on every OK result. Empty-mask chamfer is stored as None, never inf, so the strict JSON diagnostics contract is preserved. This is the measurement behind the shaped reward's teacher term. Honors: 7314334
The private library's proven training signal, constants byte-identical: 0.05 executable floor + 0.40*iou_n + 0.15*dice_n + 0.40*teacher_nb, with raw scoring on rectangle-degenerate targets (iou_rect >= 0.95). iou_n normalizes against the bbox-rectangle fill so area-filling earns nothing and one reward scale means the same thing on every task; the boundary-chamfer teacher pays for count progress IoU cannot see. SOURCE_REJECTED stays 0.0 (ship-mode purity semantics). Honors: 7314334
A/B stay at the no-think 4096 operating point; C/D repeat the +-L0-SFT pair at the thinking 60k operating point RC-H05 measured at 0.218 zero-shot. D's SFT is sanity-gated against the renderer-collapse precedent; smoke runs price both operating points before full launch.
TrackAEnv computes the task's cached rectangle baseline for the shaped policy, logs the iou_n/teacher_nb/degenerate breakdown into step metrics, and the RL builder defaults change to reward_policy shaped_v3b with 8 rollouts per group (the old lineage's group size; G=4 halves the within-group signal GRPO feeds on). Honors: 7314334
Deletes rl/studies (three baseline/campaign studies + two extensions), rl/evaluation (the zero-shot harness and its W&B mirror), rl/track_b (the Inkling TML shim), rl/track_c (an empty reservation), and track_a's recipe-lock/mastery/promotion surface. The launch-night record shows this machinery aborting its own runs three times and relaxing its own gates; none of it touches the learning signal. Level helpers move to curriculum.py; preflight keeps the real gates (frozen-release digests, git state, whitelist crosscheck, token audit, ground-truth positive control now scored through shaped_v3b) decoupled from the recipe lock; train_sft/train_rl become lean single-stage entry points with prompt identity between SFT and the RL that resumes it. Honors: 7314334
rl/track_a/run.py sequences each campaign (SFT stage where the arm has one, then staged RL L0..L4) as per-stage subprocesses chained on final sampler weights, resumable by re-running the same command. One deterministic 80-task probe (16 per level, sorted opaque ids from depth/validation) is evaluated at campaign start — the base-model baseline; no separate baselining studies — and after every stage, so every point on the curve is comparable. A and D gate RL behind the post-SFT probe beating the base probe on L0; D falls back to RL-from-base on failure. Smoke mode prices an operating point with 2 GRPO steps before anything full-scale launches. Honors: 7314334
The dataset's model image is the max-visibility view (per-axis magnified, not physical aspect). Measured over depth/train: 50% of rows exceed 1.6x aspect distortion, p90 = 5.0x, and 78% of L4 exceeds the threshold (max 31.6x). Above 1.6x the prompt now states the per-axis render scale, computed only from the attached image's ink bbox and the stated footprint — the same honest-input note the private library shipped for the same reason (its spec: 'image plus scale is the task definition'). Below the threshold the prompt is unchanged. Commitment: The direct-reconstruction prompt is contract v3: it discloses per-axis render scale above 1.6x anisotropy, computed from model-visible inputs only. Type: decision Scope: rl/common/prompt.py Caused: anisotropy measurement over depth/train (scratchpad/anisotropy.json; 50% of rows > 1.6x, L4 78% > 1.6x)
rl/README.md and track_a/README.md now describe exactly what exists: the shaped_v3b reward contract, the four-run campaign, the fixed probe protocol, and the preflight gate. Spec updated with the measured anisotropy decision. Honors: 7314334
The cookbook auto-resumes from checkpoints.jsonl (state + optimizer), so an interrupted paid stage continues from its last checkpoint when the runner re-executes it; a pre-created dir with only a console log proceeds fresh.
tinker 0.22.7 defaults to a reqwest/rustls backend whose bundled trust store rejects the Google Trust Services WE1 chain this host receives for tinker.thinkingmachines.dev (UnknownIssuer), while httpx/certifi and curl validate it in 0.2s. The SDK's kill-switch for the transport is server-side only, so an auth bootstrap that dies on pyqwest can never be told to stop using it. Entry points now swap the transport factory for httpx's default before any client is constructed; verified: ServiceClient connects in 1.8s with the patch.
Two launch-night fixes: (1) Tinker's load_weights rejects sampler- weights checkpoints — stages chain on the record's state_path (weights/...) while probes keep sampler paths. (2) Four campaigns share one Docker daemon; a transient container fault after a single retry was aborting whole stages (invalid exit status during peak container churn). The evaluator now retries infrastructure faults three times and per-run worker count is tunable via PIXCELL_EVAL_WORKERS; persistent faults still abort — no biased zeros either way.
Three decisions, code-live since launch night, landed together once signing recovered: 1) Evaluation tier: training-time candidate execution uses the AST-gate + rlimits subprocess tier; the ceremony-era Docker tier caused four operational failure classes in one night and is now opt-in (PIXCELL_REQUIRE_ISOLATION=1) for record evals. Eval workers and wall-timeout are env knobs (PIXCELL_EVAL_WORKERS/_TIMEOUT) so shared-host contention slows measurements instead of falsifying them. 2) Depth variants: --steps-per-level (-sN name suffix) and the legacy rl-l0 extension fork (--fork-l0-from) behind the 30-vs-60 ablation. 3) Branch campaigns: RunPlans generalize to per-level SFT insertions (sft_levels/sft_epochs), a fixed backward-replay share (replay_groups_per_step; the mixed arm's 50% blend), and prefix forking (--fork-from/--fork-through) seeding completed stages + the fork-point probe as gate reference. Level-aware gates: absolute rule for fresh lineages, relative rule (no exec loss beyond slack, IoU gain required) for mid-lineage insertions, failure = skip SFT and continue. New plans: E (A-fork -> L1-SFT gated -> RL), F (A-fork -> 50% backward mix), G (full ladder, held until E). train_sft gains load_checkpoint_path. Commitment: Training-time candidate execution uses the AST-gate + rlimits subprocess tier; Docker isolation is opt-in for record evaluations, not required for training. Type: decision Scope: rl/ Caused: launch-night failure analysis (four Docker-tier failure classes vs zero incidents on the subprocess tier across the private library's full campaign)
User re-scoped Inkling in for one experiment the record shows was designed twice but never run: GRPO under the shaped reward. The TML shim returns exactly as deleted (with its tests), the renderer branch and an explicit effort knob return to the RL entry point, and the tml-renderers pin returns to the stack. The arm is RL-only per the five-times-replicated no-SFT-on-capable-models law; L4-only with the standard replay mix, bounded steps, canaries every 5. (Unsigned by user authorization — overnight session, 1Password locked.) Commitment: Inkling runs RL-only under the shaped reward as a bounded L4 arm; no SFT serialization ever reaches it. Type: decision Scope: rl/ Caused: user direction 2026-07-29 + the untried-experiment gap (old ship config never launched; ceremony campaign died at one step under raw IoU) Supersedes: the Inkling-out-of-scope aspect of the ceremony-removal decision
harvest.py samples from a trained checkpoint over TRAIN tasks only, verifies every completion with the deterministic evaluator, and keeps gate-passing winners (iou_n >= 0.5, degenerate branch handled) deduped best-per-task with per-representation caps. apply_winner_labels projects the train split onto a winner corpus so the SFT builder (winner_labels_path) distills the policy's own verified programs — the entropy-preserving warm start the old library's law prescribes and its paper lineage used (entry entropy 0.32 vs canon-SFT's 0.096, both now measured). Not launched: sampling waits for a free lane per the overnight concurrency discipline.
The deployment protocol from the private library, minimally ported: init round of 8 attempts, champion selection, deterministic number- redacted feedback (closeness bucket, EXTRA/MISSING material direction from full-mask chamfer asymmetry now recorded by the evaluator), then revision rounds seeded by the champion. Champions carry their code for gallery rendering.
…ault The design spec and plan quoted private-campaign history and local paths; the public rationale lives in rl/README.md. Launch scripts pin the runs root explicitly so running campaigns are unaffected.
data/training/qwen-rl-v2: the validated record behind the released adapter (qpaig-mit/pixcell) - F1-F8 benchmark JSONs (base 0/64 executable -> mean 0.228 / best-of-8 0.467), the attempt-measure-revise loop record (mean champion IoU 0.491) with every champion program, and the re-executed render gallery. rl/track_a/make_gallery.py regenerates the gallery from the record alone. READMEs register the release; runs-root and sandbox wording brought in line with the shipped defaults.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Replaces the ceremony-era
rl/tree with a single lean training path over the frozen dataset, restoring the training signal that demonstrably works — and ships the model it trained, with its validated evidence record.shaped_v3b) — rectangle-baseline-normalized IoU/Dice (iou_n) + boundary-chamfer teacher + 0.05 executable floor, constants pinned by tests. The prior tree'sraw_ioupolicy starved GRPO of gradient and never produced a trained checkpoint.rl/studies/,rl/evaluation/,rl/track_c/, and the recipe-lock/mastery/promotion surface.rl/common/(hash-verified loader, prompt contract, evaluator) stays.track_a/run.py): per-level SFT/RL stages as resumable subprocesses chained on training weights; fixed 80-task cross-run probes; level-aware SFT gates (failed gate = skip/fallback, never a poisoned lineage); prefix forking, depth variants, backward-replay knob.benchmark_eval.py), the iterative attempt-measure-revise loop (agent_loop.py), and the gallery composer (make_gallery.py) that re-executes champion programs and re-renders them against the frozen fixtures.Results and the released model
A raw-base L0→L4 campaign (Run B: 150 GRPO steps, no SFT, no critic) on the released dataset:
agent_loop.py: 8 attempts, then 3 rounds × 4 champion revisions under number-free verifier feedback): mean champion IoU 0.491.qpaig-mit/pixcell(LoRA r32 for Qwen3.6-35B-A3B), carded as the iterative system it is.data/training/qwen-rl-v2/versions the validated evidence: benchmark JSONs and the full loop record with every champion program. The render gallery is not stored —make_gallery.pyregenerates it from the record (the composite ships on the HF model card). Raw step-level ledgers, checkpoints, and W&B state stay outside the repo.What this does not touch
dataset/(frozen, digest-verified before every paid launch),src/,arxiv/,release/. Changes are confined torl/plus thedata/training/qwen-rl-v2/evidence release and its two README registrations.CI:
pytest tests/ rl/ release/green at branch head;ruff check rl/clean.Merge note: recommend a squash merge — several overnight commits are unsigned (authorized; signing agent unavailable) and message-verbose; a squash lands one clean signed commit on main.