Skip to content

fix(agent-challenge): owned-only staging CVM teardown + selfdeploy hotpatches - #5

Open
alpha1122x wants to merge 40 commits into
mainfrom
fix/ac-owned-cvm-teardown-selfdeploy-hotpatch
Open

fix(agent-challenge): owned-only staging CVM teardown + selfdeploy hotpatches#5
alpha1122x wants to merge 40 commits into
mainfrom
fix/ac-owned-cvm-teardown-selfdeploy-hotpatch

Conversation

@alpha1122x

@alpha1122x alpha1122x commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

  • Safety: staging run_staging.sh no longer account-sweeps Phala CVMs. It deletes only ids this run owns (cvms.txt + work/owned_cvms.txt), logs the plan before delete, refuses foreign ids, and exposes --dry-run-teardown.
  • Selfdeploy hotpatches: shape-mismatch formatter, plan-bound instance types, loopback-only insecure HTTP opt-in, pre-artifact compose-hash matching + encrypt allowlist scoping.
  • Staging stack + docs (secrets gitignored; no execution proof claimed).
  • Prod compose upgrade path (docs only): packages/challenges/agent-challenge/docs/prod-compose-upgrade.md — exact daf0f209… → 9a550b2d… delta, ordered production change list, measurement capture rules, rollback, and verification plan. Does not execute the upgrade.

What is verified

  • CLI/client/measurement/eval unit coverage under:
    cd packages/challenges/agent-challenge && uv run pytest tests -k "selfdeploy or measurement or client or eval or staging or phala" -q
    → 13 failed / 765 passed / 2 skipped on this filter (same failure set as pre-change baseline for the pre-existing failures; new staging teardown + shape/loopback + compose-hash-determine suites green).
  • Full package baseline context from the brief: 55 failed / 3305 passed / 9 skipped / 0 errors — do not treat this PR as clearing the global 55.
  • Public AC endpoint https://chain.joinbase.ai/challenges/agent-challenge removes the need for SSH and for any Cloudflare tunnel when driving the production validator API.
  • Owned-teardown proof: dry-run plan with account ids [cvm_owned_only, cvm_prod_submission_11, cvm_noise] selects only cvm_owned_only; foreign ids appear under will_not_delete_foreign. assert_id_owned / --check-id exits non-zero for foreign ids.
  • Phala GET /cvms count 0 after this work (no leftover CVMs).
  • ruff clean on touched Python files.
  • Secret scan on staged paths: no secrets committed (mnemonic hits are code that generates throwaway hotkeys, not committed secrets).
  • Staging harvested dstack-client-trust.crt is gitignored (class config/*.crt) with committed .example placeholder.

What is NOT verified

  • The staging stack has never completed a successful end-to-end run (review → eval → guest_artifact_proof). Do not treat compose/scripts as proven green.
  • Production compose upgrade was not executed (no SSH, no live pin cut). See the upgrade doc for the authorized follow-up.

Blocking finding (execution proof)

The production eval compose pin daf0f209… was measured WITHOUT CHALLENGE_PHALA_EVAL_ARTIFACT_{URL,TOKEN} in allowed_envs.

Consequences:

  • Phala cannot inject the artifact grant into the guest.
  • The guest cannot download the miner ZIP.
  • A matching guest_artifact_proof is therefore structurally impossible on that pin (the earlier T8 run on the same pin also produced none).

Prerequisite (blocking, not a footnote): prod must adopt an artifact-aware compose pin and a fresh submission before real execution proof can exist.

Exact upgrade path (reviewable)

Full procedure: docs/prod-compose-upgrade.md

Derived offline (generator + tests; not guessed):

Pin Image Artifact envs
Live daf0f2090c02546c694bc7dc49516fd2629f4b8f9dd89e9bc2ed5c4156b662df agent-challenge-eval@sha256:bf598fb8… absent
Target 9a550b2dc0f06797976194bd4b53b8d7bfc8630f6390689f51b0bfebd36de622 agent-challenge-canonical@sha256:753e2296… present (CHALLENGE_PHALA_EVAL_ARTIFACT_{URL,TOKEN})

Same-image alternative (eval@bf598 + artifact envs only) hashes to 3a81feaf… — also artifact-capable, not labeled 9a550b2d…. New MRTD/RTMR/os_image_hash must be measured, never invented.

This PR does not obtain execution proof and does not cut the production pin.

Incident (submission 11)

During a live staging run today, the previous account-wide CVM sweep deleted a production eval CVM mid-execution right after the key-release grant. That permanently wedged submission 11 (eval_running, key_grant_state=granted, retryable=false, cancel/failure both 409) until ~15:03Z expiry.

How this PR prevents recurrence:

  • Teardown selects deletes only from the run track + durable work/owned_cvms.txt.
  • phala_delete_cvm hard-refuses any id not in that owned set.
  • Pre-run no longer sweeps the account when foreign CVMs are live (warn only).
  • --dry-run-teardown prints the plan without deleting; tests lock foreign-id non-selection.

Changes

  • selfdeploy/measurements.py — format_eval_shape_mismatch_error
  • selfdeploy/cli.py — plan-derived instance types; explicit mismatch fails closed
  • selfdeploy/client.py — http:// only loopback + SELFDEPLOY_ALLOW_INSECURE_LOOPBACK=1
  • selfdeploy/eval.py — compose-hash candidates incl. pre-artifact allowed_envs; encrypt scoped to measured allowlist
  • scripts/staging/ — owned-only teardown policy + runner; examples/pins; secrets + harvested *.crt gitignored
  • docs/prod-compose-upgrade.md — blocking prod pin cut procedure (docs only)
  • docker-compose.staging.yml, docs/staging.md, root AGENTS.md staging note

Deliberately left out

  • packages/challenges/agent-challenge/uv.lock (dated Jul 20, pre-dates this work). Workspace resolution is owned by the monorepo root uv.lock; a nested package lock is redundant and not part of this hotpatch.
  • Real config/challenge_token, challenge.env, config/kr/**, dstack-client-trust.crt, review_evidence_encryption_key, data/, work/ (runtime + secrets).
  • Any production host mutation / pin cut.

Test plan

  • uv run pytest tests -k "selfdeploy or measurement or client or eval or staging or phala" -q (from package dir)
  • uv run pytest tests/test_staging_cvm_teardown_policy.py tests/test_selfdeploy_shape_loopback_policy.py tests/test_eval_compose_hash_determine.py -q
  • Offline hash asserts: daf0f209… and 9a550b2d… reproduce from generator
  • python3 scripts/staging/cvm_teardown_policy.py --owned-file … --account-ids-json … --dry-run proves foreign id not selected
  • ruff on touched files
  • GET /cvms count 0
  • git status shows only untracked uv.lock (intentionally excluded); certs gitignored
  • Authorized follow-up (not this PR): execute docs/prod-compose-upgrade.md, then fresh submission → guest_artifact_proof with hash 61cca9bc…

CI

  • Waiting for required checks to go green on this head SHA.

Notes

echobt and others added 30 commits July 26, 2026 00:34
@echobt
feat(agent-challenge): sync eval wire GHCR pins and score-chain gates
16a89ce 
Mirror standalone agent-challenge dual-flag admission: GHCR live-registry
refs, package_tree_sha score-chain refuse, progress module, and fixture pins.
@echobt
style(tests): wrap E501 GHCR digest assertion lines
263eeb1 
Keep package-scoped ruff line-length 100 after GHCR path retarget.
@echobt @sisyphus-dev-ai
feat(agent-challenge): add eval artifact import and download routes
71da131 
Guest CVMs fetch the miner ZIP over a short-lived bearer grant instead of
baking bytes into the image. Compose allowlists the delivery env names so
the generator hash stays intentional.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
feat(agent-challenge): enforce guest dual-hash artifact provenance
74ae2bb 
Guest hashes download and executed bytes; schema-v2 envelopes carry
guest_artifact_proof; host scoring rejects success results whose proof
describes a different artifact than the submission (distinct reason codes).

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
feat(agent-challenge): discover Phala app_id per account at provision
1b18bf7 
Static app_id pins only worked for the operator account. Discovery omits
nonce and app_id so Phala mints the account-derived handle; live 422
showed nonce without app_id is illegal.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
feat(agent-challenge): add miner agent packaging and loop scripts
6489d24 
Reference miner loop and ZIP builder for submission path exercises
against a controlled hotkey without on-chain registration.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
feat(agent-challenge): add authenticated raw weight push loop
61651c2 
Optional background task posts winner-take-all weights to master with a
ledger so retries stay idempotent; wires eval artifact router on create_app.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
feat(db): add digest allowlist and attestation nonce tables
d82e476 
Persist image digest allowlist and one-time attestation nonces so master
can gate Lium/constation evidence without in-memory-only state.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
feat(compute): add Lium attestation and constation custody path
185ceba 
Shared attestation payload, nonce issue/consume, digest allowlist checks,
and constation poller/runner so pod evidence can be corroborated offline.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
feat(master): expose constation attestation HTTP routes
5a38603 
Wire nonce/allowlist/bundle endpoints through app proxy and CLI helpers
so workers can complete the Lium attestation handshake end-to-end.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
feat(prism): gate scoring on constation attestation evidence
6d9c756 
Ingest and check constation bundles before accepting prism results;
attestation routes and breakglass keep offline characterization intact.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt
fix(compute): lock Lium prism training to 1-GPU Blackwell offers
8ec278d 
Prod hotpatch refuses multi-GPU hosts that reject gpu_count=1 splits and
exposes LiumClient.for_prism_training for the capacity scheduler path.
@echobt
feat(compute): add Lium capacity scheduler and orphan reconciliation
1417e8d 
Bring prod hotpatch modules for master-owned training admission
(LiumCapacityScheduler) and orphan pod cleanup, plus settings wiring helpers.
@echobt
feat(config): add ConstationSettings and LiumTrainingSettings
4302db4 
Prod hotpatch settings surface for fail-closed constation orchestration
and optional master-owned Lium training admission (default off).
@echobt
feat(constation): land prod allowlist pin API and custody runtime mod…
f451840 
…ules

Hotpatched allowlist_repository gains get_active_pin and identity payload
mapping. Also restore image-resident custody/orchestrator/pod_binding/
attestation_keys/bundle_seal/routes pieces required by the prod main
wiring but missing from this branch tip.
@echobt
feat(master): stamp constation pins and tick Lium capacity in orchest…
0bda6db 
…ration

Prod hotpatch adds ConstationPinSource stamping on Prism dispatch and
optional LiumCapacityAdmission.tick during run_once.
@echobt
feat(cli): wire constation runtime and Lium capacity scheduler into m…
e00c170 
…aster

Prod hotpatch builds custody runtime, pre-forward hook, pin source, and
try_build_lium_capacity_scheduler when constructing the orchestration driver.
@echobt
fix(master): load per-challenge embed.env into isolated child env
9eac481 
Prod entrypoint hotpatch (and fix/master-embed-ac-env-file) restore the
operator extension point dropped by env -i isolation so attestation and
review switches from embed.env actually reach embedded challenges.
Host-only Prism policy defaults are intentionally not copied.
@echobt
fix(agent-challenge): resolve review assets under site-packages and g…
976a4a4 
…ateway-free analyzer

Prod hotpatches: REPO_ROOT finds docker/review when installed in
site-packages; host analyzer skips missing Base LLM gateway parking and
completes AST+similarity under gateway-free attested product mode.
@echobt
feat(prism): add pod_boot helpers and plagiarism LLM adjudicator
be4e62b 
Prod hotpatch modules: safe pod boot env/install planning, and the
OpenRouter dual-gate plagiarism adjudicator (wired when queue lands).
@echobt
feat(prism): add plagiarism LLM settings and checkpoint intake status
666f4b3 
Prod hotpatch: PrismSettings plagiarism_llm_* / openrouter fields, and
CheckpointIntakeService last_status observability with fail-closed publish.
@echobt
test(agent-challenge): align lifecycle tests with gateway-free analyzer
663c9a8 
Prod hotpatch no longer parks missing Base LLM gateway as llm_standby.
Update the two contracts to expect first-pass allow and no requeue loop.
@echobt
style: ruff-fix imports and line length on landed hotpatches
6b807c2
@echobt
feat(selfdeploy): land disk guards, lifecycle disk cost, and HTTP CVM…
6ecf431 
… delete

Prod bucket-D shapes/lifecycle/phala fixes: stage disk bounds, eval
tdx.xlarge default, disk-aware budget, and PhalaCloudClient.delete_cvm
so validators can tear down without a phala binary.
@echobt
fix(agent-challenge): resolve dataset-digest path for site-packages i…
8795622 
…nstalls

Prefer env/manifest and /app|/opt golden layouts over Path(__file__).parents[3]
so eval/prepare no longer 503s when the package is installed under site-packages.
@echobt
feat(agent-challenge): merge eval-path concurrency, progress wire, an…
53caf7d 
…d compose envs

Land miner n_concurrent plan binding, optional hydration_digest, mid-run
progress validators, and EVAL_PROGRESS/DSTACK_DOCKER allowed envs. Keep
guest_artifact_proof, RA-TLS KR authority checks, dual residual review
gate, and raw-weight-push settings from the repo side.
@echobt
feat(selfdeploy): add disk_size to discovery deploy and CLI teardown …
fc730c0 
…handoff

Provision disk_size without re-pinning app_id or reintroducing nonce.
CLI gains HTTP teardown, disk flags, and eval run-token handoff while
keeping multi-account discovery and artifact grant minting.
@echobt
fix(agent-challenge): tighten review runtime submit_verdict tool-call…
0f7e15a 
… prompt

Land prod tool-call contract wording so Grok tool_choice=auto emits a
closed submit_verdict call. Keep the joinbase REVIEW_API_BASE_URL pin.
@echobt
feat(prism): land worker-plane no-op and plagiarism LLM adjudication
c6a577e 
When worker_plane is enabled the master queue no longer claims GPU work.
Borderline/attach plagiarism pairs go through the OpenRouter adjudicator
instead of terminal-rejecting the quarantine band.
@echobt
test(agent-challenge): cover bucket-D merges and align pin/budget fix…
82e4549 
…tures

Add behavioral tests for digest path, n_concurrent, progress wire, disk
guards, and dual residual signature. Update compose pin hash and
lifecycle/disk/teardown expectations for the landed product changes.
echobt and others added 10 commits July 28, 2026 09:36
@echobt @sisyphus-dev-ai
fix(selfdeploy): bind instance types to signed plan and format shape …
8ee59da 
…errors

Omit --eval/--review-instance-type to follow the plan; explicit mismatch
still fails closed with an operator-safe message instead of AttributeError.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
fix(selfdeploy): allow http challenge URLs only on loopback with opt-in
b830e51 
http:// is accepted only for 127.0.0.1/localhost/::1 when
SELFDEPLOY_ALLOW_INSECURE_LOOPBACK=1; all other cases stay https-only.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
test(selfdeploy): cover shape mismatch formatter and loopback HTTP po…
7ae256f 
…licy

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
fix(selfdeploy): match pre-artifact compose hash and scope encrypt al…
dea28d0 
…lowlist

Search KR/allowed_envs candidates so prod pin daf0f209… can be reproduced,
and encrypt only secrets listed on the measured compose allowlist.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
fix(staging): delete only owned Phala CVMs and never account-sweep
15df3e9 
Track created ids per run and in work/owned_cvms.txt; refuse foreign ids
in phala_delete_cvm; log the teardown plan; add --dry-run-teardown proof.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
chore(staging): add local AC staging compose stack and secret gitignores
29b4c23 
Ship compose + example/pin fixtures; keep tokens, KR PKI, challenge.env,
and runtime data/work dirs out of git.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
docs(staging): document owned-only CVM teardown and local staging loop
e39ae07 
Prefer run_staging.sh over SSH to prod; note signature-only keypairs and
that real Phala CVMs must always be torn down (owned ids only).

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt @sisyphus-dev-ai
fix(staging): resolve owned vm_uuid tracks to API cvm_* delete ids
023de37 
Deploy acks record vm_uuid while GET /cvms returns id=cvm_*; map owned
track entries through listing aliases so teardown can delete what it created
without selecting foreign CVMs.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@echobt
fix(selfdeploy): fail loud on unknown Phala CVM list shapes
a58e584 
Legacy GET /cvms parsing treated unrecognized envelopes as an empty
list, under-reporting live spend as count 0. Pin CLI API version
2026-06-23, list via /cvms/paginated, and raise on unknown shapes so
teardown confirmation never succeeds when the count is indeterminate.
@echobt
docs(agent-challenge): document prod artifact-aware compose upgrade
acc67b1 
Add prod-compose-upgrade.md with the derived daf0f209→9a550b2d delta
(allowed_envs + image), ordered production change list, measurement
capture rules, rollback, and guest_artifact_proof verification plan.
Cover harvested dstack-client-trust.crt under staging gitignore with a
placeholder .example. Staging runner also surfaces CVM-missing mid-poll.
No production execution.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alpha1122x @echobt