cli: tear a head run's learners down from the head and confirm every cluster at the cloud (re-land #61 on main) - #67
Merged
Conversation
…cloud Merges prB/head-side-teardown onto the stack (its Modal-vs-head conflict in cmd_down is resolved by classifying clusters first: Modal islands stop with the app, a head run's learners are torn down from the head, the rest go through this machine's sky) and closes the gaps that still leaked H100s: - a local sky.down error other than "does not exist" no longer counts as gone; a head run's learners never take the local path at all - every cluster this machine downs is confirmed at the cloud through terminate_and_verify (which now takes a down hook and, without a probe, trusts only a clean down or "does not exist"); the head-side script verifies learners the same way before the head is deleted - Modal islands are confirmed via the app's state (stopped, 0 tasks) - "run is down" and exit 0 appear only when everything is confirmed; otherwise the run is marked TEARDOWN_INCOMPLETE with the unconfirmed clusters so yeto status shows it and a rerun picks up
sky's StatusVersion enum defines only __ge__, so the '<' in _cloud_live_instances_probe raised and every cluster fell back to 'trusting sky.down' with a one-line warning; cloud verification had never actually run. Verified read-only against a stopped Nebius head: the probe now returns its instance id.
…he td1/td2 teardown runs
…run-teardown spec
Picks up the relocation of the external-router test, which removes the conflict with #59. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#61 was merged into its stacked base branch after that branch had already landed on main, so its changes never reached main. This PR carries the same branch onto main; its diff against main is exactly #61's changes.
🤖 Generated with Claude Code