feat(recovery): reconcile persistent deployment intent - #101
feat(recovery): reconcile persistent deployment intent#101qinyue0508-cmd wants to merge 2 commits into
Conversation
rjckkkkk
left a comment
There was a problem hiding this comment.
I found two blocking behavioral issues. go test ./..., go vet ./..., and targeted race tests pass, but the missing-container reconciliation path and persisted numeric config path are not covered by those tests.
| if restartDelta > 0 || next.ConsecutiveFailureCount >= next.Policy.ConsecutiveFailures { | ||
| next.RecoveryState = StateWaiting | ||
| } | ||
| return Decision{Intent: next, Action: ActionNone} |
There was a problem hiding this comment.
[P1] Recover container runtimes when the runtime object is absent. This branch returns ActionNone for every Docker/K3S failure. When observation.Exists == false, no restart delta can ever be observed, so AttemptCount remains zero and the intent sits in waiting forever. Docker's unless-stopped cannot recreate a removed container, and K3S deploys a standalone kind: Pod, whose restartPolicy: Always only restarts containers inside an existing Pod; it does not recreate a deleted Pod object. Thus an externally removed container/Pod violates the persisted desired_state=running indefinitely. Please distinguish confirmed absence from an unhealthy-but-existing container and recreate (or otherwise terminate/quarantine) the missing object.
|
|
||
| func decodeDeploymentIntentConfigJSON(value string) (map[string]any, error) { | ||
| decoder := json.NewDecoder(bytes.NewBufferString(value)) | ||
| decoder.UseNumber() |
There was a problem hiding this comment.
[P2] Keep persisted numeric overrides compatible with the fit checks. UseNumber makes every recovered numeric config value a json.Number, and RecoveryApply feeds this map back through resolveDeployment. However, knowledge.CheckFit converts gpu_memory_utilization, mem_fraction_static, and tensor_parallel_size through toFloat64, which handles only Go floats/ints and turns json.Number into zero. As a result, an automatic recovery silently skips the unified-memory reserve, current-free-VRAM adjustment, and GPU-count validation that the original deploy received. A focused test with json.Number("0.95") and json.Number("2") returns Fit:true with no adjustments on a one-GPU unified-memory host. Please normalize decoded numeric values or extend all relevant numeric consumers before replaying the persisted config.
Summary
Scope
This PR is based on the latest develop branch and contains only generic deployment intent, reconciler, and runtime recovery work. It does not include AMD395 binaries, vendor branches, or engine inventory/lifecycle changes.
Regression coverage
Validation