Skip to content

feat(k8s): trace-instrument the workflow — every error clear, waits live - #215

Merged
fedemaleh merged 53 commits into
betafrom
feat/trace-instrumentation
Sep 2, 2026
Merged

feat(k8s): trace-instrument the workflow — every error clear, waits live#215
fedemaleh merged 53 commits into
betafrom
feat/trace-instrumentation

Conversation

@pablovilas

@pablovilas pablovilas commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Instruments the k8s scope workflows with the tracing SDK so a custom-scope deployment tells the SAME story as a native scope on the deployment page — declared plan, live waits, lineage, affordances, honest failures.

What the wire declares

  • Job identity (trace.job + labels entity / strategy / scope.provider): the deployment page previews the full pending checklist before the first event arrives — the provider (service-specification id) is the stable join across every scope this agent serves.
  • Stations (trace.groups): Setting up / Waiting for instances / Switching traffic / Finalize, with placeholder stations the later workflows fill.
  • Declared steps only (trace.default: false on every deployment workflow): engine fragments without a trace block (override-injected plumbing, permanently-skipped scale no-ops, metrics bookkeeping) stay off the wire. A full deploy is ~60 nodes instead of ~150; one traffic increment is 2–4 honest steps instead of 12. Script-emitted child steps (per-manifest applies) still fill the technical fold with real observable work.
  • The wait narrates itself: live explain per situation change (graced boot wording, humanized restart/problem reasons with kubernetes' verbatim message and last crash log), an instances-health meter affordance, count progress, the consumed image and produced log as io + lineage, a deploy-log affordance. What it waits on rides the tracing.signal facet — never label chips.
  • Skips are sentences: np_step_skip reasons read as narrative ("instance counts are set by the manifests for the blue_green strategy — nothing to scale").
  • Delete lineage: removed workloads/services/ingresses ride the finalize milestones as removed-* pointers.

Vendored SDK (catalog-tracing-sh)

Bumped for: adopted-node coordinate triple, progress-unit whitelist, staged context replacing by identity (io by name, affordances by kind, facets by namespace), and wire-shaped edge bindings ({name} — the full descriptor dead-lettered every produces/consumes edge).

Companion PRs

Validated live end-to-end on the Stage scope (deploys 1313906633 / 1318035402 / 1322164171): zero dead-letters, native-parity page from birth to receipt.

@pablovilas

Copy link
Copy Markdown
Contributor Author

Update (5e74817): SWM-shaped plans. After deep-inspecting main-scope-workflow-manager's k8s implementation (provision_plans.js + emission sites), the workflows now declare the platform's exact step vocabulary via the np CLI's new trace: blocks (nullplatform/cli#216):

Scope provisioncreate-namespace* · create-ingress · wait-for-ingress (route53-only) · create-dns — the same four stations as EKS_SCOPE_PLAN.
Deploy → per-manifest sub-steps under apply: create-secret · create-deployment · create-hpa · create-service · create-pod-disruption-budget · configure-ingress, then wait-for-instances (group waiting-instances — the dashboard's instance pips) and traffic-switch (group switching-traffic).

Step names are untouched — overrides keep working. Plumbing is trace: false. Flavor-unreachable stations (flavors: [route53], workflow flavor: $DNS_TYPE) are never declared on AKS/external_dns; branch-skipping scripts call np_step_skip so their step closes skipped, never a hollow completed. Full suite: 1057 tests, 0 failures. Degrades gracefully on an older CLI (unknown YAML keys ignored; skip helper guarded).

Merge order: catalog-tracing-sh#6 → cli#216 → this.

@pablovilas

Copy link
Copy Markdown
Contributor Author

Update (3749e97): full SWM narrative parity. Now depends on catalog-tracing-sh#7 (inline io + error details; the vendored artifact here is built from it) in addition to cli#216. Merge order: sh#7 → cli#216 → this.

The platform now nests each custom-scope workflow inside its provisioning
run (SWM injects the trace carrier into the action parameters; the np CLI
adopts it and traces one step per workflow fragment). What was missing is
the WHY: a failed step read as "exited with status 1" while the real
reason scrolled past in the log.

One seam fixes that structurally: `log error` — every one of its 700+ call
sites across the k8s scripts now also lands the message as a tracing.error
facet ON the step it happened in, via the vendored shell SDK adopting
NP_TRACE at call time. A silent failure (no log error on the way down) is
covered by an ERR/EXIT trap pair that reports the failing command itself.
Failures in a custom scope now read exactly like native ones: which step,
and what went wrong.

The two long waits (ALB active, deployment active) also mark their step
`waiting` with live progress labels, flushed per beat with a hard 2s bound
so a down tracing API costs at most ~6% of a wait's poll budget and never
extends its wall-clock timeout.

All of it is best-effort and inert by default: no nptrace.sh, no
NP_API_KEY, or no NP_TRACE means plain logging, byte-identical to before
(asserted byte-for-byte in tests). Heartbeat call sites are guarded so
overrides reusing these scripts without k8s/logging stay clean.
…ait loop

Phases inside a script can now be their own step in the trace:
np_scope_step_begin/end/timeout open a keyed sub-step under the platform
step, log errors and heartbeats attach to the innermost open node, a
sub-step still open when the shell dies inherits the shell's outcome, and
a wait that hits its deadline closes as timed_out rather than failed.

Every wait loop is instrumented — wait_for_alb and wait_deployment_active
upgraded from bare heartbeats to sub-steps (rollout heartbeats now carry
desired/launched/ready/available/updated replica counts as labels), and
wait_on_balancer, verify_ingress_reconciliation and
verify_http_route_reconciliation gain sub-steps and heartbeats.

scheduled_task/logging picks up the same tracing hooks, so scheduled-task
scopes trace like k8s ones. The blue-green switch needs no iteration
steps: it is one-shot, and switch_traffic.yaml's named steps already are
the plan.
Custom scopes now report the same data lineage the scope-workflow-manager
does, by the same canonical dataset ids so the graphs join by value:

  • apply_templates turns kubectl apply output into produces edges for the
    kinds the platform lineage model knows (k8s-deployment, k8s-service,
    k8s-ingress) — one chokepoint covers every workflow
  • the namespace create records k8s-namespace:<ns>
  • manage_dns CREATE records dns-record:<fqdn>
  • wait_for_alb records the load-balancer:<arn> dependency
  • wait_deployment_active records the cross-flow join back to the build
    (consumes <asset.type>:<asset.url>), produces this deploy's log dataset
    with the concrete log query as its pointer, declares the deploy-log
    affordance the UI renders as "view logs", and reports rollout progress
    ({current, target, unit: instances}) per heartbeat

Vendored nptrace.sh rebuilt from catalog-tracing-sh#6, which adds
np_trace_produces/consumes/affordances/progress to the shell SDK.
…r custom scopes

The workflows declare each step's trace identity (np CLI trace: blocks)
using the exact step keys, titles and plan groups the platform's own
scopes emit — create-namespace, create-ingress, wait-for-ingress,
create-dns for the scope provision; wait-for-instances (group
waiting-instances, where the dashboard attaches the instance pips) and
traffic-switch (group switching-traffic) for deployments. Step NAMES are
untouched: customer overrides anchor on them.

The actual create-* deploy steps are emitted per applied manifest by
apply_templates as keyed sub-steps (deployment-*.yaml → create-deployment,
secret-* → create-secret, scaling-* → create-hpa, service-* →
create-service, pdb-* → create-pod-disruption-budget, ingress-* →
configure-ingress/create-ingress) — the custom flow renders then
bulk-applies, so this is where those resources really get created.

Plumbing (load logging, assume role, renders, context builds) is
trace: false; route53-only stations carry flavors: [route53] so an AKS or
external_dns scope never declares a station it cannot reach; and scripts
whose branch isn't taken declare np_step_skip so their step closes
skipped, never a hollow completed.

Requires nullplatform/cli#216 for the trace: blocks and np_step_skip;
both degrade to exactly today's behavior on an older CLI (unknown YAML
keys are ignored; np_step_skip is guarded).
K8S_FLAVOR names the cluster variant (eks in the base values, aks and aro
in the overlays) — an independent dimension from DNS_TYPE, so the
workflows declare both as trace flavor tokens
(flavors: ["$K8S_FLAVOR", "$DNS_TYPE"]) and each step gates on the
dimension it genuinely depends on.
…t even as skipped

The trace is for users: verify_networking_reconciliation has nothing to
dispatch on azure/aro, so it is flavor-gated out of those plans entirely,
like the ALB stations. skipped stays reserved for declared steps whose
branch genuinely may or may not run on that flavor (a namespace that
already exists).
k8s-scope-create, k8s-deployment-initial and k8s-deployment-switch-traffic
carry their plans as previewable job definitions (version derived from the
plan hash per flavor), and each run links instance_of — the same
plan_source: job model the platform's own scopes use.
…plains, traffic set

wait_deployment_active now reports everything the platform's own
wait-for-instances does, from the data the script genuinely observes: the
'instances' counted io (healthy/launched/desired, plus per-pod restart
history from the pods' containerStatuses), the instances-health affordance
meter with unhealthy counts and probe reasons, and a per-heartbeat explain
at the operator's severity — warn while probes fail or restarts accumulate
('All 3 instances healthy — after 4 restarts' keeps a healed crash-loop
visible), plain while booting, error with structured details when it gives
up.

The blue/green switch reports the same set the platform's tracedTrafficSwitch
emits — labels, the from→to narrative, the traffic-switch affordance with
the resulting split, the request as input, what landed as output, and the
convergence toward 100% — from apply_templates, gated explicitly by
TRACE_TRAFFIC_SWITCH on the switch_traffic step.

Vendored nptrace.sh rebuilt from catalog-tracing-sh#7 (inline io +
error --details).
…lavor, the run's own reason

Every workflow now declares its trace identity, not just the three
lifecycle ones: blue_green overrides the job it would otherwise inherit
from initial (k8s-deployment-blue-green, with its own groups), update
overrides create's (k8s-scope-update), finalize/rollback/delete and the
scheduled-task trigger get their own jobs, finalize's rollout wait is
keyed wait-for-instances, and the plumbing steps (load logging, assume
role, deployment build context) are trace: false across all twenty-odd
workflow files.

A fatal exit now mirrors the REAL reason onto the RUN as well as the step
— the last log error's message, not the exit mechanism — so list views
name the cause without drilling into steps. A run-level NP_TRACE (no step
segment) mirrors nowhere extra.
…y theirs

Verified by executing the failure stories end to end: a Forbidden apply
previously traced as a bare 'Failed to apply' — kubectl's stderr (the
principal, the resource, the namespace) never reached the trace. The
capture now includes stderr, and the failure names the manifest and the
server's reason. Restart narratives gain their cause the same way:
'4 restarts so far (OOMKilled)' instead of just the count, with
restart_reasons on the instances-health meter.
…he app's own

Verified by executing the failure stories: an ImagePullBackOff now traces
the registry's verbatim message ('Back-off pulling image …: manifest
unknown') live within one poll and cause-first at the terminal; a
crash-loop attaches the previous container's last log lines and leads with
the app's own final line ('FATAL: bind: address already in use'). The
classification is live POD STATE (waiting reasons and messages), so
problems that predate the wait or throttle their events are still named —
the event sweep only fills in what state cannot say (which probe, which
path). Boot churn is never presented as a problem, and the narrative
re-evaluates every poll, emitting only when the situation changes.

The 'possible causes / how to fix' bursts no longer shadow the cause:
facets fold last-writer-wins, so recording each hint line as the error
left the LAST HINT as the step's message. The burst's first message is now
the error everywhere (step, run mirror, terminal — one cause-first message,
mechanism in parentheses) and the hints ride as structured details.hints.
End-to-end verification (real CLI executing the real create workflow against
a wire sink) caught scripts that capture the provider's error output and then
log a generic message: build_service_account, create_role (OIDC, account id,
create-role, policy attach), delete_role and require_resource. Each now
carries the captured reason on the traced error, apply_templates-style.

Trace error messages also drop the console decoration (indentation, the
marker emoji) so the cause reads clean on the trace while console output
stays exactly as it was.
The tracing SDK now rides as a submodule pinned to the SDK's release
history instead of a hand-vendored copy of nptrace.sh. The logging hooks
look for the submodule first and still fall back to a root-vendored
nptrace.sh, so repo copies that lose the submodule (archive downloads,
docker build contexts) keep working — and with neither present the
scripts degrade to plain logging exactly as before. CI checks out
submodules so the trace tests exercise the real SDK.
…p_error

When the engine's preamble publishes np_step_error, the first log error
cause of a step is stated through it, so the engine's own failure
terminal carries the real diagnosis instead of a bare exit status — one
message on the wire no matter which side reports last. The exit trap's
synthetic mechanism message never states: it must not outrank the
engine's own evidence (the shell's last stderr words).
A build that loses the tracing submodule (archive download, docker build
context) used to degrade to plain logging in SILENCE — the CLI side traces,
the structure looks fine, and the scope-side story (sub-steps, live waits,
real errors, lineage, affordances) is just absent, with nothing anywhere
saying why. Exactly the failure nobody sees. Now, when the engine is
tracing the run (NP_TRACE) and credentials are present (NP_API_KEY) but
nptrace.sh is not in the bundle, the workflow log carries one loud warning
naming the missing file. Untraced runs stay silent as before.
@pablovilas
pablovilas changed the base branch from main to beta August 28, 2026 20:02
@pablovilas
pablovilas force-pushed the feat/trace-instrumentation branch from ef62f1b to 1f3aaba Compare August 28, 2026 20:02
Validated against real blue/green deployments on a custom scope, side by
side with the native experience:

- finalize declares its own group — its steps used to file under
  Setting up / Waiting for instances, twins of the deploy workflow's
  stations, and the ending never got a station of its own. The entity
  ending overlay names it (Finalized / Rolled back).
- rollback likewise: one Finalize group — a rollback's wrap-up is the
  ending station's story.
- switch_traffic scopes ALL its steps under Switching traffic:
  per-increment runs fold there as attempts (the native reading)
  instead of re-lighting Setting up / Waiting on every increment.
- the wait steps get workflow-distinct keys (finalize-instances-check,
  switch-instances-check): three workflows declared the same
  wait-for-instances key, so the fold merged semantically different
  waits into one row across runs.
Every step a user sees carries a clear title in the native tone
(Validate load balancer capacity, Promote new deployment, Restore
traffic routing, Remove previous deployment); build/template plumbing
is trace: false. rollback gains its full trace pass (it had none).
Steps injected by the overrides repo keep their keys — the dashboard
humanizes an untitled key rather than showing snake_case.
… no submodules

The agent materializes the package source with a plain clone, so the
catalog-tracing-sh submodule arrives empty and every run logged
'tracing SDK not bundled' (observed live: zero step io on three
deployments that declared it). The logging loader already prefers the
submodule and falls back to a root copy — ship the copy. Kept in sync
by the existing re-vendor chore.

Also: blue_green/initial declare placeholder groups (switching-traffic,
finalize) so the whole journey renders pending from the first paint —
paired with the CLI emitting an optional placeholder step per empty
declared group.
- the rollout wait IS the planned step (Instance health check): its
  heartbeats, instance counts and narrative attach to that step
  directly instead of a same-span sub-step — one human moment, one
  node, and the row's detail shows the diagnosis.
- apply lineage records after the per-manifest sub-step closes, so the
  workloads/services/ingress produced land on the planned Apply
  manifests row (technical sub-steps never enter the human timeline;
  their evidence must not vanish with them).
Enrichments on the CLI's planned step (io, explain, wait labels) were
rejected by ingest — the adopted derived-path node emitted no
key/attempt/iteration. Vendors catalog-tracing-sh#10.
Each np_scope_* call minted a fresh adopt handle, so every foreign
re-emit carried a single fact and the API's last-writer-wins facet fold
kept only the final one (apply showed one pointer instead of three; a
lost tail event took its fact with it). The adopted handle is now cached
per NP_TRACE value, so the bag is cumulative and every re-emit carries
the full picture.
- The instance wait speaks each phase's words (WAIT_TITLE): provision keeps
  'Instance health check'; the switch phase verifies scaled instances;
  finalize verifies final capacity.
- Narratives translate provider codes to plain words (the native scopes'
  reason table), and a calm boot narrates itself instead of going silent.
- Diagnose bookkeeping (notify before/after each check, context build) goes
  untraced — the checks are the story.
- Evidence gaps closed: replicas io on rolling scales (an honest skip on
  blue-green), removed-* pointers on deletes, verification summaries on the
  networking verifiers, quota numbers on the capacity pre-flight.
- Scope workflows curated to the same grammar: titles on the observable
  acts, trace: false on template generation and context plumbing.
The finalize station folded eight 0ms override-plumbing fragments as
'technical steps' — native shows none of that. Every deployment
workflow (initial, blue_green, finalize, rollback, delete) now opts
into declared-steps-only: engine fragments without a trace block stay
off the wire, while script-emitted child steps (the per-manifest
create-secret/deployment/service acts) keep filling the fold with real
observable work. publish_alb_metrics untraced everywhere (bookkeeping).
io lineage edges (image consumed, logs/ingress/workload produced) were
silently dead-lettering: the binding carried the io descriptor's
kind/uri, which the API's tracing.binding schema rejects. The bumped
SDK sends {name} on the edge and keeps the full descriptor on the node.
np_scope_wait_heartbeat filed per-poll diagnostics (wait.what/ready/
state/desired/..., including an unresolved-config literal) as LABELS,
which the dialog renders as tag chips — machine noise on a human
surface. What a phase waits on is the tracing.signal facet: name +
direction + deadline; the counts already ride the meter and progress
facets, elapsed rides the node's own clock.
The io names said 'workload' — a k8s category, not a manifest kind.
The dialog now names what kubectl actually reported: deployment (and
removed-deployment on deletes); the dataset ids were always exact
(k8s-deployment:<ns>/<name>).
Every workflow's trace.job carries namespace: @context:scope.provider —
the scope's service specification. Two providers shipping the same
workflow name with an identical plan hash previously converged on ONE
job definition (namespace 'workflows') and would fight over its
identity labels; per-provider namespaces make each scope type own its
jobs by construction.
The page titled these runs by humanizing `labels.workflow` — this file's basename —
which meant the name on screen was whatever the file happened to be called. `trace.title`
says it outright, landing on the run's explain.title where every consumer looks first.
…ow stand

Four assertions still described the behaviour these changes replaced, so the branch
was red on work that is correct:

- a wait states `tracing.signal` (name, direction, deadline in ms), not `wait.*`
  labels — the labels were rendering as code-y chips on the phase;
- an edge's binding names WHICH io the edge is about and nothing more (sending the
  whole descriptor is what dead-lettered every lineage edge);
- progress units are the closed set, so `count` stays `count`.

The extra-k=v-labels case is replaced by its inverse — a guard that no wait
bookkeeping reaches the labels — and a wait whose timeout never resolved is now
covered too.
catalog-tracing-sh#10 is merged, and its branch is gone. The submodule pointed at a
commit that now only exists inside main's history — the vendored content is identical,
but .gitmodules declares `branch = main`, so the pointer should be main's tip.
… stop advertising work they cannot do

'Promote new deployment' (and rollback's 'Restore previous deployment') run
scale_deployments, which is a no-op for every strategy but rolling — so on a
blue-green scope they were permanently grey rows nobody could explain.

The scope's deploy strategy becomes a flavor DIMENSION, read from the same
context path the scripts read, and those steps gate on `rolling`. One job still,
two plan versions — which is exactly what hashing the effective plan is for.

Every remaining finalize/rollback step now carries a one-line description, so the
checklist reads without prior knowledge of the workflow.
The console hints already classify a failing wait — 'The application did not pass its
health check at /health-bad. Detected: Startup probe — app responded with HTTP 404
(expected 2xx).' plus a targeted fix — but that never reached the trace, so a reader who
opened the phase saw counts and a timeout and had to go to the logs for the reason.

The same classifier now feeds the step's explain: WHY on the reason, NEXT on the fix.
It rides the LIVE narrative too, not just the give-up — the reader asks 'why?' the moment
the meter goes amber, not ten minutes later.
The phase read '1 with Startup, probe, failing'. The reason list splits on the comma —
but IFS was ', ', so it split on whitespace too, and the event sweep's already-human
reasons ('Startup probe failing') were shredded into three codes that matched nothing
and were echoed verbatim.

Split on the comma alone, trimming the space that follows it. Codes still translate and
still dedupe after translation.
A container that is OOM-killed reports two things: `state.waiting.reason`
CrashLoopBackOff — the mechanism that keeps restarting it — and
`lastState.terminated.reason` OOMKilled, why it actually died. Both the
classifier and the wait narrative read the waiting reason first, so every
surface named the symptom:

  Waiting for 0/1 instances to be healthy — 1 with crashing repeatedly:
  back-off 20s restarting failed container=application
  pod=d-253247585-630235960-dd685c5-trn9s_nullplatform(498dc0ab-…)

  Reason: The container started and crashed repeatedly.
  Next:   Review application logs for startup errors (…, panics).

That sends the reader hunting for a bug in their startup path when the
answer is the memory limit — and the OOM branch the classifier already
carries could never fire. The OOMKilled fact was on the step all along,
three panels down under "Restarted → Reason".

Now the backoff wrappers (CrashLoopBackOff, BackOff) defer to the
termination reason; every other waiting reason IS a cause and still wins,
so ImagePullBackOff is unaffected. The line becomes:

  Waiting for 0/1 instances to be healthy — 1 with out of memory

  Reason: The container exceeded its memory limit (256Mi) and was terminated.
  Next:   Increase ram_memory for scope 'Stage' or reduce memory usage.

The back-off message is dropped from the narrative — it restates the reason
and then buries the line in two ids nobody reads — and what remains is
capped to a line. Neither is lost: the verbatim message, the CrashLoopBackOff
wrapper (as `reason`) and the new `cause` all stay on the io, so an agent
still sees the loop, the ids and the untruncated text.
Auditing the other warn/failure paths after the OOM fix turned up three
gaps, one of them introduced by that fix:

- `Error` had no branch. Deferring the backoff wrapper to the termination
  reason sends the ORDINARY crash loop — a panic, a bad config, a missing
  dependency, all of which terminate as `Error` — to the nameless default:
  "Pods are failing with reason: Error", with no suggested fix at all. That
  is worse than the generic wording it replaced. `Error` now carries the
  crash-loop advice plus its exit code, the first thing anyone greps for.

- The restart-only narrative never diagnosed. A container that crashed and
  is running again sits between waiting states, so nothing classified it,
  and the branch said only THAT it restarted — an amber phase with no cause
  and no next step. It calls the classifier like every other surface now.

- Evicted, Unschedulable and DeadlineExceeded were in the humanize table but
  had no branch, so they reached the reader named and unexplained.

The unknown-reason default keeps its EMPTY suggested fix: that emptiness is
the signal that hands the reader the full generic troubleshooting checklist,
which is richer than any line this branch could invent for a code it does
not know. Filling it in swapped that checklist for a one-liner.
`ram_memory` is the capability KEY. The scope form calls it "RAM Memory", so
"Increase ram_memory for scope 'Stage'" sent the reader looking for a field
that is not on the page — the same leak the OOM diagnosis itself was: the
right answer, spoken in a vocabulary the reader has no way to act on.

Three strings named an internal or foreign identifier:

- ram_memory -> "the RAM Memory of scope 'X'".
- initialDelaySeconds/timeoutSeconds are the KUBERNETES spellings; the scope's
  own settings are `health_check.initial_delay_seconds` / `timeout_seconds`,
  shown as Initial Delay and Timeout.
- activeDeadlineSeconds is not a scope setting at all, so it named a field the
  console does not have. Reworded to say the thing instead of the field.

Genuine Kubernetes objects (PVC, configmap, nodeSelector/affinity) keep their
Kubernetes names: those are what they are, and renaming them would make them
unfindable in the cluster the reader has to go look at.

Three tests asserted the old wording; two of them encoded a field key as the
user-facing contract, which is what let this ship.
Seven of the sixteen workflows declared no `trace:` block at all, so their runs
reached a consumer with every naming rung empty:

  run_id  : workflow-diagnose-395c6e18-5574-4b54-b39e-735541b16aa2
  explain : null      <- no declared title
  job     : null
  key     : null

The title contract then lands on the run id, and a consumer that will not print
a uuid substitutes its own neutral word: the deployment page showed the failure
diagnostics as "Activity · 21 steps · all succeeded", which names nothing the
reader can act on.

The name was in `labels.workflow: "diagnose"` the whole time — and that is
exactly the label a consumer must NOT title from: it is `filepath.Base` of this
file, private to this engine's layout, and the next producer's `workflow` label
means something else. So the fix is to say it, not to have it read.

Titled: the deployment diagnose and kill_instance workflows, and the scope
diagnose, pause/resume-autoscaling, restart-pods and set-desired-instance-count
workflows. Every workflow in the package now names itself.
Every check already decides a verdict — 10 call sites report `failed`, 3
`warning` — and builds real evidence: a summary, affected pods, details and
suggested actions. All of it went into the results file for the diagnose API
and none of it onto the trace. On the wire every one of the 21 check steps read:

  completed  sev=-  Memory-Limits            err=-  out=0
  completed  sev=-  Container-Crash-Detection err=- out=0

So the page showed "21 steps · all succeeded" directly under a deployment that
had just died of an OOM — including the two checks that found it.

`update_check_result` is the one funnel every check goes through, so the verdict
is mirrored there: the finding becomes the step's `explain.what`, its first
suggested action becomes `next`, and the evidence rides along as an output so
the dialog can show the affected pods.

The step's STATUS is deliberately left alone. A check that finds a problem did
its job; failing the step would conflate "this check broke" with "this check
found something". The finding rides `explain.severity` — error for a failed
check, warn for a warning — the same channel every other producer surface uses.
A passing or skipped check stays silent: 21 lines of green is how nothing gets
read.
Comment thread k8s/deployment/tests/print_failed_deployment_hints.bats Outdated
Comment thread k8s/deployment/tests/print_failed_deployment_hints.bats Outdated
Review feedback: these messages are the main output the deploying dev sees,
so a test should fail when one of them changes. Asserting fragments
("Detected: Startup probe", "not yet listening") let a message be reworded
without a single test noticing.

Every pod-diagnostic branch now asserts the complete emitted lines -- the
reason, the detected line, the details, the recent-warnings block and the
suggested fix -- so breaking any one of them is a red test. Verified by
reordering four words inside one message: test 11 fails, where the old
fragment assert passed.

Also drops the explanatory prose comments from the file; the repo is public
and the asserts now say what the tests were explaining.
…heir plans

The deploy workflows (blue_green, initial) register their jobs with identity
labels, which is how the deployment page shows the whole checklist pending the
instant an operation is requested. The lifecycle workflows (rollback, finalize,
delete, switch_traffic) registered theirs with no labels at all -- so a cancel's
rollback plan, though registered and current, was unfindable, and the page
showed an empty canvas for the ~13 seconds between the entity saying
"cancelling" and the agent picking the rollback up, then popped every step in
at once.

Same grammar as the deploy jobs: `entity` + `scope.provider` scope the search;
`operation` names the lifecycle act -- the word the deployment entity's own
statuses use (rollback, finalize, delete, switch-traffic) -- where a deploy job
uses `strategy`.
…find their plans

The deployment lifecycle jobs already register with identity labels; the scope
lifecycle jobs (create, update, delete) registered with none -- so no surface
can find a scope operation's plan before its run exists. Same grammar:
entity + operation + scope.provider.
@fedemaleh
fedemaleh merged commit 178e86a into beta Sep 2, 2026
3 checks passed
@fedemaleh
fedemaleh deleted the feat/trace-instrumentation branch September 2, 2026 18:27
fedemaleh added a commit that referenced this pull request Sep 2, 2026
docs(changelog): tracing for k8s scope and deployment workflows (#215)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants