Conversation
…h-annotations fix(k8s): stop hardcoding the CloudWatch logging annotations - Beta
* fix(k8s-logs): honor the requested end time so agent log queries stop returning today's lines * docs(changelog): record the k8s log end-time fix * test(k8s-logs): cover the log time-range contract and reject non-numeric bounds * perf(k8s-logs): stop reading a pod's stream once it passes the requested end time * fix(k8s-logs): reject a window bound that is not RFC3339 instead of ignoring it * docs(changelog): cover bound validation and the early stop * fix(k8s-logs): keep an instance's cursor when it contributes nothing to a log page The pagination token was regenerated from the entries of the current page alone, so an instance with no new lines — or whose remaining lines fall past end_time — lost its position. determineSinceTime then fell back to start_time on the next page and the instance re-read the window from the beginning, re-delivering lines the caller had already seen. With more than one instance the pages take turns evicting each other and pagination never reaches the end of the range. GenerateToken now carries the incoming cursors forward. An empty page still yields an empty token: that is how the caller learns there are no more pages, so the cursors are deliberately not carried across it. The ordering, the limit cut and the token move into pagination.Page. The token records the newest entry kept per instance, and that is the cut point only because the entries are sorted ascending and the cut keeps the oldest ones — keeping the three steps in one function documents that coupling and makes it testable outside package main. Linux binaries rebuilt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(k8s-logs): cover the log pagination contract end to end The multi-page test walks a window page by page through the production functions — determineSinceTime, getLastReadTime, ProcessLinesFromChannel and pagination.Page — and fakes only the Kubernetes stream, which is the one thing that cannot run here. It asserts that paging terminates, that every line in the window is delivered exactly once and in order, and that a line past end_time is never delivered. One of its instances has no lines inside the window at all: that instance never earns a cursor and is re-read on every page, which is inherent to a timestamp cursor and cheap because the processor stops at its first line past the window. Both tests fail against the previous behavior — the multi-page one never terminates and delivers some lines five times. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(changelog): fold the log time range entries into one The branch had accumulated one entry per commit for what a reader sees as a single fix: the range you select is the range you get. They collapse into one entry that names the symptoms first and the resulting behavior second. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * build(k8s-logs): rebuild the shipped binary on a current Go and dependencies govulncheck reported 40 vulnerabilities in the committed binary — 31 from the standard library, 8 from golang.org/x/net, 1 from golang.org/x/text — of which 17 are reachable from this code: TLS handshake and x509 chain-building issues on the API server connection, net/url parsing, and an infinite loop in x/text. The x/net/html findings that Dependabot flags for this module are present but not reachable, since the binary never parses HTML. The standard library is linked into the binary, so the binary carries the standard library of whatever compiler built it, and the go directive was a floor low enough that a machine ten patch releases behind satisfied it. Raising the directive to the version that fixes these makes the go command fetch a matching toolchain, so a rebuild cannot quietly ship an older standard library again. x/net and x/text move to their fixed versions; x/sys and x/term follow as their requirements. Verified: govulncheck reports no vulnerabilities in source mode and against all three rebuilt binaries, and go vet, go test and the bats suite are clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: pin the Go version to the one the log module requires The loose '1.25' let setup-go keep the Go that ships with the runner image, which is behind the version go.mod now requires, and setup-go pins GOTOOLCHAIN=local so the go command is not allowed to fetch the one it needs: `go vet ./...` failed with "go.mod requires go >= 1.25.13 (running go 1.25.12; GOTOOLCHAIN=local)". Pinning the exact version keeps the checks on the same toolchain the shipped binaries are built with, which is the point of raising the directive in the first place. The shared workflow cannot read this module's go.mod instead — its setup step hardcodes go.mod at the repository root, and this module lives in a subdirectory — so the version lives in two places and has to move in both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(changelog): state the log range fix functionally, in two lines The entry had grown into a paragraph about how the fix works. This file is the source for the customer release notes, so it should say what changed for the user and nothing about the mechanism. The bound-validation line is dropped: an error message for an input the console does not let you type is not release-note material. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(k8s-logs): cut the commentary down to what the code cannot say Review asked for it: most of these restated the code or narrated a decision, and this is a public repository. What is left is one or two lines each, kept only where a reader would otherwise have to reconstruct something — why the upper bound is applied client-side, why the stream read breaks instead of continuing, what previous is for in the token, why the producer needs cancelling, and in bash the numeric guard before bc and the SECONDS rename. The comment above the bound validation went entirely: it described what would happen without the check while sitting next to the exit(1) that prevents it, which reads as a contradiction. The bats banner comments stay as they are, they match every other test file in the repo. Binaries rebuilt: dropping lines moves the line numbers recorded in the binary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: drop the commentary from the Go version pin Review asked for it. The version and the file it has to match are visible in the diff. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Ignacio Boudgouste <ignacio.boudgouste@nullplatform.io> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
… artifacts Manually-verified publish-images workflow (v* tag trigger) that builds each scope worker image via the org reusable ECR workflow and registers it as an oci_image platform artifact (visible-to organization=*), fanned out over the 3 images: containers (k8s base), scheduled-task, containers-datadog (metric overlay). Uses the existing NP_API_KEY secret. Rebased onto beta so the PR carries only the feature (no main-only drift such as the cloudwatch deployment annotations). CHANGELOG: Publish containers and scheduled task scopes as docker images.
ci: publish scope worker images + register artifacts (manual trigger)
Tags produced ECR images and registered artifacts, but no GitHub release — the published digests were invisible to consumers. Add a finalize job that upserts a release for the tag whose body carries an Artifacts table for all three images (containers, scheduled-task, containers-datadog): tag, digest, and the copyable pinned image@digest reference. Idempotent: an existing release only gains the table once, and re-runs with the same digests no-op. Same information the chained release pipeline writes in scopes-lambda (actions-nullplatform release-publish-oci); hand-rolled here because this repo fans out three images with an ordering dependency and its tags are human-pushed rather than cut by release-please. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ci: publish GitHub release with artifact metadata on every tag
…ntainer-orchestration provider Adds a provider-based layer for the traffic-manager sidecar version, mirroring the pattern already used by main_traffic_manager_port. The container-orchestration provider's traffic_manager.version now sets the default image tag for non-websocket scopes, while web_sockets scopes keep forcing websocket2 regardless of the provider. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…n-provider feat(k8s): allow cluster-wide traffic-manager version override via container-orchestration provider
The platform now nests each custom-scope workflow inside its provisioning run (SWM injects the trace carrier into the action parameters; the np CLI adopts it and traces one step per workflow fragment). What was missing is the WHY: a failed step read as "exited with status 1" while the real reason scrolled past in the log. One seam fixes that structurally: `log error` — every one of its 700+ call sites across the k8s scripts now also lands the message as a tracing.error facet ON the step it happened in, via the vendored shell SDK adopting NP_TRACE at call time. A silent failure (no log error on the way down) is covered by an ERR/EXIT trap pair that reports the failing command itself. Failures in a custom scope now read exactly like native ones: which step, and what went wrong. The two long waits (ALB active, deployment active) also mark their step `waiting` with live progress labels, flushed per beat with a hard 2s bound so a down tracing API costs at most ~6% of a wait's poll budget and never extends its wall-clock timeout. All of it is best-effort and inert by default: no nptrace.sh, no NP_API_KEY, or no NP_TRACE means plain logging, byte-identical to before (asserted byte-for-byte in tests). Heartbeat call sites are guarded so overrides reusing these scripts without k8s/logging stay clean.
…ait loop Phases inside a script can now be their own step in the trace: np_scope_step_begin/end/timeout open a keyed sub-step under the platform step, log errors and heartbeats attach to the innermost open node, a sub-step still open when the shell dies inherits the shell's outcome, and a wait that hits its deadline closes as timed_out rather than failed. Every wait loop is instrumented — wait_for_alb and wait_deployment_active upgraded from bare heartbeats to sub-steps (rollout heartbeats now carry desired/launched/ready/available/updated replica counts as labels), and wait_on_balancer, verify_ingress_reconciliation and verify_http_route_reconciliation gain sub-steps and heartbeats. scheduled_task/logging picks up the same tracing hooks, so scheduled-task scopes trace like k8s ones. The blue-green switch needs no iteration steps: it is one-shot, and switch_traffic.yaml's named steps already are the plan.
Custom scopes now report the same data lineage the scope-workflow-manager
does, by the same canonical dataset ids so the graphs join by value:
• apply_templates turns kubectl apply output into produces edges for the
kinds the platform lineage model knows (k8s-deployment, k8s-service,
k8s-ingress) — one chokepoint covers every workflow
• the namespace create records k8s-namespace:<ns>
• manage_dns CREATE records dns-record:<fqdn>
• wait_for_alb records the load-balancer:<arn> dependency
• wait_deployment_active records the cross-flow join back to the build
(consumes <asset.type>:<asset.url>), produces this deploy's log dataset
with the concrete log query as its pointer, declares the deploy-log
affordance the UI renders as "view logs", and reports rollout progress
({current, target, unit: instances}) per heartbeat
Vendored nptrace.sh rebuilt from catalog-tracing-sh#6, which adds
np_trace_produces/consumes/affordances/progress to the shell SDK.
…n catalog-tracing-sh#6
…r custom scopes The workflows declare each step's trace identity (np CLI trace: blocks) using the exact step keys, titles and plan groups the platform's own scopes emit — create-namespace, create-ingress, wait-for-ingress, create-dns for the scope provision; wait-for-instances (group waiting-instances, where the dashboard attaches the instance pips) and traffic-switch (group switching-traffic) for deployments. Step NAMES are untouched: customer overrides anchor on them. The actual create-* deploy steps are emitted per applied manifest by apply_templates as keyed sub-steps (deployment-*.yaml → create-deployment, secret-* → create-secret, scaling-* → create-hpa, service-* → create-service, pdb-* → create-pod-disruption-budget, ingress-* → configure-ingress/create-ingress) — the custom flow renders then bulk-applies, so this is where those resources really get created. Plumbing (load logging, assume role, renders, context builds) is trace: false; route53-only stations carry flavors: [route53] so an AKS or external_dns scope never declares a station it cannot reach; and scripts whose branch isn't taken declare np_step_skip so their step closes skipped, never a hollow completed. Requires nullplatform/cli#216 for the trace: blocks and np_step_skip; both degrade to exactly today's behavior on an older CLI (unknown YAML keys are ignored; np_step_skip is guarded).
…h nothing to verify
K8S_FLAVOR names the cluster variant (eks in the base values, aks and aro in the overlays) — an independent dimension from DNS_TYPE, so the workflows declare both as trace flavor tokens (flavors: ["$K8S_FLAVOR", "$DNS_TYPE"]) and each step gates on the dimension it genuinely depends on.
…t even as skipped The trace is for users: verify_networking_reconciliation has nothing to dispatch on azure/aro, so it is flavor-gated out of those plans entirely, like the ALB stations. skipped stays reserved for declared steps whose branch genuinely may or may not run on that flavor (a namespace that already exists).
k8s-scope-create, k8s-deployment-initial and k8s-deployment-switch-traffic carry their plans as previewable job definitions (version derived from the plan hash per flavor), and each run links instance_of — the same plan_source: job model the platform's own scopes use.
…plains, traffic set
wait_deployment_active now reports everything the platform's own
wait-for-instances does, from the data the script genuinely observes: the
'instances' counted io (healthy/launched/desired, plus per-pod restart
history from the pods' containerStatuses), the instances-health affordance
meter with unhealthy counts and probe reasons, and a per-heartbeat explain
at the operator's severity — warn while probes fail or restarts accumulate
('All 3 instances healthy — after 4 restarts' keeps a healed crash-loop
visible), plain while booting, error with structured details when it gives
up.
The blue/green switch reports the same set the platform's tracedTrafficSwitch
emits — labels, the from→to narrative, the traffic-switch affordance with
the resulting split, the request as input, what landed as output, and the
convergence toward 100% — from apply_templates, gated explicitly by
TRACE_TRAFFIC_SWITCH on the switch_traffic step.
Vendored nptrace.sh rebuilt from catalog-tracing-sh#7 (inline io +
error --details).
…lavor, the run's own reason Every workflow now declares its trace identity, not just the three lifecycle ones: blue_green overrides the job it would otherwise inherit from initial (k8s-deployment-blue-green, with its own groups), update overrides create's (k8s-scope-update), finalize/rollback/delete and the scheduled-task trigger get their own jobs, finalize's rollout wait is keyed wait-for-instances, and the plumbing steps (load logging, assume role, deployment build context) are trace: false across all twenty-odd workflow files. A fatal exit now mirrors the REAL reason onto the RUN as well as the step — the last log error's message, not the exit mechanism — so list views name the cause without drilling into steps. A run-level NP_TRACE (no step segment) mirrors nowhere extra.
…y theirs Verified by executing the failure stories end to end: a Forbidden apply previously traced as a bare 'Failed to apply' — kubectl's stderr (the principal, the resource, the namespace) never reached the trace. The capture now includes stderr, and the failure names the manifest and the server's reason. Restart narratives gain their cause the same way: '4 restarts so far (OOMKilled)' instead of just the count, with restart_reasons on the instances-health meter.
…he app's own
Verified by executing the failure stories: an ImagePullBackOff now traces
the registry's verbatim message ('Back-off pulling image …: manifest
unknown') live within one poll and cause-first at the terminal; a
crash-loop attaches the previous container's last log lines and leads with
the app's own final line ('FATAL: bind: address already in use'). The
classification is live POD STATE (waiting reasons and messages), so
problems that predate the wait or throttle their events are still named —
the event sweep only fills in what state cannot say (which probe, which
path). Boot churn is never presented as a problem, and the narrative
re-evaluates every poll, emitting only when the situation changes.
The 'possible causes / how to fix' bursts no longer shadow the cause:
facets fold last-writer-wins, so recording each hint line as the error
left the LAST HINT as the step's message. The burst's first message is now
the error everywhere (step, run mirror, terminal — one cause-first message,
mechanism in parentheses) and the hints ride as structured details.hints.
End-to-end verification (real CLI executing the real create workflow against a wire sink) caught scripts that capture the provider's error output and then log a generic message: build_service_account, create_role (OIDC, account id, create-role, policy attach), delete_role and require_resource. Each now carries the captured reason on the traced error, apply_templates-style. Trace error messages also drop the console decoration (indentation, the marker emoji) so the cause reads clean on the trace while console output stays exactly as it was.
The tracing SDK now rides as a submodule pinned to the SDK's release history instead of a hand-vendored copy of nptrace.sh. The logging hooks look for the submodule first and still fall back to a root-vendored nptrace.sh, so repo copies that lose the submodule (archive downloads, docker build contexts) keep working — and with neither present the scripts degrade to plain logging exactly as before. CI checks out submodules so the trace tests exercise the real SDK.
…p_error When the engine's preamble publishes np_step_error, the first log error cause of a step is stated through it, so the engine's own failure terminal carries the real diagnosis instead of a bare exit status — one message on the wire no matter which side reports last. The exit trap's synthetic mechanism message never states: it must not outrank the engine's own evidence (the shell's last stderr words).
A build that loses the tracing submodule (archive download, docker build context) used to degrade to plain logging in SILENCE — the CLI side traces, the structure looks fine, and the scope-side story (sub-steps, live waits, real errors, lineage, affordances) is just absent, with nothing anywhere saying why. Exactly the failure nobody sees. Now, when the engine is tracing the run (NP_TRACE) and credentials are present (NP_API_KEY) but nptrace.sh is not in the bundle, the workflow log carries one loud warning naming the missing file. Untraced runs stay silent as before.
Validated against real blue/green deployments on a custom scope, side by side with the native experience: - finalize declares its own group — its steps used to file under Setting up / Waiting for instances, twins of the deploy workflow's stations, and the ending never got a station of its own. The entity ending overlay names it (Finalized / Rolled back). - rollback likewise: one Finalize group — a rollback's wrap-up is the ending station's story. - switch_traffic scopes ALL its steps under Switching traffic: per-increment runs fold there as attempts (the native reading) instead of re-lighting Setting up / Waiting on every increment. - the wait steps get workflow-distinct keys (finalize-instances-check, switch-instances-check): three workflows declared the same wait-for-instances key, so the fold merged semantically different waits into one row across runs.
Every step a user sees carries a clear title in the native tone (Validate load balancer capacity, Promote new deployment, Restore traffic routing, Remove previous deployment); build/template plumbing is trace: false. rollback gains its full trace pass (it had none). Steps injected by the overrides repo keep their keys — the dashboard humanizes an untitled key rather than showing snake_case.
… no submodules The agent materializes the package source with a plain clone, so the catalog-tracing-sh submodule arrives empty and every run logged 'tracing SDK not bundled' (observed live: zero step io on three deployments that declared it). The logging loader already prefers the submodule and falls back to a root copy — ship the copy. Kept in sync by the existing re-vendor chore. Also: blue_green/initial declare placeholder groups (switching-traffic, finalize) so the whole journey renders pending from the first paint — paired with the CLI emitting an optional placeholder step per empty declared group.
io lineage edges (image consumed, logs/ingress/workload produced) were
silently dead-lettering: the binding carried the io descriptor's
kind/uri, which the API's tracing.binding schema rejects. The bumped
SDK sends {name} on the edge and keeps the full descriptor on the node.
np_scope_wait_heartbeat filed per-poll diagnostics (wait.what/ready/ state/desired/..., including an unresolved-config literal) as LABELS, which the dialog renders as tag chips — machine noise on a human surface. What a phase waits on is the tracing.signal facet: name + direction + deadline; the counts already ride the meter and progress facets, elapsed rides the node's own clock.
The io names said 'workload' — a k8s category, not a manifest kind. The dialog now names what kubectl actually reported: deployment (and removed-deployment on deletes); the dataset ids were always exact (k8s-deployment:<ns>/<name>).
Every workflow's trace.job carries namespace: @context:scope.provider — the scope's service specification. Two providers shipping the same workflow name with an identical plan hash previously converged on ONE job definition (namespace 'workflows') and would fight over its identity labels; per-provider namespaces make each scope type own its jobs by construction.
The page titled these runs by humanizing `labels.workflow` — this file's basename — which meant the name on screen was whatever the file happened to be called. `trace.title` says it outright, landing on the run's explain.title where every consumer looks first.
…ow stand Four assertions still described the behaviour these changes replaced, so the branch was red on work that is correct: - a wait states `tracing.signal` (name, direction, deadline in ms), not `wait.*` labels — the labels were rendering as code-y chips on the phase; - an edge's binding names WHICH io the edge is about and nothing more (sending the whole descriptor is what dead-lettered every lineage edge); - progress units are the closed set, so `count` stays `count`. The extra-k=v-labels case is replaced by its inverse — a guard that no wait bookkeeping reaches the labels — and a wait whose timeout never resolved is now covered too.
catalog-tracing-sh#10 is merged, and its branch is gone. The submodule pointed at a commit that now only exists inside main's history — the vendored content is identical, but .gitmodules declares `branch = main`, so the pointer should be main's tip.
… stop advertising work they cannot do 'Promote new deployment' (and rollback's 'Restore previous deployment') run scale_deployments, which is a no-op for every strategy but rolling — so on a blue-green scope they were permanently grey rows nobody could explain. The scope's deploy strategy becomes a flavor DIMENSION, read from the same context path the scripts read, and those steps gate on `rolling`. One job still, two plan versions — which is exactly what hashing the effective plan is for. Every remaining finalize/rollback step now carries a one-line description, so the checklist reads without prior knowledge of the workflow.
The console hints already classify a failing wait — 'The application did not pass its health check at /health-bad. Detected: Startup probe — app responded with HTTP 404 (expected 2xx).' plus a targeted fix — but that never reached the trace, so a reader who opened the phase saw counts and a timeout and had to go to the logs for the reason. The same classifier now feeds the step's explain: WHY on the reason, NEXT on the fix. It rides the LIVE narrative too, not just the give-up — the reader asks 'why?' the moment the meter goes amber, not ten minutes later.
The phase read '1 with Startup, probe, failing'. The reason list splits on the comma —
but IFS was ', ', so it split on whitespace too, and the event sweep's already-human
reasons ('Startup probe failing') were shredded into three codes that matched nothing
and were echoed verbatim.
Split on the comma alone, trimming the space that follows it. Codes still translate and
still dedupe after translation.
A container that is OOM-killed reports two things: `state.waiting.reason` CrashLoopBackOff — the mechanism that keeps restarting it — and `lastState.terminated.reason` OOMKilled, why it actually died. Both the classifier and the wait narrative read the waiting reason first, so every surface named the symptom: Waiting for 0/1 instances to be healthy — 1 with crashing repeatedly: back-off 20s restarting failed container=application pod=d-253247585-630235960-dd685c5-trn9s_nullplatform(498dc0ab-…) Reason: The container started and crashed repeatedly. Next: Review application logs for startup errors (…, panics). That sends the reader hunting for a bug in their startup path when the answer is the memory limit — and the OOM branch the classifier already carries could never fire. The OOMKilled fact was on the step all along, three panels down under "Restarted → Reason". Now the backoff wrappers (CrashLoopBackOff, BackOff) defer to the termination reason; every other waiting reason IS a cause and still wins, so ImagePullBackOff is unaffected. The line becomes: Waiting for 0/1 instances to be healthy — 1 with out of memory Reason: The container exceeded its memory limit (256Mi) and was terminated. Next: Increase ram_memory for scope 'Stage' or reduce memory usage. The back-off message is dropped from the narrative — it restates the reason and then buries the line in two ids nobody reads — and what remains is capped to a line. Neither is lost: the verbatim message, the CrashLoopBackOff wrapper (as `reason`) and the new `cause` all stay on the io, so an agent still sees the loop, the ids and the untruncated text.
Auditing the other warn/failure paths after the OOM fix turned up three gaps, one of them introduced by that fix: - `Error` had no branch. Deferring the backoff wrapper to the termination reason sends the ORDINARY crash loop — a panic, a bad config, a missing dependency, all of which terminate as `Error` — to the nameless default: "Pods are failing with reason: Error", with no suggested fix at all. That is worse than the generic wording it replaced. `Error` now carries the crash-loop advice plus its exit code, the first thing anyone greps for. - The restart-only narrative never diagnosed. A container that crashed and is running again sits between waiting states, so nothing classified it, and the branch said only THAT it restarted — an amber phase with no cause and no next step. It calls the classifier like every other surface now. - Evicted, Unschedulable and DeadlineExceeded were in the humanize table but had no branch, so they reached the reader named and unexplained. The unknown-reason default keeps its EMPTY suggested fix: that emptiness is the signal that hands the reader the full generic troubleshooting checklist, which is richer than any line this branch could invent for a code it does not know. Filling it in swapped that checklist for a one-liner.
`ram_memory` is the capability KEY. The scope form calls it "RAM Memory", so "Increase ram_memory for scope 'Stage'" sent the reader looking for a field that is not on the page — the same leak the OOM diagnosis itself was: the right answer, spoken in a vocabulary the reader has no way to act on. Three strings named an internal or foreign identifier: - ram_memory -> "the RAM Memory of scope 'X'". - initialDelaySeconds/timeoutSeconds are the KUBERNETES spellings; the scope's own settings are `health_check.initial_delay_seconds` / `timeout_seconds`, shown as Initial Delay and Timeout. - activeDeadlineSeconds is not a scope setting at all, so it named a field the console does not have. Reworded to say the thing instead of the field. Genuine Kubernetes objects (PVC, configmap, nodeSelector/affinity) keep their Kubernetes names: those are what they are, and renaming them would make them unfindable in the cluster the reader has to go look at. Three tests asserted the old wording; two of them encoded a field key as the user-facing contract, which is what let this ship.
Seven of the sixteen workflows declared no `trace:` block at all, so their runs reached a consumer with every naming rung empty: run_id : workflow-diagnose-395c6e18-5574-4b54-b39e-735541b16aa2 explain : null <- no declared title job : null key : null The title contract then lands on the run id, and a consumer that will not print a uuid substitutes its own neutral word: the deployment page showed the failure diagnostics as "Activity · 21 steps · all succeeded", which names nothing the reader can act on. The name was in `labels.workflow: "diagnose"` the whole time — and that is exactly the label a consumer must NOT title from: it is `filepath.Base` of this file, private to this engine's layout, and the next producer's `workflow` label means something else. So the fix is to say it, not to have it read. Titled: the deployment diagnose and kill_instance workflows, and the scope diagnose, pause/resume-autoscaling, restart-pods and set-desired-instance-count workflows. Every workflow in the package now names itself.
Every check already decides a verdict — 10 call sites report `failed`, 3 `warning` — and builds real evidence: a summary, affected pods, details and suggested actions. All of it went into the results file for the diagnose API and none of it onto the trace. On the wire every one of the 21 check steps read: completed sev=- Memory-Limits err=- out=0 completed sev=- Container-Crash-Detection err=- out=0 So the page showed "21 steps · all succeeded" directly under a deployment that had just died of an OOM — including the two checks that found it. `update_check_result` is the one funnel every check goes through, so the verdict is mirrored there: the finding becomes the step's `explain.what`, its first suggested action becomes `next`, and the evidence rides along as an output so the dialog can show the affected pods. The step's STATUS is deliberately left alone. A check that finds a problem did its job; failing the step would conflate "this check broke" with "this check found something". The finding rides `explain.severity` — error for a failed check, warn for a warning — the same channel every other producer surface uses. A passing or skipped check stays silent: 21 lines of green is how nothing gets read.
Review feedback: these messages are the main output the deploying dev sees,
so a test should fail when one of them changes. Asserting fragments
("Detected: Startup probe", "not yet listening") let a message be reworded
without a single test noticing.
Every pod-diagnostic branch now asserts the complete emitted lines -- the
reason, the detected line, the details, the recent-warnings block and the
suggested fix -- so breaking any one of them is a red test. Verified by
reordering four words inside one message: test 11 fails, where the old
fragment assert passed.
Also drops the explanatory prose comments from the file; the repo is public
and the asserts now say what the tests were explaining.
…heir plans The deploy workflows (blue_green, initial) register their jobs with identity labels, which is how the deployment page shows the whole checklist pending the instant an operation is requested. The lifecycle workflows (rollback, finalize, delete, switch_traffic) registered theirs with no labels at all -- so a cancel's rollback plan, though registered and current, was unfindable, and the page showed an empty canvas for the ~13 seconds between the entity saying "cancelling" and the agent picking the rollback up, then popped every step in at once. Same grammar as the deploy jobs: `entity` + `scope.provider` scope the search; `operation` names the lifecycle act -- the word the deployment entity's own statuses use (rollback, finalize, delete, switch-traffic) -- where a deploy job uses `strategy`.
…find their plans The deployment lifecycle jobs already register with identity labels; the scope lifecycle jobs (create, update, delete) registered with none -- so no surface can find a scope operation's plan before its run exists. Same grammar: entity + operation + scope.provider.
…232) notify_results passed the aggregated check results to jq as a single --argjson value and then to np as a single --body value. Linux caps one execve argument at 128 KiB, which ulimit does not lift, so once a check began publishing application log tails the step died with "jq: Argument list too long" and diagnose published nothing. Keep the payload on disk end to end: xargs for the file list (already the case), --slurpfile instead of --argjson, and a file path for np --body. The .json suffix on the temp files is required — np reads --body from disk only when the value ends in .json, and otherwise sends the string itself as the request body. Also cap log lines at 2000 characters in lines_to_json_array. kubectl's --tail bounds how many lines we collect, not how long each one is, so a single serialized stack trace could still dominate the payload. Tests use a 2 MB payload so they fail on macOS too, where the limit is a total argv size rather than a per-argument cap. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat(k8s): trace-instrument the workflow — every error clear, waits live
chore(tracing): drop the root SDK copy, ship the vendored submodule in images
docs(changelog): tracing for k8s scope and deployment workflows (#215)
andres-nullplatform
approved these changes
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.