Skip to content

fix(k8s/diagnose): keep the results payload out of the argument list - #232

Merged
ignacioboud merged 1 commit into
betafrom
fix/diagnose-argv-limit
Sep 2, 2026
Merged

fix(k8s/diagnose): keep the results payload out of the argument list#232
ignacioboud merged 1 commit into
betafrom
fix/diagnose-argv-limit

Conversation

@ignacioboud

@ignacioboud ignacioboud commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Problem

diagnose on k8s scopes dies right after the application-log check:

info: ✓ Collected application logs from 2 of 2 problematic pod(s)
info: 🚀 Executing step: notify_check_results
info: .../diagnose/utils/diagnose_utils: line 369: /usr/bin/jq: Argument list too long
info: error: exit status 1

notify_results passed the aggregated results to jq as one --argjson value, then the body to np as one --body value. Linux caps a single execve argument at 128 KiB, and ulimit does not lift it — so there is no runtime escape.

The xargs on the line above already guarded the file list. The payload four lines later was not guarded.

It surfaced now because #181 added logs/application_log_evidence and raised that check's logs[] cap from 20 to 200 lines. kubectl logs --tail=N bounds line count, not line length, so two pods of structured logs clear 128 KiB easily.

Net effect: diagnose collected everything and published nothing.

Fix

Keep the payload on disk end to end: xargs (unchanged), --slurpfile instead of --argjson, and a file path for np --body.

The .json suffix on the temp files is required — np reads --body from disk only when the value ends in .json, and otherwise sends the string itself. A bare mktemp path would have shipped the path as the body: same outage, but silent.

notify_results now also propagates np's exit code and fails explicitly if grouping or body-building breaks, instead of continuing with an empty variable.

Separately, lines_to_json_array truncates lines past EVIDENCE_LOG_LINE_MAX_CHARS (default 2000). Every log tail passes through it, so this covers read_log_tail and update_check_result at once.

Tests

3 new tests in diagnose_utils.bats. The regression test uses a 2 MB payload so it fails on macOS too, where the limit is a total argv size rather than a per-argument cap. Its np mock mirrors the real .json-suffix rule — a looser [[ -f "$arg" ]] mock passes against a broken fix.

146 tests, 0 failures across diagnose_utils, notify_diagnose_results, notify_check_running, evidence_schema, logs/application_log_evidence, build_context, kubectl_logs, kubectl_get.

Not run locally: the networking/, scope/ and service/ suites shell out to real kubectl and hang without a cluster — pre-existing, left to CI.

Note

The bash: line 1: 3: Bad file descriptor line in the original report is not from this repo (no fd 3 anywhere in the tree); it comes from the agent wrapper that runs the steps. Unrelated.

🤖 Generated with Claude Code

@ignacioboud
ignacioboud force-pushed the fix/diagnose-argv-limit branch 2 times, most recently from 175a850 to c65c801 Compare September 2, 2026 17:16
notify_results passed the aggregated check results to jq as a single
--argjson value and then to np as a single --body value. Linux caps one
execve argument at 128 KiB, which ulimit does not lift, so once a check
began publishing application log tails the step died with "jq: Argument
list too long" and diagnose published nothing.

Keep the payload on disk end to end: xargs for the file list (already
the case), --slurpfile instead of --argjson, and a file path for
np --body. The .json suffix on the temp files is required — np reads
--body from disk only when the value ends in .json, and otherwise sends
the string itself as the request body.

Also cap log lines at 2000 characters in lines_to_json_array. kubectl's
--tail bounds how many lines we collect, not how long each one is, so a
single serialized stack trace could still dominate the payload.

Tests use a 2 MB payload so they fail on macOS too, where the limit is a
total argv size rather than a per-argument cap.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ignacioboud
ignacioboud force-pushed the fix/diagnose-argv-limit branch from c65c801 to 750eb12 Compare September 2, 2026 17:34
@ignacioboud
ignacioboud merged commit be05f36 into beta Sep 2, 2026
3 checks passed
@ignacioboud
ignacioboud deleted the fix/diagnose-argv-limit branch September 2, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants