feat: add pprof/ collector for Palette control-plane profiling by pavansokkenagaraj · Pull Request #41 · spectrocloud/support-tools

pavansokkenagaraj · 2026-09-02T13:48:10Z

What

Adds pprof/collect-pprof.sh and its README — a self-contained script that collects Go pprof
profiles, execution traces, /metrics, and pod/deployment context from Palette's
palette-controller-manager and cluster-management-agent on a workload cluster, into a single
tarball to attach to a Palette support ticket.

Scope — the README calls this out up front

Only two Deployments on a workload cluster expose Go pprof profiling behind PROFILING=enable:

Deployment	Ports profiled
`palette-controller-manager`	`manager` (:8080), `atop-manager` (:8082)
`cluster-management-agent`	:8082 (BasicAuth)

Hubble services, CAPI infrastructure providers (CAPA/CAPZ/CAPV/…), cert-manager, and any user
workload are out of scope — they do not honour this environment variable, and this script does
not attempt to profile them.

Why this belongs in support-tools

The intended runner is a support engineer or a customer, not us.
Fully self-contained: bash 3.2+, kubectl, curl, tar, POSIX userland. No python3, no
base64 -d, no seq. Runs unchanged on macOS and Linux.
Every external tool it uses is checked at startup — a missing utility fails in the first second
rather than mid-collection.
Downloadable directly from raw.githubusercontent.com in the same shape as the existing
support-bundle scripts.

Two-phase workflow, because enabling profiling restarts the pod

PROFILING=enable lives on the pod template, so setting it rolls the pod. That is exactly the
state you usually want to profile, and it costs real investigations — every profile ends up taken
seconds after container start, measuring warm-up instead of steady state, and labelled Prometheus
series don't exist yet because they're only registered on first increment.

The README leads with -P (enable-and-exit), then a wait for the workload to re-establish, then a
regular collection. If profiling is already on, one call is enough.

Behaviour highlights

Auto sample schedule. 0/300/900s when we had to enable profiling (three points distinguish
warm-up from a sustained baseline); a single as-found sample when it was already on.
Container age validity. Every sample records the container age at capture time, and the
bundle's VALIDITY verdict is derived from those recorded ages — not re-queried at archive
time. A pod restart mid-run will not mislabel a warm bundle as a cold start.
Restores what it found. EXIT trap disables what it enabled (also on Ctrl-C / failure).
-D forces off, -k keeps on, -e no never writes to the Deployment.
Won't archive an empty bundle. Refuses if no sample-* directories were produced.

Testing

End-to-end against a real workload cluster:

Both Deployments, single sample: bundle written, all three targets present with real content.
Ally BasicAuth decoded through kubectl's go-template base64decode (not base64 -d) → 206 KB
heap profile, not an empty 401 body.
-p with two active Deployments → single accurate error naming both.
-p <bogus-pod> with -d → single error, points at the pod and namespace, no misleading follow-up.
-p <real-pod> with -d → honoured, bundle written for one Deployment only.

Also verified: prereq check fires on a missing tool, and secret decoding via go-template handles
uppercase / lowercase / missing / special-character-password cases correctly.

Adds pprof/collect-pprof.sh -- a self-contained, customer-runnable script that collects Go pprof profiles, execution traces, /metrics, and pod/deployment context from Palette's palette-controller-manager and cluster-management-agent on a workload cluster, and writes a single tarball to attach to a support ticket. Only these two Deployments honour the PROFILING=enable env var; Hubble services, CAPI infrastructure providers, and other components are out of scope for this method. The README says so up front. Runs in a bare bash 3.2+ shell (macOS-compatible) with standard POSIX userland only. No python3, no `base64 -d` (GNU-only), no `seq`. Every external tool the script uses is verified up front, so a missing utility fails in the first second rather than mid-collection. Notable behaviour: - Two-phase workflow (-P): enable PROFILING and exit, so the workload can re-establish before collection. Otherwise every sample measures cold start rather than the state under investigation. - Auto sample schedule: 0/300/900s when the script had to enable profiling (three points distinguish warm-up from a sustained baseline), or a single as-found sample when profiling was already on. - Every sample is stamped with the measured container age at capture time, and the README's validity verdict is derived from those recorded ages -- not re-queried at archive time, so a container restart during the collection window can't mislabel a warm bundle as cold. - Restores the Deployment to the state it found via an EXIT trap that also runs on Ctrl-C or failure.

Copilot

🟡 Changes recommended

There are user-facing interface/documentation mismatches (and one env var advertised but unused) that can confuse operators and should be corrected before release.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a self-contained pprof collection utility to support-tools to help support engineers/customers gather Go pprof profiles, traces, /metrics, and Kubernetes context from Palette workload-cluster control-plane components into a single tarball for Support tickets.

Changes:

Added pprof/collect-pprof.sh to collect multi-sample pprof + metrics + context and bundle results.
Added pprof/README.md documenting scope, prerequisites, workflow, and bundle contents.
Updated repo README.md to advertise the new pprof tool and quick start.

File summaries

File	Description
README.md	Adds a new “pprof” section linking to the script and its documentation.
pprof/README.md	New end-user documentation for running the pprof collector safely (two-phase workflow, options, contents).
pprof/collect-pprof.sh	New bash collector script that enables profiling (optionally), port-forwards, captures pprof/metrics/context, and archives results.

Review details

Files reviewed: 3/3 changed files
Comments generated: 3
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Root README.md Adds table of contents; fills in the previously empty support-bundle section with per-script purpose, prereqs, "runs on", and quick-starts matching the sibling READMEs; hoists the pprof scope callout out of the linked doc so the exclusion (Hubble, CAPI, cert-manager, user workloads) is visible from the index. pprof/README.md Adds a copy-paste "For Palette Support engineers" message the support engineer sends to a customer verbatim -- numbered 5-step flow that keeps the two-phase order explicit (a compressed one-command version measures cold start), plus a get-nodes sanity check to catch a wrong kubeconfig before it costs a restart. Fixes the -s doc: -s is the CPU profile window; the trace length is set independently by TRACE_SECONDS (default 5) and is deliberately not on a flag -- 30s of trace on a busy controller is hundreds of MB. Copilot review comment. pprof/collect-pprof.sh Drops CONTAINER_SELECTOR from the interface: it was advertised as an environment override but selector_for() hardcoded '*manager', so setting it did nothing. The hardcoding is deliberate (setting PROFILING on the kube-rbac-proxy sidecar is pointless and breaks the rollout wait), so removed the knob rather than wired it. Comment on selector_for records why. Copilot review comment. Aligns the -s help text with the -s README fix above.

Copilot

🔵 Needs a closer look

The new collector script contains at least one confirmed correctness issue in its port-forward failure diagnostics (container name vs label) that can mislead users and produce invalid troubleshooting commands.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

pprof/collect-pprof.sh:525

The port-forward failure diagnostics use PF_LABELS (the output label) as the container name in messages and the suggested kubectl logs -c command. For cluster-management-agent, the label is cma but the container is cluster-management-agent, so the suggested command will fail and the message is misleading.
pprof/collect-pprof.sh:728
write-readme assigns to rc (and uses i) without declaring them local, which can unintentionally clobber similarly named variables used elsewhere in the script and makes the function less self-contained.

Files reviewed: 3/3 changed files
Comments generated: 0 new
Review effort level: Lite

Copilot

🟡 Changes recommended

Several correctness, portability, security, and profiling lifecycle issues could cause unintended restarts, leaked credentials, or invalid output.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (6)

pprof/collect-pprof.sh:697

fetch writes these files under $TMPDIR/$SUB, but this copy reads and writes at the bundle root. The source therefore does not exist and the silently ignored copy never creates the promised compatibility filename.

    cp "$TMPDIR/${TARGET_LABELS[$i]}-metrics-t1.txt" \
       "$TMPDIR/${TARGET_LABELS[$i]}-metrics.txt" 2>/dev/null || true

pprof/collect-pprof.sh:798

-maxdepth is not supported by the BSD find shipped with macOS. On a stated supported platform this command yields no count (stderr is hidden), so every successful run is rejected as an empty bundle. Count the first-level sample directories with shell globbing instead.

  local n
  n=$(find "$TMPDIR" -maxdepth 1 -type d -name 'sample-*' 2>/dev/null | wc -l | tr -d ' ')

pprof/collect-pprof.sh:865

-c only changes the schedule; enable-profiling still runs first and, with the default auto, sets PROFILING and rolls an unprofiled deployment. This contradicts the option's “no restart” guarantee. Force enablement off after all options have been parsed so option order cannot override the guarantee.

    c) CURRENT_ONLY=yes ;;

pprof/collect-pprof.sh:914

The explicit pod and single-deployment checks happen only after profiling is enabled. A typo in -p, or using -p without narrowing -d, therefore rolls one or both production Deployments before reporting an argument error, then rolls them again during cleanup. Validate POD before this loop.

for dep in "${ACTIVE_DEPLOYS[@]}"; do
  enable-profiling "$dep"
done

pprof/README.md:197

This customer workflow promises cleanup, but phase 2 treats the phase-1 setting as pre-existing and leaves it enabled. Update these instructions after the script can carry phase-1 ownership into phase 2; simply adding -D could incorrectly disable profiling that was enabled before phase 1.

  # 5. Phase 2 -- collect. No further restart. ~5 minutes.
  ./collect-pprof.sh

pprof/collect-pprof.sh:920

This generated follow-up drops any -d selection from the prepare run. After -P -d cluster-management-agent, following the printed command can enable and restart palette-controller-manager as well, despite the claim on the next line. Preserve the deployment list in the command.

  techo "  collect-pprof.sh -n $NS"
  techo "That run will NOT restart anything, so its numbers are a real baseline."

Files reviewed: 3/3 changed files
Comments generated: 7
Review effort level: Balanced

+  code=$(curl -s --max-time $((CPU_SECONDS + 60)) "${extra[@]}" \
+           -o "$TMPDIR/$out" -w '%{http_code}' \
+           "http://localhost:$port/$path" || echo 000)


+# Phase 2: collect. No further restart.
+./collect-pprof.sh


+  date -u "+%Y-%m-%d %H:%M:%S"
+}
+function techo() {
+  echo "$(timestamp): $*"


+function profiling-state() { # profiling-state <deployment>
+  kubectl get "deployment/$1" -n "$NS" \
+    -o jsonpath='{range .spec.template.spec.containers[*]}{range .env[?(@.name=="PROFILING")]}{.value}{"\n"}{end}{end}' \
+    2>/dev/null | grep -q enable && echo yes || echo no


+        kubectl set env "deployment/$d" -n "$NS" -c "$sel" PROFILING- >/dev/null 2>&1
+      elif [ "$FORCE_DISABLE" = yes ] && [ "$(dep_get "$d" was_set)" = yes ]; then
+        techo "[$d] disabling pre-existing PROFILING (-D). This restarts the pod."
+        kubectl set env "deployment/$d" -n "$NS" -c "$sel" PROFILING- >/dev/null 2>&1


+for dep in "${WANTED_DEPLOYS[@]}"; do
+  if kubectl -n "$NS" get "deployment/$dep" >/dev/null 2>&1; then


+    manager-*.pb.gz            heap, allocs, goroutine, threadcreate, block,
+    atop-manager-*.pb.gz       mutex, cpu (30s), execution trace, /metrics
+    cma-*.pb.gz                (twice, 60s apart)


pavansokkenagaraj requested a review from a team as a code owner September 2, 2026 13:48

pavansokkenagaraj requested a lite review from Copilot September 2, 2026 13:51

Copilot started reviewing on behalf of pavansokkenagaraj September 2, 2026 13:52 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread pprof/collect-pprof.sh Outdated

Comment thread pprof/collect-pprof.sh

Comment thread pprof/README.md Outdated

pavansokkenagaraj requested a lite review from Copilot September 2, 2026 15:00

Copilot started reviewing on behalf of pavansokkenagaraj September 2, 2026 15:01 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

pavansokkenagaraj requested a balanced review from Copilot September 2, 2026 15:09

Copilot started reviewing on behalf of pavansokkenagaraj September 2, 2026 15:09 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat: add pprof/ collector for Palette control-plane profiling - #41

feat: add pprof/ collector for Palette control-plane profiling#41
pavansokkenagaraj wants to merge 2 commits into
mainfrom
feat/pprof-collection

pavansokkenagaraj commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		for dep in "${WANTED_DEPLOYS[@]}"; do
		if kubectl -n "$NS" get "deployment/$dep" >/dev/null 2>&1; then

Uh oh!

Conversation

pavansokkenagaraj commented Sep 2, 2026

What

Scope — the README calls this out up front

Why this belongs in support-tools

Two-phase workflow, because enabling profiling restarts the pod

Behaviour highlights

Testing

Related

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Suppressed comments (2)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (6)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants