Skip to content

feat: add pprof/ collector for Palette control-plane profiling #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pavansokkenagaraj wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat: add pprof/ collector for Palette control-plane profiling #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ef7ec57
feat: add pprof/ collector for Palette control-plane profiling
pavansokkenagaraj Sep 2, 2026
7ad644e
docs: expand root README, add Support handoff, address Copilot review
pavansokkenagaraj Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 86 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,88 @@
# support-tools

## support-bundle
Diagnostic scripts for Spectro Cloud Palette environments. Each tool is self-contained, runs from a bare shell, and produces artifacts you attach to a Palette Support ticket.

## Table of contents

- [Support bundles](#support-bundles)
- [Edge support bundle](#edge-support-bundle)
- [Infrastructure support bundle](#infrastructure-support-bundle)
- [Palette controller profiling (pprof)](#palette-controller-profiling-pprof)
- [Support](#support)

---

## Support bundles

Log and state collection from Palette clusters and edge hosts. Full documentation lives in [`support-bundle/`](support-bundle/) — see [`support-bundle/README.md`](support-bundle/README.md).

### Edge support bundle

Collects system journals, `systemd` service state, `kubectl` cluster info, and Kubernetes resources from an edge host and the cluster running on it. Use this on the **host**, as `sudo`.

- **Script**: [`support-bundle/support-bundle-edge.sh`](support-bundle/support-bundle-edge.sh)
- **Documentation**: [`support-bundle/README-edge.md`](support-bundle/README-edge.md)
- **Prerequisites**: `sudo`, `journalctl`, `systemctl`, `kubectl`
- **Runs on**: Edge host (Linux)
- **Quick Start**:
```bash
# Official SpectroCloud URL
curl -sSL https://software.spectrocloud.com/scripts/support-bundle-edge.sh -o support-bundle-edge.sh
sudo bash support-bundle-edge.sh

# GitHub URL
curl -sSL https://raw.githubusercontent.com/spectrocloud/support-tools/main/support-bundle/support-bundle-edge.sh -o support-bundle-edge.sh
sudo bash support-bundle-edge.sh
```
- **Advanced**: extra namespaces, resources, or `journalctl` units via `-n`, `-r`, `-R`, `-j` flags. See the edge README for the full list.

### Infrastructure support bundle

Collects cluster state, Cluster API (CAPI) objects, and Palette resources from a Kubernetes infrastructure cluster. Use this against a Palette **management** or **workload** cluster's kubeconfig.

- **Script**: [`support-bundle/support-bundle-infra.sh`](support-bundle/support-bundle-infra.sh)
- **Documentation**: [`support-bundle/README-infra.md`](support-bundle/README-infra.md)
- **Prerequisites**: `kubectl` with cluster access
- **Runs on**: Any workstation that can reach the cluster's API server
- **Quick Start**:
```bash
# Official SpectroCloud URL
curl -sSL https://software.spectrocloud.com/scripts/support-bundle-infra.sh -o support-bundle-infra.sh
bash support-bundle-infra.sh

# GitHub URL
curl -sSL https://raw.githubusercontent.com/spectrocloud/support-tools/main/support-bundle/support-bundle-infra.sh -o support-bundle-infra.sh
bash support-bundle-infra.sh
```
- **Advanced**: extra namespaces and resources via `-n`, `-r`, `-R` flags. See the infra README for the full list.

---

## Palette controller profiling (pprof)

Go pprof profiles, execution traces, `/metrics`, and pod/deployment context from Palette's control-plane components on a workload cluster. Produces a single tarball to attach to a support ticket.

**Scope — read this before running.** Only two Deployments on a workload cluster expose Go pprof profiling behind the `PROFILING=enable` environment variable: `palette-controller-manager` and `cluster-management-agent`. Hubble services, CAPI infrastructure providers (CAPA/CAPZ/CAPV/…), cert-manager, and user workloads are **out of scope** — they do not honour this environment variable, and this script does not attempt to profile them.

- **Script**: [`pprof/collect-pprof.sh`](pprof/collect-pprof.sh)
- **Documentation**: [`pprof/README.md`](pprof/README.md)
- **Prerequisites**: `bash` 3.2+, `kubectl`, `curl`, `tar`, standard POSIX userland. No `python3`, no `base64`. Runs on Linux and macOS.
- **Runs on**: Any workstation that can reach the workload cluster's API server
- **Quick Start**:
```bash
# GitHub URL
curl -sSLO https://raw.githubusercontent.com/spectrocloud/support-tools/main/pprof/collect-pprof.sh
chmod +x collect-pprof.sh

export KUBECONFIG=/path/to/workload-cluster.kubeconfig
./collect-pprof.sh
```
- **Impact**: enabling profiling triggers a rolling restart of the affected pod. See the pprof README for the recommended two-phase workflow (`-P` to enable, wait, then collect) that avoids measuring cold-start behaviour instead of steady state.

---

## Support

- Open an issue in this repository for bugs, questions, or contributions.
- Contact Spectro Cloud Support for help using these scripts on a specific cluster.
- Each script has its own detailed README linked above — flags, output layout, and troubleshooting live there.
218 changes: 218 additions & 0 deletions pprof/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
# Palette Controller Profiling Collection

`collect-pprof.sh` gathers Go pprof profiles, execution traces, controller metrics, and pod/deployment context from Palette's control-plane components on a workload cluster, and bundles everything into a single tarball to attach to a Palette Support ticket.

Palette Support may ask you to run this when investigating a CPU spike, memory growth, or a slow reconciliation issue on a workload cluster's `palette-controller-manager` or `cluster-management-agent`.

---

## Which components does this cover?

**Only two components on a workload cluster expose Go pprof profiling behind the `PROFILING=enable` environment variable:**

| Namespace | Deployment | Containers profiled |
|---|---|---|
| `cluster-<uid>` | `palette-controller-manager` | `manager` (:8080), `atop-manager` (:8082) |
| `cluster-<uid>` | `cluster-management-agent` | `cluster-management-agent` (:8082, BasicAuth) |

**Everything else — Hubble services, CAPI infrastructure providers (CAPA/CAPZ/CAPV/…), cert-manager, and any user workload — is out of scope for this script.** They do not honour the `PROFILING` env var, and Palette Support will not ask you to profile them. If Support needs data from those, they will provide a different procedure.

---

## Prerequisites

The script is designed to run in a bare shell on any machine that can reach your Palette workload cluster's API server:

| | |
|---|---|
| **Access** | A `kubeconfig` for the workload cluster with permission to read pods, read the `palette-agent-debug-server-creds` Secret, set env on the two Deployments above, and `port-forward` into their pods. |
| **Tools** | `bash` 3.2+, `kubectl`, `curl`, `tar`, plus standard POSIX userland (`awk`, `sed`, `grep`, `tr`, `wc`, `find`, `mktemp`, `date`, `cp`, `id`, `uname`). All checked up front — the script fails in the first second if any are missing, rather than mid-collection. No `go` toolchain, no `python3`, no `base64`. Works on Linux and macOS. |
| **Network** | Whatever your `kubectl port-forward` normally traverses (usually 443 to the cluster's API server). |

---

## Quick start

Download the script from either URL, point `kubectl` at the affected workload cluster, and run it:

```bash
# Using GitHub URL
curl -sSLO https://raw.githubusercontent.com/spectrocloud/support-tools/main/pprof/collect-pprof.sh
chmod +x collect-pprof.sh

export KUBECONFIG=/path/to/workload-cluster.kubeconfig
./collect-pprof.sh
```

The script prints the tarball path when it finishes — attach that `pprof-cluster-<uid>-<timestamp>.tar.gz` to your Palette Support ticket.

Everything else goes to stderr, so `TARBALL=$(./collect-pprof.sh)` captures just the path.

---

## ⚠️ Impact — read this before you run it

`PROFILING=enable` is set on the Deployment's pod template, so **enabling it triggers a rolling restart** of the pod. On the workload cluster this means:

- Cluster reconciliation pauses briefly while the pod rolls (typically under a minute).
- Any profile captured immediately after the restart measures **startup**, not the steady state that's actually of interest.

For the second reason, the recommended workflow is two-phase — enable first, wait for the workload to re-establish, then collect:

```bash
# Phase 1: enable profiling. The pod restarts now. Exits immediately.
./collect-pprof.sh -P

# ... let the cluster run for at least a few minutes so whatever Support is
# investigating has time to re-appear on the newly started pod ...

# Phase 2: collect. No further restart.
./collect-pprof.sh
Comment on lines +69 to +70
```

If profiling is already enabled when the script starts, **nothing restarts** and one call is enough — the two-phase workflow is only needed when profiling has to be turned on first.

By default the script **restores whatever state it found**. If it enabled profiling, it disables it again on exit — including if you Ctrl-C or the collection fails partway. Nothing to clean up by hand.

| Situation | Flags | On exit | Restarts |
|---|---|---|---|
| Profiling was off before you ran it | *(default)* | disabled — undoes its own change | 2 |
| Profiling was already on | *(default)* | left on — the script didn't enable it, so it doesn't revoke it | 0 |
| Profiling was already on | `-D` | disabled unconditionally | 1 |
| Profiling was off | `-k` | left on | 1 |
| Profiling was off | `-P` | left on by design — you'll come back and collect | 1 |

---

## Options

```
-d LIST Deployments to profile (comma-separated). Default: BOTH
palette-controller-manager AND cluster-management-agent.
-d cluster-management-agent # cluster-management-agent only
-d palette-controller-manager # palette-controller-manager only
-n NAMESPACE Namespace. Default: auto-discovered.
-p POD Explicit pod name (single deployment only; pair with -d).
-s SECONDS CPU profile window in seconds. Default 30. The execution
trace length is set separately by the TRACE_SECONDS env
(default 5) and is not on a flag -- traces grow large fast.
-S LIST Explicit sample offsets in seconds, e.g. -S 0,600,1800.
"0" = single sample.
-c CURRENT STATE ONLY: one sample, no restart, no waiting.
Use this when profiling is already on and you want a
single as-found snapshot.
-P PREPARE ONLY: enable profiling and exit. Phase 1 above.
-w SECONDS Legacy settle delay before first sample. Default 0.
-o DIR Output directory for the tarball. Default: current directory.
-e MODE Enable profiling: auto (only if unset) | yes | no. Default auto.
-k Keep profiling enabled after collection.
-D Always disable profiling on exit, even if it was already on.
-h Full help.
```

Every flag also has an environment-variable twin (`DEPLOYS`, `NS`, `CPU_SECONDS`, `SAMPLE_SCHEDULE`, …). Run `./collect-pprof.sh -h` for the complete list.

### Sample schedule — chosen for you

If you pass neither `-S` nor `-c`, the schedule is picked automatically:

| When | Schedule | Why |
|---|---|---|
| The script had to enable profiling (`t=0` is a real cold start) | `0, 300, 900` seconds | Three points let Support see whether load **decays** (warm-up) or **stays flat** (real baseline). One sample cannot distinguish those. |
| Profiling was already on | Single sample | Offsets on an already-warm process would just be "time since the script started" — one honest sample is better. |

Every sample is stamped with the **measured container age** so Support can validate the data regardless of how the schedule was chosen.

---

## What's in the tarball

```
pprof-cluster-<uid>-<timestamp>/
README.txt summary + validity verdict
SAMPLES.txt per-sample offsets and measured container age
sample-<offset>/ one full profile set per sample point
manager-*.pb.gz heap, allocs, goroutine, threadcreate, block,
atop-manager-*.pb.gz mutex, cpu (30s), execution trace, /metrics
cma-*.pb.gz (twice, 60s apart)
Comment on lines +135 to +137
pod-top.txt 4 x `kubectl top pod` samples 15s apart
palette-controller-manager/ pod.yaml, deployment.yaml, pod-describe,
logs-manager.txt, logs-atop-manager.txt
cluster-management-agent/ pod.yaml, deployment.yaml, pod-describe,
logs-cluster-management-agent.txt
packs-t0.yaml / packs-t1.yaml namespace Pack CRs, 10s apart
FAILURES.txt present only if a fetch failed
```

**Please review the bundle before attaching it externally.** The pod and deployment manifests carry environment variables, image references, and secret names for your cluster. The logs contain whatever the controller logged during the collection window. If any of that is sensitive to your organization, redact it before sending — or share the bundle directly through the Support ticket, which is not public.

---

## Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| `ERROR: kubectl not found in PATH` (or similar) | A required tool is missing. | Install the missing tool. The script names it and exits before doing anything. |
| `ERROR: KUBECONFIG is not set or the cluster is unreachable` | `kubectl` cannot talk to the cluster. | `export KUBECONFIG=...` to a valid file for the workload cluster and confirm with `kubectl get nodes`. |
| `no palette-controller-manager in <ns>` | The kubeconfig points at a management cluster or the wrong workload cluster. | Confirm the target with `kubectl config current-context` and switch to the workload cluster you want to profile. |
| `-p/POD overrides a single pod, but 2 deployments are active` | You passed `-p` without narrowing to one deployment. | Pair `-p` with `-d palette-controller-manager` (or `-d cluster-management-agent`). |
| `[<dep>] timed out after 120s waiting for a Ready pod carrying PROFILING=enable` | The rollout is stuck (image pull, admission webhook, resource pressure). | `kubectl -n <ns> rollout status deployment/<dep>` and `kubectl -n <ns> describe pod ...` to see why. |
| `cma/debug/pprof/mutex http=404` in `FAILURES.txt` | Known limitation — `cluster-management-agent` doesn't currently expose the mutex profile. | Nothing you need to do; Support already knows. The rest of the bundle is unaffected. |
| Bundle name reports "COLD START" | The youngest container has been running less than 2 minutes. | If you used `-P` first and waited, this shouldn't happen — check that the workload had time to re-establish. Otherwise re-run after leaving the pod alone for a few minutes. |

---

## For Palette Support engineers — customer-ready message

> Paste verbatim into the ticket. Fill `<NAME>`. **Do not compress the two phases into one command** — that measures cold start instead of the real state.

```text
Hi <NAME>,

To investigate, please run this Go profiling capture on the affected workload
cluster and attach the resulting tarball to this ticket. It touches only
palette-controller-manager and cluster-management-agent; nothing else is
modified.

Enabling profiling triggers a rolling restart of that one Deployment
(typically under a minute). The script disables it again on exit, including
on Ctrl-C or failure.

Requirements: kubectl, curl, tar, bash 3.2+ (Linux or macOS).

# 1. Download
curl -sSLO https://raw.githubusercontent.com/spectrocloud/support-tools/main/pprof/collect-pprof.sh
chmod +x collect-pprof.sh

# 2. Point kubectl at the WORKLOAD cluster
export KUBECONFIG=/path/to/workload-cluster.kubeconfig
kubectl get nodes # sanity check

# 3. Phase 1 -- enable profiling. Pod restarts now. Exits immediately.
./collect-pprof.sh -P

# 4. Wait 10-15 minutes so the behaviour we're investigating re-appears.

# 5. Phase 2 -- collect. No further restart. ~5 minutes.
./collect-pprof.sh

Step 5 prints a path like: pprof-cluster-<uid>-<timestamp>.tar.gz
Please attach that single file to this ticket. The bundle contains
manifests and controller logs; the ticket is private, but feel free to
review and redact anything sensitive before uploading.

If anything errors, paste the full console output into the ticket.

Thanks,
<YOUR NAME>
Spectro Cloud Support
```

---

## Support

For issues, questions, or contributions:

- Open an issue in this repository
- Contact Spectro Cloud Support
Loading