diff --git a/README.md b/README.md index 5ca2633..9694c8c 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,88 @@ # support-tools -## support-bundle +Diagnostic scripts for Spectro Cloud Palette environments. Each tool is self-contained, runs from a bare shell, and produces artifacts you attach to a Palette Support ticket. + +## Table of contents + +- [Support bundles](#support-bundles) + - [Edge support bundle](#edge-support-bundle) + - [Infrastructure support bundle](#infrastructure-support-bundle) +- [Palette controller profiling (pprof)](#palette-controller-profiling-pprof) +- [Support](#support) + +--- + +## Support bundles + +Log and state collection from Palette clusters and edge hosts. Full documentation lives in [`support-bundle/`](support-bundle/) — see [`support-bundle/README.md`](support-bundle/README.md). + +### Edge support bundle + +Collects system journals, `systemd` service state, `kubectl` cluster info, and Kubernetes resources from an edge host and the cluster running on it. Use this on the **host**, as `sudo`. + +- **Script**: [`support-bundle/support-bundle-edge.sh`](support-bundle/support-bundle-edge.sh) +- **Documentation**: [`support-bundle/README-edge.md`](support-bundle/README-edge.md) +- **Prerequisites**: `sudo`, `journalctl`, `systemctl`, `kubectl` +- **Runs on**: Edge host (Linux) +- **Quick Start**: + ```bash + # Official SpectroCloud URL + curl -sSL https://software.spectrocloud.com/scripts/support-bundle-edge.sh -o support-bundle-edge.sh + sudo bash support-bundle-edge.sh + + # GitHub URL + curl -sSL https://raw.githubusercontent.com/spectrocloud/support-tools/main/support-bundle/support-bundle-edge.sh -o support-bundle-edge.sh + sudo bash support-bundle-edge.sh + ``` +- **Advanced**: extra namespaces, resources, or `journalctl` units via `-n`, `-r`, `-R`, `-j` flags. See the edge README for the full list. + +### Infrastructure support bundle + +Collects cluster state, Cluster API (CAPI) objects, and Palette resources from a Kubernetes infrastructure cluster. Use this against a Palette **management** or **workload** cluster's kubeconfig. + +- **Script**: [`support-bundle/support-bundle-infra.sh`](support-bundle/support-bundle-infra.sh) +- **Documentation**: [`support-bundle/README-infra.md`](support-bundle/README-infra.md) +- **Prerequisites**: `kubectl` with cluster access +- **Runs on**: Any workstation that can reach the cluster's API server +- **Quick Start**: + ```bash + # Official SpectroCloud URL + curl -sSL https://software.spectrocloud.com/scripts/support-bundle-infra.sh -o support-bundle-infra.sh + bash support-bundle-infra.sh + + # GitHub URL + curl -sSL https://raw.githubusercontent.com/spectrocloud/support-tools/main/support-bundle/support-bundle-infra.sh -o support-bundle-infra.sh + bash support-bundle-infra.sh + ``` +- **Advanced**: extra namespaces and resources via `-n`, `-r`, `-R` flags. See the infra README for the full list. + +--- + +## Palette controller profiling (pprof) + +Go pprof profiles, execution traces, `/metrics`, and pod/deployment context from Palette's control-plane components on a workload cluster. Produces a single tarball to attach to a support ticket. + +**Scope — read this before running.** Only two Deployments on a workload cluster expose Go pprof profiling behind the `PROFILING=enable` environment variable: `palette-controller-manager` and `cluster-management-agent`. Hubble services, CAPI infrastructure providers (CAPA/CAPZ/CAPV/…), cert-manager, and user workloads are **out of scope** — they do not honour this environment variable, and this script does not attempt to profile them. + +- **Script**: [`pprof/collect-pprof.sh`](pprof/collect-pprof.sh) +- **Documentation**: [`pprof/README.md`](pprof/README.md) +- **Prerequisites**: `bash` 3.2+, `kubectl`, `curl`, `tar`, standard POSIX userland. No `python3`, no `base64`. Runs on Linux and macOS. +- **Runs on**: Any workstation that can reach the workload cluster's API server +- **Quick Start**: + ```bash + # GitHub URL + curl -sSLO https://raw.githubusercontent.com/spectrocloud/support-tools/main/pprof/collect-pprof.sh + chmod +x collect-pprof.sh + + export KUBECONFIG=/path/to/workload-cluster.kubeconfig + ./collect-pprof.sh + ``` +- **Impact**: enabling profiling triggers a rolling restart of the affected pod. See the pprof README for the recommended two-phase workflow (`-P` to enable, wait, then collect) that avoids measuring cold-start behaviour instead of steady state. + +--- + +## Support + +- Open an issue in this repository for bugs, questions, or contributions. +- Contact Spectro Cloud Support for help using these scripts on a specific cluster. +- Each script has its own detailed README linked above — flags, output layout, and troubleshooting live there. diff --git a/pprof/README.md b/pprof/README.md new file mode 100644 index 0000000..3d2b8e3 --- /dev/null +++ b/pprof/README.md @@ -0,0 +1,218 @@ +# Palette Controller Profiling Collection + +`collect-pprof.sh` gathers Go pprof profiles, execution traces, controller metrics, and pod/deployment context from Palette's control-plane components on a workload cluster, and bundles everything into a single tarball to attach to a Palette Support ticket. + +Palette Support may ask you to run this when investigating a CPU spike, memory growth, or a slow reconciliation issue on a workload cluster's `palette-controller-manager` or `cluster-management-agent`. + +--- + +## Which components does this cover? + +**Only two components on a workload cluster expose Go pprof profiling behind the `PROFILING=enable` environment variable:** + +| Namespace | Deployment | Containers profiled | +|---|---|---| +| `cluster-` | `palette-controller-manager` | `manager` (:8080), `atop-manager` (:8082) | +| `cluster-` | `cluster-management-agent` | `cluster-management-agent` (:8082, BasicAuth) | + +**Everything else — Hubble services, CAPI infrastructure providers (CAPA/CAPZ/CAPV/…), cert-manager, and any user workload — is out of scope for this script.** They do not honour the `PROFILING` env var, and Palette Support will not ask you to profile them. If Support needs data from those, they will provide a different procedure. + +--- + +## Prerequisites + +The script is designed to run in a bare shell on any machine that can reach your Palette workload cluster's API server: + +| | | +|---|---| +| **Access** | A `kubeconfig` for the workload cluster with permission to read pods, read the `palette-agent-debug-server-creds` Secret, set env on the two Deployments above, and `port-forward` into their pods. | +| **Tools** | `bash` 3.2+, `kubectl`, `curl`, `tar`, plus standard POSIX userland (`awk`, `sed`, `grep`, `tr`, `wc`, `find`, `mktemp`, `date`, `cp`, `id`, `uname`). All checked up front — the script fails in the first second if any are missing, rather than mid-collection. No `go` toolchain, no `python3`, no `base64`. Works on Linux and macOS. | +| **Network** | Whatever your `kubectl port-forward` normally traverses (usually 443 to the cluster's API server). | + +--- + +## Quick start + +Download the script from either URL, point `kubectl` at the affected workload cluster, and run it: + +```bash +# Using GitHub URL +curl -sSLO https://raw.githubusercontent.com/spectrocloud/support-tools/main/pprof/collect-pprof.sh +chmod +x collect-pprof.sh + +export KUBECONFIG=/path/to/workload-cluster.kubeconfig +./collect-pprof.sh +``` + +The script prints the tarball path when it finishes — attach that `pprof-cluster--.tar.gz` to your Palette Support ticket. + +Everything else goes to stderr, so `TARBALL=$(./collect-pprof.sh)` captures just the path. + +--- + +## ⚠️ Impact — read this before you run it + +`PROFILING=enable` is set on the Deployment's pod template, so **enabling it triggers a rolling restart** of the pod. On the workload cluster this means: + +- Cluster reconciliation pauses briefly while the pod rolls (typically under a minute). +- Any profile captured immediately after the restart measures **startup**, not the steady state that's actually of interest. + +For the second reason, the recommended workflow is two-phase — enable first, wait for the workload to re-establish, then collect: + +```bash +# Phase 1: enable profiling. The pod restarts now. Exits immediately. +./collect-pprof.sh -P + +# ... let the cluster run for at least a few minutes so whatever Support is +# investigating has time to re-appear on the newly started pod ... + +# Phase 2: collect. No further restart. +./collect-pprof.sh +``` + +If profiling is already enabled when the script starts, **nothing restarts** and one call is enough — the two-phase workflow is only needed when profiling has to be turned on first. + +By default the script **restores whatever state it found**. If it enabled profiling, it disables it again on exit — including if you Ctrl-C or the collection fails partway. Nothing to clean up by hand. + +| Situation | Flags | On exit | Restarts | +|---|---|---|---| +| Profiling was off before you ran it | *(default)* | disabled — undoes its own change | 2 | +| Profiling was already on | *(default)* | left on — the script didn't enable it, so it doesn't revoke it | 0 | +| Profiling was already on | `-D` | disabled unconditionally | 1 | +| Profiling was off | `-k` | left on | 1 | +| Profiling was off | `-P` | left on by design — you'll come back and collect | 1 | + +--- + +## Options + +``` +-d LIST Deployments to profile (comma-separated). Default: BOTH + palette-controller-manager AND cluster-management-agent. + -d cluster-management-agent # cluster-management-agent only + -d palette-controller-manager # palette-controller-manager only +-n NAMESPACE Namespace. Default: auto-discovered. +-p POD Explicit pod name (single deployment only; pair with -d). +-s SECONDS CPU profile window in seconds. Default 30. The execution + trace length is set separately by the TRACE_SECONDS env + (default 5) and is not on a flag -- traces grow large fast. +-S LIST Explicit sample offsets in seconds, e.g. -S 0,600,1800. + "0" = single sample. +-c CURRENT STATE ONLY: one sample, no restart, no waiting. + Use this when profiling is already on and you want a + single as-found snapshot. +-P PREPARE ONLY: enable profiling and exit. Phase 1 above. +-w SECONDS Legacy settle delay before first sample. Default 0. +-o DIR Output directory for the tarball. Default: current directory. +-e MODE Enable profiling: auto (only if unset) | yes | no. Default auto. +-k Keep profiling enabled after collection. +-D Always disable profiling on exit, even if it was already on. +-h Full help. +``` + +Every flag also has an environment-variable twin (`DEPLOYS`, `NS`, `CPU_SECONDS`, `SAMPLE_SCHEDULE`, …). Run `./collect-pprof.sh -h` for the complete list. + +### Sample schedule — chosen for you + +If you pass neither `-S` nor `-c`, the schedule is picked automatically: + +| When | Schedule | Why | +|---|---|---| +| The script had to enable profiling (`t=0` is a real cold start) | `0, 300, 900` seconds | Three points let Support see whether load **decays** (warm-up) or **stays flat** (real baseline). One sample cannot distinguish those. | +| Profiling was already on | Single sample | Offsets on an already-warm process would just be "time since the script started" — one honest sample is better. | + +Every sample is stamped with the **measured container age** so Support can validate the data regardless of how the schedule was chosen. + +--- + +## What's in the tarball + +``` +pprof-cluster--/ + README.txt summary + validity verdict + SAMPLES.txt per-sample offsets and measured container age + sample-/ one full profile set per sample point + manager-*.pb.gz heap, allocs, goroutine, threadcreate, block, + atop-manager-*.pb.gz mutex, cpu (30s), execution trace, /metrics + cma-*.pb.gz (twice, 60s apart) + pod-top.txt 4 x `kubectl top pod` samples 15s apart + palette-controller-manager/ pod.yaml, deployment.yaml, pod-describe, + logs-manager.txt, logs-atop-manager.txt + cluster-management-agent/ pod.yaml, deployment.yaml, pod-describe, + logs-cluster-management-agent.txt + packs-t0.yaml / packs-t1.yaml namespace Pack CRs, 10s apart + FAILURES.txt present only if a fetch failed +``` + +**Please review the bundle before attaching it externally.** The pod and deployment manifests carry environment variables, image references, and secret names for your cluster. The logs contain whatever the controller logged during the collection window. If any of that is sensitive to your organization, redact it before sending — or share the bundle directly through the Support ticket, which is not public. + +--- + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `ERROR: kubectl not found in PATH` (or similar) | A required tool is missing. | Install the missing tool. The script names it and exits before doing anything. | +| `ERROR: KUBECONFIG is not set or the cluster is unreachable` | `kubectl` cannot talk to the cluster. | `export KUBECONFIG=...` to a valid file for the workload cluster and confirm with `kubectl get nodes`. | +| `no palette-controller-manager in ` | The kubeconfig points at a management cluster or the wrong workload cluster. | Confirm the target with `kubectl config current-context` and switch to the workload cluster you want to profile. | +| `-p/POD overrides a single pod, but 2 deployments are active` | You passed `-p` without narrowing to one deployment. | Pair `-p` with `-d palette-controller-manager` (or `-d cluster-management-agent`). | +| `[] timed out after 120s waiting for a Ready pod carrying PROFILING=enable` | The rollout is stuck (image pull, admission webhook, resource pressure). | `kubectl -n rollout status deployment/` and `kubectl -n describe pod ...` to see why. | +| `cma/debug/pprof/mutex http=404` in `FAILURES.txt` | Known limitation — `cluster-management-agent` doesn't currently expose the mutex profile. | Nothing you need to do; Support already knows. The rest of the bundle is unaffected. | +| Bundle name reports "COLD START" | The youngest container has been running less than 2 minutes. | If you used `-P` first and waited, this shouldn't happen — check that the workload had time to re-establish. Otherwise re-run after leaving the pod alone for a few minutes. | + +--- + +## For Palette Support engineers — customer-ready message + +> Paste verbatim into the ticket. Fill ``. **Do not compress the two phases into one command** — that measures cold start instead of the real state. + +```text +Hi , + +To investigate, please run this Go profiling capture on the affected workload +cluster and attach the resulting tarball to this ticket. It touches only +palette-controller-manager and cluster-management-agent; nothing else is +modified. + +Enabling profiling triggers a rolling restart of that one Deployment +(typically under a minute). The script disables it again on exit, including +on Ctrl-C or failure. + +Requirements: kubectl, curl, tar, bash 3.2+ (Linux or macOS). + + # 1. Download + curl -sSLO https://raw.githubusercontent.com/spectrocloud/support-tools/main/pprof/collect-pprof.sh + chmod +x collect-pprof.sh + + # 2. Point kubectl at the WORKLOAD cluster + export KUBECONFIG=/path/to/workload-cluster.kubeconfig + kubectl get nodes # sanity check + + # 3. Phase 1 -- enable profiling. Pod restarts now. Exits immediately. + ./collect-pprof.sh -P + + # 4. Wait 10-15 minutes so the behaviour we're investigating re-appears. + + # 5. Phase 2 -- collect. No further restart. ~5 minutes. + ./collect-pprof.sh + +Step 5 prints a path like: pprof-cluster--.tar.gz +Please attach that single file to this ticket. The bundle contains +manifests and controller logs; the ticket is private, but feel free to +review and redact anything sensitive before uploading. + +If anything errors, paste the full console output into the ticket. + +Thanks, + +Spectro Cloud Support +``` + +--- + +## Support + +For issues, questions, or contributions: + +- Open an issue in this repository +- Contact Spectro Cloud Support diff --git a/pprof/collect-pprof.sh b/pprof/collect-pprof.sh new file mode 100755 index 0000000..58ea87a --- /dev/null +++ b/pprof/collect-pprof.sh @@ -0,0 +1,1029 @@ +#!/bin/bash +# Copyright 2026 Spectro Cloud +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# collect-pprof.sh -- collect Go pprof profiles + controller metrics from a +# Spectro Cloud controller and bundle them into a single tarball. +# +# --------------------------------------------------------------------------- +# READ THIS FIRST: enabling profiling RESTARTS the pod. +# +# The PROFILING=enable env var lives on the Deployment's pod template, so setting +# it rolls the pod. That destroys the exact state you usually want to profile -- +# a long-running process with a slow leak or a hot reconcile loop -- and leaves +# you measuring startup instead. +# +# This has bitten a real investigation: all four collected bundles were taken 5-6s +# after container start. The 30s CPU window measured cold-start discovery, and +# labeled Prometheus series (workqueue_adds_total, controller_runtime_reconcile_*) +# had not been incremented yet so they were ABSENT from /metrics entirely -- the +# single most diagnostic signal for a hot-reconcile loop, silently missing. +# +# Hence two phases. Prefer them over a one-shot run: +# +# Phase 1, once: collect-pprof.sh -P # enable + exit; pod restarts NOW +# ... let the workload run and the symptom re-establish (hours is fine) ... +# Phase 2, later: collect-pprof.sh # no restart, real baseline +# +# A one-shot run still works and will enable, wait SETTLE_SECONDS, then collect -- +# but a settle is a mitigation, not a fix. If PROFILING is already on, nothing +# restarts and you get a clean read. +# --------------------------------------------------------------------------- + +PPROF_VERSION=20260901 + +# ==== Targets ==== +# PROFILING=enable is a shared Spectro convention, not palette-specific -- the +# same mechanism exists in palette (pkg/utils/https_pprof.go), ally +# (services/cluster-management-agent/service/pprof/https_pprof.go) and stylus +# (pkg/pprof/profiling_server.go). Only the container/port/auth differ, so the +# target is data, not hardcoded control flow. +# +# Format: