Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ resolution. See [STATUS.md](STATUS.md) for what is open.
`### PERF_STAT ###` section.
- Agent compresses with in-process zstd (vendored). Server decompresses
in-process via the `zstandard` package (external `zstd` binary as
fallback). Typical ratio 20–40×.
fallback). About 20× on device captures.

### Key design decisions
- The agent is a zero-dependency static C binary. The server is a normal
Expand Down Expand Up @@ -123,7 +123,7 @@ resolution. See [STATUS.md](STATUS.md) for what is open.
runs at `nice 5` so the profiler yields to the workload, every perf child
runs with `LC_ALL=C` in its own process group, and `perf stat` rounds run
back to back so every interval is counted.
- Single agent implementation: a static musl C binary (~0.7 MB, vendored
- Single agent implementation: a static musl C binary (~0.6 MB, vendored
zstd, zero deps) that cross-compiles for five architectures from the
toolchains on the repo's `toolchains` GitHub release, installs with one
curl command (install-agent.sh), and self-updates with --update after
Expand Down
21 changes: 11 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
<a href="#quick-start"><img alt="quick start" src="https://img.shields.io/badge/quick_start-60s-3fb950?style=flat-square"/></a>
<img alt="license" src="https://img.shields.io/badge/license-MIT-blue?style=flat-square"/>
<img alt="agent" src="https://img.shields.io/badge/agent-static_C_binary-58a6ff?style=flat-square"/>
<img alt="arch" src="https://img.shields.io/badge/arch-x86__64_%7C_aarch64-c084fc?style=flat-square"/>
<img alt="arch" src="https://img.shields.io/badge/arch-x86__64_%7C_aarch64_%7C_armv7_%7C_aarch64__be_%7C_armeb-c084fc?style=flat-square"/>
<img alt="wire" src="https://img.shields.io/badge/wire-zstd_%7C_5--byte_header-d29922?style=flat-square"/>
<img alt="install" src="https://img.shields.io/badge/install-uvx_perflens-f85149?style=flat-square"/>
</p>
Expand All @@ -23,20 +23,20 @@

<p align="center">
<img src="docs/demo.gif" alt="Live demo: function table updating in real time as perf samples stream in, then flame graph, then source view" width="100%"/>
<br><sub><em>Sample counts climb live as <code>perf record</code> rounds stream in. Flip to flame graph, click a function, drop into source with line-level heat. Zero polling — Server-Sent Events.</em></sub>
<br><sub><em>Sample counts climb live as chunks stream in from a continuous <code>perf record</code> pipeline. Flip to flame graph, click a function, drop into source with line-level heat. Zero polling — Server-Sent Events.</em></sub>
</p>

# PerfLens

**PerfLens** is a remote Linux performance profiler with a real-time web UI. Drop the agent on any Linux device (ARM or x86), point it at a PID, and watch flame graphs, function tables, `perf stat` metrics, and line-level annotated source update live in your browser.

No Docker, no sudo. A modern React + TypeScript UI shipped **prebuilt** inside the Python wheel (end users never need Node), and a single static C agent binary (~2 MB) with zero runtime dependencies — it runs on anything from bare-metal embedded boards to servers, installs with one curl command, and updates itself with `--update`.
No Docker, no sudo. A modern React + TypeScript UI shipped **prebuilt** inside the Python wheel (end users never need Node), and a single static C agent binary (about 0.6 MB) with zero runtime dependencies — it runs on anything from bare-metal embedded boards to servers, installs with one curl command, and updates itself with `--update`.

---

## Highlights

- **Real-time streaming** — `perf record` runs in ~8s rounds; each round is compressed with zstd and streamed over a 5-byte framed TCP protocol
- **Real-time streaming** — one continuous `perf record | perf script` pipeline on the target, cut into chunks every interval (8 s by default) or every 16 MB of text, zstd-compressed and streamed over a 5-byte framed TCP protocol; discrete rounds are the fallback where a target's perf cannot keep call chains through a pipe
- **Live web UI** — Server-Sent Events push parsed function tables, flame graphs, and `perf stat` panels to the browser as new data arrives
- **Source-level annotation** — `addr2line` maps samples back to source lines; the UI heat-colors hot lines red/amber/green
- **Differential profiling** — snapshot a baseline (or pick a saved session) and the flame graph recolors by change (red grew, blue shrank) while the function table shows per-function Δ; did-my-fix-help in one glance
Expand All @@ -48,10 +48,11 @@ No Docker, no sudo. A modern React + TypeScript UI shipped **prebuilt** inside t
- **Cross-compilation toolchain support** — `--toolchain-prefix` derives addr2line and readelf from a single prefix; `--sysroot` resolves shared libraries and source files under a sysroot tree
- **ARM + x86** — same agent code runs on aarch64, aarch64_be, armv7, armeb, x86_64
- **Session save / replay** — raw chunks saved to disk, replayed lazily on demand via the UI's session list
- **MCP server for AI agents** — `perflens mcp` (the optional `[mcp]` extra) lets Claude Code or any MCP client ask a running server for hot functions, hot stacks, source hot lines, IPC and miss rates, per-thread breakdowns and session comparisons
- **Static C agent** — single binary with vendored zstd, no runtime dependencies; cross-compiles to aarch64, aarch64_be, armv7, armeb, x86_64; one-line curl install and built-in self-update
- **Zero-friction server install** — `uvx perflens` (or `pipx` / `pip install --user`); everything resolves user-space, no sudo, corporate-machine friendly. Missing binutils? `perflens provision` downloads static addr2line/readelf into `~/.perflens/bin`
- **Capability probing** — the agent discovers which perf events and call-graph modes (`fp` / `dwarf` / `lbr`) actually work on the target before collecting
- **Zstd compression** — typical perf script payloads compress 20–40× before hitting the wire
- **Zstd compression** — `perf script` payloads compress about 20× before hitting the wire (measured on x86_64 and ARM64 device captures)

---

Expand All @@ -65,11 +66,11 @@ The pipeline in one sentence: **`perf record` → agent → TCP+zstd → server

### Target device

- The agent probes the kernel's `perf_event_paranoid`, enumerates candidate events (`cycles`, `instructions`, `cache-*`, `branch-*`, `page-faults`, `context-switches`, `cpu-migrations`), tries call-graph modes in order (`fp`, `dwarf`, `lbr`), and picks the first that produces non-empty stacks
- Each collection round runs `perf record` and `perf stat` in parallel for N seconds, then `perf script` to flatten the output
- The agent probes the kernel's `perf_event_paranoid`, enumerates candidate events (`cycles`, `instructions`, `cache-*`, `branch-*`, the `cpu-clock` and `task-clock` software events that PMU-less targets sample on, and the stat-only `page-faults`, `context-switches`, `cpu-migrations`), tries call-graph modes in order (`fp`, `dwarf`, `lbr`), and picks the first that produces call chains
- Collection is one continuous `perf record -o - | perf script -i -` pipeline, cut into chunks at sample boundaries every interval or every 16 MB of text, with `perf stat` rounds running back to back so every interval is counted. Where a target's perf drops call chains through a pipe (measured on perf 4.4), the agent falls back to discrete rounds, starting the next round's `perf record` while the previous round's `perf script` runs
- The combined text is compressed with in-process zstd (level 1) and framed with a 5-byte header
- Reconnects with exponential backoff if the server drops
- Single static binary — **no Python, no libc, no zstd needed on the target**. Suitable for old or minimal ARM/x86 Linux devices.
- Single static musl binary of about 0.6 MB — **no Python, no libc, no zstd needed on the target**. Suitable for old or minimal ARM/x86 Linux devices, big-endian ARM included.

### Local machine

Expand Down Expand Up @@ -110,7 +111,7 @@ The protocol is bidirectional — data and health metrics flow agent → server,
| `3` | agent → server | Command response / `hello` handshake (JSON) |
| `4` | agent → server | Device health metrics (JSON, every 2s: CPU, memory, temperature, network, per-process stats; opt-in disk I/O and per-thread CPU via `configure_metrics`) |

The server reads the 5 header bytes first, then exactly `LEN` more. Compression is in-process zstd on both ends (vendored in the agent, the `zstandard` package on the server, external `zstd` binary as a fallback). Typical ratio on real `perf script` output is **20–40×**.
The server reads the 5 header bytes first, then exactly `LEN` more. Compression is in-process zstd on both ends (vendored in the agent, the `zstandard` package on the server, external `zstd` binary as a fallback). Measured on real device captures, `perf script` output compresses **about 20×**.

### Handshake

Expand Down Expand Up @@ -292,7 +293,7 @@ Options:
| `--pid PID` | — | PID of process to profile (required for `--output`; set via UI wizard in daemon modes) |
| `--port PORT` | `9999` | TCP port (listen or connect) |
| `--frequency HZ` | `99` | `perf record -F` sampling frequency |
| `--duration SECS` | `8` | Length of each collection round |
| `--duration SECS` | `8` | Chunk interval in continuous mode; the round length in the fallback round mode and `--output` |
| `--rounds N` | `1` | Number of collection rounds (`--output` mode only) |
| `--bind ADDR` | `0.0.0.0` | Address to listen on in `--listen` mode |
| `--token SECRET` | — | Pairing code the server must present (or `PERFLENS_TOKEN`). In `--listen` mode one is generated and logged if you don't supply it. **Never sent over the wire.** |
Expand Down
38 changes: 27 additions & 11 deletions docs/architecture.html
Original file line number Diff line number Diff line change
Expand Up @@ -73,23 +73,39 @@ <h3 id="probing">Capability probing</h3>
</p>
<ul>
<li>Reads <code>/proc/sys/kernel/perf_event_paranoid</code> and warns if &gt; 1.</li>
<li>Enumerates candidate events (<code>cycles</code>, <code>instructions</code>, <code>cache-misses</code>, <code>cache-references</code>, <code>branch-misses</code>, <code>branch-instructions</code>, <code>page-faults</code>, <code>context-switches</code>, <code>cpu-migrations</code>) and keeps only the ones <code>perf record</code>/<code>perf stat</code> actually accepts.</li>
<li>Enumerates candidate events (<code>cycles</code>, <code>instructions</code>, <code>cache-misses</code>, <code>cache-references</code>, <code>branch-misses</code>, <code>branch-instructions</code>, the <code>cpu-clock</code> and <code>task-clock</code> software events that targets without a hardware PMU sample on, <code>page-faults</code>, <code>context-switches</code>, <code>cpu-migrations</code>) and keeps only the ones <code>perf record</code>/<code>perf stat</code> actually accepts.</li>
<li>Tries call-graph modes in order: <code>fp</code> &rarr; <code>dwarf</code> &rarr; <code>lbr</code>, picks the first that produces non-empty stacks.</li>
<li>Probes whether <code>perf script -F</code> is supported (perf &ge; ~3.12) and falls back to the default output format on older kernels.</li>
</ul>
<p>
This costs roughly <strong>6&ndash;12 seconds on first connection</strong>
and is a one-time hit.
The probe is batched &mdash; one <code>perf stat</code> over every candidate, one
<code>perf record</code> over the survivors &mdash; and took
<strong>10.8&nbsp;s on a hybrid x86_64 container and 12.7&nbsp;s on an
8-core ARM64 board</strong> when measured for 0.12.0. It runs once per
connection; switching to another process only re-checks that the new
pid can be recorded.
</p>

<h3 id="rounds">Collection rounds</h3>
<h3 id="collection">Continuous collection</h3>
<p>
Each round runs <code>perf record</code> and <code>perf stat</code> in parallel for
<em>N</em> seconds (default 8), then <code>perf script</code> flattens the trace.
The result &mdash; perf script text optionally followed by a
<code>### PERF_STAT ###</code> section &mdash; is compressed with
in-process zstd (level 1, vendored in the agent) and pushed over TCP
with a 5-byte header. Typical compression: <strong>20&ndash;40&times;</strong>.
Collection is one long-lived pipeline, <code>perf record -o - | perf script -i -</code>,
so there is no sampling dead time and symbol tables are parsed once. The agent
cuts its output into chunks at sample boundaries every <em>N</em> seconds
(default 8) or every 16&nbsp;MB of text, whichever comes first, and runs
<code>perf stat</code> rounds back to back so every interval is counted. Each
chunk &mdash; perf script text optionally followed by a
<code>### PERF_STAT ###</code> section &mdash; is compressed with in-process
zstd (level 1, vendored in the agent) and pushed over TCP with a 5-byte
header. Measured compression on device captures: <strong>about 20&times;</strong>.
</p>
<p>
Some old <code>perf</code> builds produce samples through a pipe but drop
their call chains (measured on perf 4.4). The capability probe detects
that, and the agent falls back to discrete rounds: <code>perf record</code>
to a file for <em>N</em> seconds, then <code>perf script</code> on it, with the
next round's <code>perf record</code> started while the previous round's
<code>perf script</code> runs. <code>perf script</code> runs at
<code>nice 5</code> in both modes, so the profiler yields to the workload.
</p>

<h3 id="health">Health metrics</h3>
Expand Down Expand Up @@ -255,7 +271,7 @@ <h2 id="limits">Known limits</h2>
<ul>
<li><a href="#target">On the target</a></li>
<li><a href="#probing">Capability probing</a></li>
<li><a href="#rounds">Collection rounds</a></li>
<li><a href="#collection">Continuous collection</a></li>
<li><a href="#health">Health metrics</a></li>
<li><a href="#local">On the local machine</a></li>
<li><a href="#parser">Parser</a></li>
Expand Down
10 changes: 5 additions & 5 deletions docs/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ <h1>Real-time Linux profiling,<br><span class="grad">straight in your browser.</
<div class="hero-badges">
<img alt="license" src="https://img.shields.io/badge/license-MIT-blue?style=flat-square">
<img alt="agent" src="https://img.shields.io/badge/agent-static_C_binary-58a6ff?style=flat-square">
<img alt="arch" src="https://img.shields.io/badge/arch-x86__64_%7C_aarch64_%7C_armv7-c084fc?style=flat-square">
<img alt="arch" src="https://img.shields.io/badge/arch-x86__64_%7C_aarch64_%7C_armv7_%7C_aarch64__be_%7C_armeb-c084fc?style=flat-square">
<img alt="wire" src="https://img.shields.io/badge/wire-zstd_%7C_5--byte_header-d29922?style=flat-square">
<img alt="install" src="https://img.shields.io/badge/install-uvx_perflens-f85149?style=flat-square">
</div>
Expand All @@ -83,7 +83,7 @@ <h1>Real-time Linux profiling,<br><span class="grad">straight in your browser.</
<img src="demo.gif" alt="Live demo: function table updating in real time, flame graph, then source view" loading="lazy" style="display: block; width: 100%;">
</div>
<p style="color: var(--text-tertiary); font-size: 13.5px; margin-top: 14px;">
Counts update as each <code>perf record</code> round streams in. Flip to flame graph, click a function, land in source with line-level heat. Zero polling &mdash; Server-Sent Events.
Counts update as each chunk streams in from a continuous <code>perf record</code> pipeline. Flip to flame graph, click a function, land in source with line-level heat. Zero polling &mdash; Server-Sent Events.
</p>
</div>
</section>
Expand All @@ -100,7 +100,7 @@ <h2>One tool, the full <code>perf</code> pipeline.</h2>
<div class="feature">
<div class="feature-icon">&#9889;</div>
<h3>Real-time streaming</h3>
<p>The agent runs <code>perf record</code> in 8-second rounds. Each round is zstd-compressed and pushed over TCP. Browser sees flame graphs update as new data arrives.</p>
<p>The agent runs one continuous <code>perf record</code> pipeline and cuts it into chunks every interval or every 16&nbsp;MB. Each chunk is zstd-compressed and pushed over TCP, and the browser sees flame graphs update as new data arrives.</p>
</div>
<div class="feature">
<div class="feature-icon">&#128293;</div>
Expand Down Expand Up @@ -130,7 +130,7 @@ <h3>Per-thread analysis</h3>
<div class="feature">
<div class="feature-icon">&#128737;</div>
<h3>Static C agent</h3>
<p>Single static C agent binary (~2 MB, vendored zstd, zero runtime deps) &mdash; runs on anything from bare-metal embedded boards to servers. One-line curl install, built-in self-update.</p>
<p>Single static C agent binary (about 0.6&nbsp;MB, vendored zstd, zero runtime deps) &mdash; runs on anything from bare-metal embedded boards to servers. One-line curl install, built-in self-update.</p>
</div>
<div class="feature">
<div class="feature-icon">&#129518;</div>
Expand Down Expand Up @@ -404,7 +404,7 @@ <h2>How a sample travels.</h2>
to every connected browser via Server-Sent Events.
</p>
<p style="color: var(--text-secondary); margin-bottom: 22px;">
Typical zstd ratio on real perf script output: <strong>20&ndash;40&times;</strong>.
Measured zstd ratio on real perf script output from devices: <strong>about 20&times;</strong>.
</p>
<a class="btn btn-ghost" href="architecture.html">Read the architecture &rarr;</a>
</div>
Expand Down
Loading
Loading