steel-dev · nibzard · Jul 30, 2026 · Jul 30, 2026
diff --git a/.claude/skills/seo/SKILL.md b/.claude/skills/seo/SKILL.md
@@ -10,7 +10,7 @@ A repeatable **monthly** SEO check for Steel's two web properties — **docs.ste
 ## What it produces
 
 Each run writes, under `scripts/seo/pulse/`:
-- **Snapshots** (`snapshots/volumes-YYYY-MM.json`, `ranks-YYYY-MM.json`, `tech-YYYY-MM.json`) — the raw state for this month. Next month's run diffs against these.
+- **Snapshots** (`snapshots/volumes-YYYY-MM.json`, `ranks-YYYY-MM.json`, `tech-YYYY-MM.json`, `perf-YYYY-MM.json`) — the raw state for this month. Next month's run diffs against these.
 - **`reports/seo-YYYY-MM.md`** — the human-readable delta report (the deliverable).
 
 Snapshots are gitignored (regenerable state); reports are kept for history.
@@ -31,9 +31,20 @@ The keyword data comes from DataForSEO's `keywords_data/google_ads/search_volume
 
 So: weight **volume × CPC × momentum** together, and treat the numbers as directional, not exact.
 
+### Impression sinks: always report an ex-sinks view
+
+A couple of pages pull huge impression volumes at near-zero CTR, so any site-level clicks/impressions/CTR figure that includes them describes those pages instead of the docs. They live in `config.json` → `impression_sinks` (with the reason each one is listed). **Whenever you report a site-level click, impression, or CTR number, report the ex-sinks figure next to it** and say which pages were excluded. Never delete the as-exported number: keep both, so a shift in the sinks themselves stays visible.
+
 ## The monthly workflow
 
-Run the three deterministic scripts in order (they print summaries and write snapshots), then add judgment for drift/discovery, then synthesize the report. The split is deliberate — **scripts handle the mechanical, repeatable data; you handle synthesis and judgment.** (A prior attempt to do everything in one big agent fan-out got stuck in loops; scripts are reliable and cheap to re-run.)
+Run the deterministic scripts in order (they print summaries and write snapshots), then add judgment for drift/discovery, then synthesize the report. The split is deliberate — **scripts handle the mechanical, repeatable data; you handle synthesis and judgment.** (A prior attempt to do everything in one big agent fan-out got stuck in loops; scripts are reliable and cheap to re-run.)
+
+### Phase 0: Search performance (needs a Search Console export)
+
+```bash
+node .claude/skills/seo/scripts/gsc-perf.mjs <path-to-gsc-pages-export.csv>
+```
+Ask the user for a Search Console **Pages** export (docs.steel.dev, last 3 months, CSV). The script prints site clicks/impressions/CTR **as exported and ex-sinks**, breaks out each `impression_sinks` page with its share of site impressions, writes `perf-YYYY-MM.json`, and prints last month's ex-sinks totals for comparison. Skip the phase if no export is available and say so in the report: don't quote site CTR from memory.
 
 ### Phase 1 — Refresh demand
 
@@ -76,6 +87,14 @@ Read the three script summaries + your Phase 4 notes and write `scripts/seo/puls
 ## TL;DR
 2–4 sentences: did the P0 title fixes ship / move rankings? Any regressions? Biggest new opportunity?
 
+## Search performance
+| View | Clicks | Impressions | CTR |
+|---|---|---|---|
+| Site (as exported) | … | … | … |
+| Site (ex-sinks) | … | … | … |
+
+Ex-sinks excludes `config.json` → `impression_sinks`; list the excluded pages with their impressions and CTR. Compare month over month on the **ex-sinks** row.
+
 ## Demand movers
 - 📈 Rising: <term> (prev → now, +X% trend) — why it matters / suggested action
 - 📉 Falling: <term> …
@@ -113,11 +132,13 @@ All tracked keywords, key pages, and competitors live in `.claude/skills/seo/con
 - Add newly-discovered ownable terms to `tracked_keywords` (and `rank_keywords` if worth tracking position).
 - Add a `key_pages` entry (with `target_any` = the intent phrase the title should carry) whenever a new high-value page is published, so the audit tracks whether its title gets optimized.
 - Remove terms that have gone stale.
+- Add a page to `impression_sinks` once it pulls large impression volumes at near-zero CTR (and remove it when it stops), so the ex-sinks view keeps tracking real docs performance.
 
 ## Cost awareness
 
 - **Volumes**: ~$0.02/run (45 keywords, batched 25/call). Negligible — run freely.
 - **SERP ranks**: ~$0.05–0.10/keyword → ~$2–3 for the full `rank_keywords` set. This is the costly phase and the account balance is finite. Use `--limit`, and if you see `40200`, stop and flag it. If budget is a recurring constraint, trim `rank_keywords` to the ~10 terms that matter most.
 - **Tech audit**: free (live fetches only).
+- **Search performance**: free (parses a Search Console CSV export).
 
 If DataForSEO is unavailable (not verified, or balance exhausted), you can still run Phase 3 (free) and do a qualitative Phase 4 via `WebSearch` — note the gap in the report rather than skipping the whole pulse.
diff --git a/.claude/skills/seo/config.json b/.claude/skills/seo/config.json
@@ -51,6 +51,9 @@
     { "url": "https://docs.steel.dev/cookbook/langgraph",        "section": "cookbook", "target_any": ["browser agent", "langgraph browser"] }
   ],
 
+  "$comment_impression_sinks": "Pages that accumulate large volumes of impressions at near-zero CTR, so they drag every site-level clicks/impressions/CTR number away from what real docs pages do. Excluded from the 'ex-sinks' view in reports (gsc-perf.mjs), never from the site as exported. `/llms-full.txt` is a single machine-readable bundle of the whole docs corpus, so it surfaces for a huge spread of loosely-related queries that humans do not click. `/overview/self-hosting/railway` matches queries about Railway the deployment platform rather than about Steel. Edit this list when another page starts behaving the same way.",
+  "impression_sinks": ["/llms-full.txt", "/overview/self-hosting/railway"],
+
   "competitors": [
     "browserbase.com", "browserless.io", "hyperbrowser.ai", "apify.com",
     "scraperapi.com", "scrapingbee.com", "brightdata.com", "zenrows.com", "zyte.com"

diff --git a/.claude/skills/seo/scripts/gsc-perf.mjs b/.claude/skills/seo/scripts/gsc-perf.mjs
@@ -0,0 +1,121 @@
+#!/usr/bin/env node
+// Phase 0, search performance. Reads a Search Console "Pages" CSV export and reports site
+// clicks/impressions/CTR twice: as-is, and "ex-sinks" with the `impression_sinks` pages from
+// config.json removed, so month-over-month comparisons aren't distorted by them.
+// Writes a dated snapshot + delta vs last month. No DataForSEO needed.
+// Usage: node gsc-perf.mjs <path-to-gsc-pages-export.csv>
+import { loadConfig, monthStamp, writeSnapshot, latestSnapshot, log } from './_lib.mjs';
+
+// Path-only comparison key: full URLs and bare paths in an export must match the same sink entry.
+function normalizePath(page) {
+  let path = page.trim();
+  try {
+    path = new URL(path).pathname;
+  } catch {
+    // already a path
+  }
+  return path.length > 1 ? path.replace(/\/$/, '') : path;
+}
+
+function splitCsvLine(line) {
+  const cells = [];
+  let cell = '';
+  let quoted = false;
+  for (const ch of line) {
+    if (ch === '"') quoted = !quoted;
+    else if (ch === ',' && !quoted) { cells.push(cell); cell = ''; }
+    else cell += ch;
+  }
+  cells.push(cell);
+  return cells.map((c) => c.trim());
+}
+
+function toNumber(cell) {
+  return Number(cell.replace(/[,%\s]/g, '')) || 0;
+}
+
+// Parse a Search Console pages export into { page, clicks, impressions } rows. Column order and
+// the exact page-column header ("Top pages", "Page", "URL") vary by export; match on the header.
+export function parseGscCsv(text) {
+  const lines = text.replace(/^/, '').split('\n').filter((l) => l.trim());
+  if (!lines.length) return [];
+  const headers = splitCsvLine(lines[0]).map((h) => h.toLowerCase());
+  const col = (re) => headers.findIndex((h) => re.test(h));
+  const pageCol = col(/page|url|address/);
+  const clicksCol = col(/click/);
+  const impressionsCol = col(/impression/);
+  if (pageCol < 0 || clicksCol < 0 || impressionsCol < 0) {
+    throw new Error(`unexpected export columns: ${headers.join(', ')}`);
+  }
+  return lines.slice(1).map((line) => {
+    const cells = splitCsvLine(line);
+    return {
+      page: cells[pageCol],
+      clicks: toNumber(cells[clicksCol]),
+      impressions: toNumber(cells[impressionsCol]),
+    };
+  });
+}
+
+function ratio(clicks, impressions) {
+  return impressions ? clicks / impressions : 0;
+}
+
+function totals(rows) {
+  const clicks = rows.reduce((s, r) => s + r.clicks, 0);
+  const impressions = rows.reduce((s, r) => s + r.impressions, 0);
+  return { clicks, impressions, ctr: ratio(clicks, impressions) };
+}
+
+// Site totals, the same totals with the sink pages removed, and each sink's own numbers.
+export function summarize(rows, sinkPaths) {
+  const sinkKeys = new Set(sinkPaths.map(normalizePath));
+  const isSink = (row) => sinkKeys.has(normalizePath(row.page));
+  const total = totals(rows);
+  const sinks = sinkPaths.map((path) => {
+    const matched = rows.filter((r) => normalizePath(r.page) === normalizePath(path));
+    const t = totals(matched);
+    return { path, ...t, impressionShare: ratio(t.impressions, total.impressions) };
+  });
+  return { total, exSinks: totals(rows.filter((r) => !isSink(r))), sinks };
+}
+
+function pct(value) {
+  return `${(value * 100).toFixed(2)}%`;
+}
+
+function line(label, t) {
+  return `${label}: ${t.clicks} clicks · ${t.impressions.toLocaleString('en-US')} impressions · ${pct(t.ctr)} CTR`;
+}
+
+if (import.meta.url === `file://${process.argv[1]}`) {
+  const csvPath = process.argv[2];
+  if (!csvPath) {
+    log('usage: node gsc-perf.mjs <path-to-gsc-pages-export.csv>');
+    process.exit(1);
+  }
+  const config = loadConfig();
+  const month = monthStamp();
+  const { readFileSync } = await import('node:fs');
+  const summary = summarize(parseGscCsv(readFileSync(csvPath, 'utf8')), config.impression_sinks);
+
+  const snapshotPath = writeSnapshot(config, 'perf', month, summary);
+  log(`wrote ${snapshotPath}`);
+
+  console.log(`\n# Search performance, ${month} (source: ${csvPath})`);
+  console.log(line('\nSite (as exported)', summary.total));
+  console.log(line('Site (ex-sinks)', summary.exSinks));
+  console.log('\nImpression sinks excluded (see config.json → impression_sinks):');
+  for (const s of summary.sinks) {
+    console.log(
+      `  ${s.path}: ${s.impressions.toLocaleString('en-US')} impressions (${pct(s.impressionShare)} of site), ${s.clicks} clicks, ${pct(s.ctr)} CTR`,
+    );
+  }
+
+  const prev = latestSnapshot(config, 'perf', month);
+  if (prev?.data?.exSinks) {
+    console.log(`\n${line(`vs ${prev.month} (ex-sinks)`, prev.data.exSinks)}`);
+  } else {
+    console.log('\nNo previous perf snapshot, so this run is the ex-sinks baseline.');
+  }
+}
diff --git a/scripts/seo/pulse/reports/seo-2026-07.md b/scripts/seo/pulse/reports/seo-2026-07.md
@@ -73,3 +73,29 @@ _Deferred — DataForSEO balance exhausted (40200). No positions captured this m
 - `snapshots/volumes-2026-07.json` (47 keywords, real data)
 - `snapshots/tech-2026-07.json` (16 key pages)
 - `snapshots/ranks-2026-07.json` — _not written (balance exhausted)_
+
+## Addendum (2026-07-30): Search Console baseline and the mid-July impression cliff
+
+A Search Console export for docs.steel.dev (Apr 29 to Jul 28 2026) landed after this report was written. Two pages absorb a third of all impressions at near-zero CTR, so the site-level numbers only mean something with those pages removed. Both are now listed in `config.json` → `impression_sinks`, and `gsc-perf.mjs` reports the two views side by side.
+
+| View | Clicks | Impressions | CTR |
+|---|---|---|---|
+| Site (as exported) | 613 | 208,324 | 0.29% |
+| **Site (ex-sinks)** | **586** | **134,125** | **0.44%** |
+
+Excluded pages (impression sinks):
+
+| Page | Impressions | Share of site | Clicks | CTR |
+|---|---|---|---|---|
+| `/llms-full.txt` | 37,527 | 18.0% | 10 | 0.03% (avg pos 17.9) |
+| `/overview/self-hosting/railway` | 36,672 | 17.6% | 17 | 0.05% (ranks for Railway-the-platform queries) |
+
+For scale, the pages people actually click run 0.85% to 1.7% CTR (pricing, proxies, docker), so 0.44% is the honest floor to beat, not 0.29%.
+
+**The Jul 10 to Jul 18 impression cliff was Google-side, not a regression from our changes.** Impressions/day fell from roughly 2,700 to roughly 1,300 across that window: last 28 days impressions are down 35% vs the prior 28 days while clicks held (down 9%) and CTR rose from 0.26% to 0.36%.
+
+- **Timing rules out our commits.** The decline starts and finishes before any July SEO work shipped: the P0 title/meta rewrites (#72) and the JSON-LD pass (#73) both merged Jul 23, cross-linking (#75) Jul 24, and the llms.txt/markdown work (#81 to #90) on Jul 29 and 30.
+- **The shape matches junk impressions evaporating.** Impressions fell while clicks stayed flat and CTR improved, which is what losing impressions that never converted looks like, not what losing traffic looks like. The two sink pages are the obvious source: they hold 35.6% of impressions at 0.03% to 0.05% CTR. Confirm next month by diffing their per-page impressions in `perf-2026-08.json`.
+- Monthly view, as exported (still sink-inflated, kept for continuity): May 200 clicks / 71,368 impressions / 0.28% / avg pos 12.45; June 214 / 82,792 / 0.26% / 7.99; July (28d) 182 / 49,963 / 0.36% / 7.52. Average position improving from 12.45 to 7.52 across the same stretch is another sign this is a matching change, not a demotion.
+
+**For next month's pulse:** compare against the ex-sinks row (586 clicks / 134,125 impressions / 0.44% CTR for Apr 29 to Jul 28), and read as-exported impression deltas as sink noise until those two pages settle.
diff --git a/tests/seo-ex-sinks.test.ts b/tests/seo-ex-sinks.test.ts
@@ -0,0 +1,91 @@
+// ABOUTME: Tests the SEO pulse Search Console summary: CSV parsing plus the ex-sinks totals
+// ABOUTME: that report site clicks/impressions/CTR with the impression-sink pages removed.
+import { describe, expect, test } from 'bun:test';
+import { readFileSync } from 'node:fs';
+import { fileURLToPath } from 'node:url';
+import { parseGscCsv, summarize } from '../.claude/skills/seo/scripts/gsc-perf.mjs';
+
+const CONFIG = JSON.parse(
+  readFileSync(
+    fileURLToPath(new URL('../.claude/skills/seo/config.json', import.meta.url)),
+    'utf8',
+  ),
+);
+
+const EXPORT_CSV = [
+  'Top pages,Clicks,Impressions,CTR,Position',
+  'https://docs.steel.dev/llms-full.txt,10,"37,527",0.03%,17.9',
+  'https://docs.steel.dev/overview/self-hosting/railway,17,"36,672",0.05%,10.1',
+  'https://docs.steel.dev/overview/pricing,100,"10,000",1%,6.2',
+  'https://docs.steel.dev/overview/stealth/proxies,50,"5,000",1%,7.4',
+].join('\n');
+
+describe('parseGscCsv', () => {
+  test('reads page, clicks, and impressions, tolerating quotes and thousands separators', () => {
+    const rows = parseGscCsv(EXPORT_CSV);
+    expect(rows).toHaveLength(4);
+    expect(rows[0]).toEqual({
+      page: 'https://docs.steel.dev/llms-full.txt',
+      clicks: 10,
+      impressions: 37527,
+    });
+    expect(rows[2].impressions).toBe(10000);
+  });
+
+  test('accepts a BOM and a "Page" header variant', () => {
+    const rows = parseGscCsv('Page,Impressions,Clicks\n/overview/pricing,10000,100\n');
+    expect(rows).toEqual([{ page: '/overview/pricing', clicks: 100, impressions: 10000 }]);
+  });
+});
+
+describe('summarize', () => {
+  const sinks = ['/llms-full.txt', '/overview/self-hosting/railway'];
+
+  test('reports site totals and the same totals excluding the impression sinks', () => {
+    const { total, exSinks } = summarize(parseGscCsv(EXPORT_CSV), sinks);
+    expect(total).toEqual({ clicks: 177, impressions: 89199, ctr: 177 / 89199 });
+    expect(exSinks).toEqual({ clicks: 150, impressions: 15000, ctr: 0.01 });
+  });
+
+  test('breaks out each sink with its share of site impressions', () => {
+    const { sinks: rows } = summarize(parseGscCsv(EXPORT_CSV), sinks);
+    expect(rows.map((r: { path: string }) => r.path)).toEqual(sinks);
+    expect(rows[0]).toEqual({
+      path: '/llms-full.txt',
+      clicks: 10,
+      impressions: 37527,
+      ctr: 10 / 37527,
+      impressionShare: 37527 / 89199,
+    });
+  });
+
+  test('matches sinks written as paths or full URLs, with or without a trailing slash', () => {
+    const rows = parseGscCsv(
+      ['Page,Clicks,Impressions', '/llms-full.txt,10,100', 'https://docs.steel.dev/x/,1,10'].join(
+        '\n',
+      ),
+    );
+    const { exSinks } = summarize(rows, ['https://docs.steel.dev/llms-full.txt', '/x']);
+    expect(exSinks).toEqual({ clicks: 0, impressions: 0, ctr: 0 });
+  });
+
+  test('treats a sink missing from the export as zero rather than failing', () => {
+    const rows = parseGscCsv('Page,Clicks,Impressions\n/overview/pricing,100,10000\n');
+    const summary = summarize(rows, sinks);
+    expect(summary.exSinks).toEqual({ clicks: 100, impressions: 10000, ctr: 0.01 });
+    expect(summary.sinks[0]).toEqual({
+      path: '/llms-full.txt',
+      clicks: 0,
+      impressions: 0,
+      ctr: 0,
+      impressionShare: 0,
+    });
+  });
+});
+
+describe('config', () => {
+  test('tracks the impression sinks that distort site-level numbers', () => {
+    expect(CONFIG.impression_sinks).toEqual(['/llms-full.txt', '/overview/self-hosting/railway']);
+    expect(CONFIG.$comment_impression_sinks).toContain('impression');
+  });
+});