feat: pg_stat_io latency, six performance findings, and the attribute-before-tune skill layer

[elkaix]

Summary

Adds the "attribute before you tune" layer: a pg_stat_io collector with per-read latency, six new deterministic findings, a memory-envelope fix for work_mem_overcommit, a Little's-law pool-size line in pgbot tune, and the storage verdict in pgbot why --duration. Everything stays read-only; the JSON contract goes 1.5.0 → 1.6.0 additively (one bump on top of main's 1.5.0; schema generated by tools/schemagen, TestSchema_matchesModel green).

Collectors / model

• io_stats (pg_stat_io, PG16+): double-sampled read/write/fsync rates per backend_type × object × context and, with track_io_timing on for the whole window, mean ms per op. PG18 wal rows count toward the rates but stay out of the latency means (they're timed by track_wal_io_timing). Latency, not the miss count, is what separates a page-cache hit from a wait on the device.
• settings.params gains the PG18 AIO knobs (io_method, io_workers, io_max_concurrency), effective_io_concurrency, hash_mem_multiplier, plan_cache_mode, max_slot_wal_keep_size, autovacuum_vacuum_max_threshold (PG18).
• PartitionRollup gains the hottest/largest leaf; WaitStudy gains io (pg_stat_io bracketed around the sampling window).

Findings (each with a docs page + unit tests)

id	fires when
io_read_latency_high	mean physical read ≥ 5 ms over ≥ 500 reads (critical ≥ 20 ms); names the heaviest reader
io_concurrency_low	effective_io_concurrency ≤ 1 on an SSD-backed managed provider
plan_cache_mode_forced	plan_cache_mode pinned away from auto cluster-wide
slot_wal_keep_unbounded	max_slot_wal_keep_size = -1 with replication slots present
partition_skew	hottest leaf ≥ 4× the per-partition average scans or rows
autovacuum_table_tuning	≥ 1M-row write-active table still on the global scale factor; shows current vs suggested per-table trigger (PG18: capped by autovacuum_vacuum_max_threshold)

• work_mem_overcommit now computes shared_buffers + work_mem × hash_mem_multiplier × max_connections vs effective_cache_size.
• wait_io_bound and the live why storage branch carry the pg_stat_io verdict: "device served them → cut blocks read first" vs "page cache served them → volume, not latency".
• connections_overprovisioned and pgbot tune print the average-busy-backends estimate and a pool size.

Docs / skill

• skills/postgres-diagnostics/SKILL.md: evidence → next hypothesis → refused reflex table, the layer order (query/schema → concurrency → memory/IO/WAL/vacuum → replicas → partitioning → sharding), and the one-change-with-cost/verify/rollback hand-off shape.

Also on this branch

• test(collect): port the replica-identity fixture to pggo: feat(findings): replica_identity_missing — a published table that can't be updated #113 landed after Switch the PostgreSQL driver from pgx to pgGo #114 dropped pgx from go.mod, so go vet / go test ./... on main fail at package setup (every CI job on main is red for this). Happy to split it into its own PR.

Release workflow

• Fork-safe: image namespace from IMAGE_REPO, GitHub release targets the running repository, brew-smoke gated to upstream. No behavior change for pgrundev/pgbot.

Test plan

Rebased on main @ acd485d (2026-10-01).

• go test ./..., go vet, gofmt, golangci-lint (0 issues), scripts/gate.sh (4 arches)
• Integration + doc-verify (PGBOT_TEST_SUPERUSER_DSN) against postgres:18 with track_io_timing=on, a hash-partitioned table with a hot tenant, and a 1.2M-row updated table: partition_skew rollup fields populated, autovacuum_table_tuning surfaced by tune, why --duration JSON carries study.io
• After rebase: go test -race ./..., vet, golangci-lint (0 issues); integration on postgres:16/17/18 (doc-verify on 18: 114 statements); new tests verified to fail with each fix reverted
• goreleaser check (config valid; the brews deprecation predates this PR)

[alexshapalov]

Hi @elkaix — apologies, the schema version has moved under you twice while this sat, and that is on me for not sequencing it sooner.

Where things stand: main is now at 1.4.0. 1.3.0 went to the collation_version_mismatch finding (#55) and 1.4.0 to inspect --all-instances (#38). This PR bumps to 1.3.0 and ships both a pgbot-context-1.3.0.json and a 1.4.0.json, so it collides with both.

This PR should claim 1.5.0. Nothing else is queued behind it, so that number is yours.

Rebasing onto main (you are 30 commits behind), the conflicts are:

CONFLICT (content): README.md
CONFLICT (content): internal/model/schema_version.go
CONFLICT (add/add): schema/pgbot-context-1.3.0.json
CONFLICT (add/add): schema/pgbot-context-1.4.0.json

For the two schema files: take main version of each as-is, drop your copies, and let 1.5.0 be the only one this branch adds.

One thing that will save you a confusing CI failure — do not hand-edit the schema JSON. It is generated, and TestSchema_matchesModel diffs the committed file against a fresh generation from the Go types, so any difference in key order or whitespace fails. After you bump the const in internal/model/schema_version.go, run:

go run ./tools/schemagen

That writes schema/pgbot-context-1.5.0.json for you. I got caught by this on #38.

One more worth a careful eye: internal/findings/findings.go merges cleanly, but both sides have changed it substantially since you branched — #55 added the collation finding, #65 fixed the replica-lag threshold gate. Git resolving it textually is not the same as it being right, so it is worth reading the merged result rather than trusting the clean merge.

Everything else in the 32 files still applies. Thanks for your patience on this one — the pg_stat_io work is the part of the backlog I most want to land.

[alexshapalov]

@elkaix pls resolve conflicts

[elkaix]

@alexshapalov rebased onto main and conflicts resolved.

• Schema is 1.6.0, not 1.5.0. feat(findings): replica_identity_missing — a published table that can't be updated #113 took 1.5.0 after your comment. Main's 1.3.0/1.4.0 files are taken as-is, and this branch adds only pgbot-context-1.6.0.json, generated by go run ./tools/schemagen.
• findings.go: I read the merged result rather than trusting the clean merge. feat(findings): collation_version_mismatch — the collation library changed under the data #55's collation finding and fix: preserve critical replica lag when warnings are raised #65's replica-lag gate are intact.
• Main is red independently of this PR. replident_integration_test.go (feat(findings): replica_identity_missing — a published table that can't be updated #113) still imports pgx after Switch the PostgreSQL driver from pgx to pgGo #114 removed it from go.mod, so every CI job fails at package setup. test(collect): port the replica-identity fixture to pggo fixes it. Happy to move it to its own PR if you'd rather land it first.
• Fixes from re-verifying against PG16/17/18, in fix(io_stats,findings): …:
  • PG18 pg_stat_io wal rows are timed by track_wal_io_timing, so they no longer dilute the latency means.
  • autovacuum_table_tuning respects PG18's autovacuum_vacuum_max_threshold cap.
  • idle_replication_slot_timeout is labelled PG18+.
  • A zero sample interval no longer emits Inf/NaN.

CI is waiting on "Approve and run" (fork PR).