inspect --all-instances: every Aurora writer and reader behind one endpoint (experimental)

[alexshapalov]

Implements #23 within its own constraints: SQL and DNS only, no AWS credentials, CLI, SDK, or API.

How it works

• aurora_replica_status() lists the members; exactly one writer is expected, anything else is refused rather than mislabeled.
• Each instance endpoint is derived from the entry endpoint's DNS name (<instance>.<cluster-id>.<region>.rds.amazonaws.com). Cluster, reader, custom, and instance endpoints all work as entry points; a custom domain is followed through its CNAME; RDS Proxy and non-RDS names are refused.
• aurora_db_instance_identifier() must confirm each derived endpoint reached the member it names before anything is collected.
• The --all-databases fan-out is generalized to (member, database) targets, so the two flags compose. Writer first, then readers, one member at a time. Cluster-wide findings dedupe per member.
• Output: text banners each target; --json carries server.instance and server.instance_role (contract 1.3.0, additive); SARIF/JUnit objects are prefixed instance:<id>/; Prometheus series gain instance and role labels.
• Any member that can't be reached is reported as partial coverage and the run exits 3.

Needs validation on a real cluster. I have no Aurora to run this against, so it is marked experimental and should not merge until someone with a cluster confirms it. @paul-enz, since you asked for this and validated the PgDog work so carefully, would you try it?

git fetch origin pull/38/head:aurora && git checkout aurora && go build -o /tmp/pgbot ./cmd/pgbot

# 1. discovery + derivation + identity check, no collection
PGBOT_AURORA_TEST_DSN="$AURORA_CLUSTER_URL" go test ./internal/conn/ -run TestIntegration_auroraInstances -v

# 2. the real thing
/tmp/pgbot inspect "$AURORA_CLUSTER_URL" --all-instances
/tmp/pgbot inspect "$AURORA_CLUSTER_URL" --all-instances --all-databases --json | jq '.[] | .server | {instance, instance_role, database}'

What I most want to know: whether every member was found and reached, whether the writer/reader roles are right, and, if derivation fails, the exact shape of your cluster endpoint (hostname with the ids redacted is fine). Also worth trying: the reader endpoint and an instance endpoint as the entry point, and the run through RDS Proxy, which should refuse cleanly.

[paul-enz]

@alexshapalov sorry I haven't got around to this yet. I do intend to test it out tomorrow and I'll get back to you.

[paul-enz]

Hey @alexshapalov Sorry for the wait. I gave this a run against an Aurora PostgreSQL 18.3 cluster, connecting straight to the cluster endpoint, and --all-instances refuses to start:

pgbot: discover instances: not an Aurora cluster (aurora_version() is missing) — --all-instances needs a native Aurora endpoint

It is a real Aurora cluster. The catalog probe in internal/conn/connect.go:249-253 is what misses it:

-- Aurora exposes aurora_version(). Look it up in the catalog rather
-- than CALLING it: on every other server the call fails, which writes
-- an ERROR to the server log and books a rollback in pg_stat_database
-- on each pgbot run — the very counter pgbot reports.
(SELECT count(*) FROM pg_proc WHERE proname = 'aurora_version') > 0

On 18.3 that returns 0. There are no aurora% rows in pg_proc at all, even though the functions are there and return exactly what discovery wants:

select aurora_version();            -> 18.3.4
select server_id, session_id from aurora_replica_status();
  -> one writer row (MASTER_SESSION_ID) plus one reader row

With IsAurora false, detectProvider falls through to rds, so AuroraInstances bails out before it ever runs the query.

Swap the probe to to_regprocedure. It finds them where pg_proc does not, and resolves the signature instead of calling it, so nothing lands in the server log or in pg_stat_database:

--- a/internal/conn/connect.go
+++ b/internal/conn/connect.go
@@ -246,7 +246,11 @@
 		       pg_is_in_recovery(),
-		       -- Aurora exposes aurora_version(). Look it up in the catalog rather
-		       -- than CALLING it: on every other server the call fails, which writes
-		       -- an ERROR to the server log and books a rollback in pg_stat_database
-		       -- on each pgbot run — the very counter pgbot reports.
-		       (SELECT count(*) FROM pg_proc WHERE proname = 'aurora_version') > 0`
+		       -- Aurora exposes aurora_version() and aurora_replica_status(), but
+		       -- does not catalogue them in pg_proc on 18.3. to_regprocedure finds
+		       -- them by signature without CALLING them, so no ERROR is written to
+		       -- the server log and no rollback is booked in pg_stat_database.
+		       to_regprocedure('aurora_version()') IS NOT NULL
+		       OR to_regprocedure('aurora_replica_status()') IS NOT NULL`

and the marker comment goes stale too:

--- a/internal/conn/provider.go
+++ b/internal/conn/provider.go
@@ -30,1 +30,1 @@
-	IsAurora    bool // aurora_version() exists in pg_proc
+	IsAurora    bool // aurora_version() or aurora_replica_status() is resolvable

Both predicates are true on 18.3. I built that locally and discovery gets past detection and on to host derivation.

[alexshapalov]

Merged, with your to_regprocedure fix — thank you for chasing this down. Resolving both functions by signature rather than looking them up in pg_proc is exactly right, and it keeps the property the original probe was written for. I confirmed that half on plain PostgreSQL 15.12: the predicate returns false without raising, and pg_stat_database.xact_rollback is unchanged across repeated runs, while SELECT aurora_version() does raise and does increment it.

The branch also had to move its schema bump to 1.4.0 — 1.3.0 went to the collation finding in #55 while this was open.

Could I ask you for one more run against your cluster, now on main? Your earlier report confirmed detection gets past and on to host derivation, which is the part that was broken — but not that a full --all-instances run completes. That is the half nobody has seen yet, and it is the part most likely to have a second problem hiding in it.

Most useful to know:

1. Does the run finish, and does it reach every member — one writer plus each reader?
2. Does endpoint derivation produce reachable hostnames for your cluster endpoint shape? That is the piece I would most expect to break on a cluster that is not shaped like the common case, and it is why the flag ships marked experimental.
3. If a member is unreachable, does it fail loudly and exit 3 rather than silently reporting partial coverage?
4. Anything odd in --json — server.instance and server.instance_role should name each member.

If derivation does fail, the cluster endpoint shape is the thing to paste; that is what I would need to fix it.

No rush, and thanks again — this would have shipped broken for every Aurora 18.3 user without your report.

[alexshapalov]

@paul-enz sorry for late response!