Known-open defects after v0.16.0 — 0.16.1 scoping

[chlunde]

Tracking list of known-open defects after v0.16.0, so 0.16.1 can be scoped deliberately rather than by whoever shouts loudest.

Read this first: almost nothing here should hold up 0.16.1. One item (A1) is a genuine 0.16.0 regression with a one-line fix and is the only thing I'd argue for blocking on. A2 is a second small regression worth taking if it's cheap. Everything in the "pre-existing" tables has been broken for one or more releases, so shipping 0.16.1 without it regresses nobody.

Verified against upstream/master @ fe39828 (2026-08-11). Detailed write-ups with proofs live in POTENTIAL_1.16_ISSUES.md (item numbers below match it); the release triage view is in the remediation-plan artifact.

Every CRD below exists twice — cluster-scoped (*.sql.crossplane.io) and namespaced (*.sql.m.crossplane.io). Fixes go in one PR covering both trees, per CLAUDE.md.

---

A. 0.16.1 candidates (regressions introduced in 0.16.0)

A1 — Deleting a Role's connection Secret rotates the live database password · P1 · Role (postgresql)

Items 54 and 55 in POTENTIAL_1.16_ISSUES.md; standalone write-up in ISSUE_348_password_rotation.md. Introduced by crossplane-contrib#348 (be0bb83), shipped in v0.16.0.

Create never stamps Status.AtProvider.LastPasswordChange — the only stamp is in Update (pkg/controller/cluster/postgresql/role/reconciler.go:359). So for every provider-created role that field stays nil forever, and shouldResetPassword (role/utils.go:76) branches on exactly that:

last := role.Status.AtProvider.LastPasswordChange
if last != nil {
    if role.Spec.ForProvider.PasswordRotationTrigger != nil {
        return role.Spec.ForProvider.PasswordRotationTrigger.After(last.Time), nil  // item 55: unreachable
    }
    return false, nil
}
// last == nil -> the connection Secret decides:
return true, nil                                        // item 54: Secret absent -> rotate
return len(s.Data[...PasswordKey]) == 0, nil            // item 54: empty password -> rotate

Two symptoms from one asymmetry:

• 54 — a GitOps prune / kubectl delete secret / namespace cleanup makes the next reconcile run ALTER ROLE ... PASSWORD ..., and consumers holding the old password stop authenticating.
• 55 — passwordRotationTrigger, the documented way to rotate, is never read for a normally-created role, so it silently does nothing.

Honest bounds (this is why it's P1 and not "drop everything"): only affects roles with a provider-generated password (passwordSecretRef unset) and writeConnectionSecretToRef set; fires only when the Secret is actually missing or has an empty password key; self-limiting, because the first reset stamps the field. BYOP roles never reach the heuristic.

Fix: stamp LastPasswordChange in Create right after the successful CREATE ROLE, both trees. One line each, and it repairs 54 and 55 together. Note master already ships TestGetPassword/NilLastPasswordChangeSecretNotFound asserting changed: true for this exact input — that test encodes the bug and has to change with the fix.

Reproduce (unit, no database): a Role with WriteConnectionSecretToReference set and LastPasswordChange == nil, test.MockClient whose Get returns NotFound; assert getPassword returns changed == false. Fails on master. For item 55: same role with a healthy Secret and PasswordRotationTrigger an hour in the future; assert changed == true. Both were run in a throwaway worktree at fe39828 and failed as expected.

Related, and worth gating: PR crossplane-contrib#424 ports this design to MySQL User with the same two defects (LastPasswordChange stamped only in UpdatePassword, missing Secret ⇒ true). Ask for Create to stamp the field before merging, or it lands the same bug in a second engine.

A2 — Legacy objectType: schema resources can't be deleted after crossplane-contrib#379 · P1 · DefaultPrivileges (postgresql)

Item 53. Introduced by crossplane-contrib#379, shipped in v0.16.0.

crossplane-contrib#379 made schema optional and added guards to Observe (default_privileges/reconciler.go:255,259). The second guard rejects objectType: schema with a schema set — which is every such resource created on 0.14/0.15, because schema was +required then.

Because an Observe error returns before the deletion block in crossplane-runtime's managed reconciler (managed/reconciler.go:1118 vs :1165), those resources can't be deleted: the finalizer is never removed and kubectl delete hangs. CEL doesn't help — XValidation runs on writes and never re-validates stored objects.

Fix (~5 lines/tree): when validation fails but meta.WasDeleted(mg) is true, return ResourceExists: false instead of an error. Create always failed for these, so there's no external state to revoke. Needs a regression test with a deletion timestamp in both trees.

Alternative if 0.16.1 needs to be small: ship a release note telling upgraders to remove spec.schema from objectType: schema resources, and fix it in 0.17.

---

B. Also new in 0.16.0, but not release-blocking

Item	CRD	Pri	Summary
52	User (mysql)	P2	authenticationPlugin.authString: "" never converges — Observe maps '' to nil, authPluginEqual (mysql/user/reconciler.go:296) reads that as drift, ALTER USER ... IDENTIFIED WITH re-fires every reconcile. Needs a deliberately empty string. ~2 lines: normalise ""↔nil, or add MinLength=1.
56	User (mysql)	P1*	Adding authenticationPlugin: {name: caching_sha2_password} to a User that already has a password makes Update emit ALTER USER ... IDENTIFIED WITH <plugin> with no AS clause (:494-506). *The provider half is proven; the engine half is not — nobody has confirmed MySQL actually clears the credential in that case. Run it against mysql:8 before treating it as P1; if the credential survives, this folds into item 52.
57	User (mysql)	P3	user_types.go:82 tells you to use passwordSecretRef with a native password plugin; the CEL rule at :38 makes those two mutually exclusive. Doc-only.
58	User (mysql)	P3	Not a new defect — crossplane-contrib#363 added call sites of the QuoteValue helper from item 12 (below), including the user-controlled authString. Just means item 12's fix has more callers than it used to.

---

C. Pre-existing — none of this regresses 0.16.x

Broken for at least one release. High user pain in places, but shipping 0.16.1 without them changes nothing for anyone.

PostgreSQL

Item	CRD	Pri	Summary
3, 46, 51	DefaultPrivileges	P1	Observe filters on neither defaclnamespace nor defaclrole, so resources differing only by schema or target role alias onto each other and never converge; ALL is never expanded on the desired side; Create's REVOKE ALL erases sibling CRs' privileges. Broken since 0.14. One query rewrite; 51 needs a semantics decision first.
16, 17, 18	Grant	P2	Observe demands the ACL equal the spec exactly, while Create only revokes the spec's own privileges. Consequences: two Grants on one object fight forever (18); a grant to the object's owner never converges (17); a drifted Grant reports ResourceExists: false, so deleting it skips the REVOKE and leaves the privilege behind (16 — e2e-verified for routine Grants too on 2026-09-03: EXECUTE remains after the CR is gone). crossplane-contrib#421 fixed the database-level case; tables/sequences/schemas still exact-match — see selectTableGrantQuery's array_agg(...) = .... Containment vs. equality is a design decision — state it in the PR.
—	Grant (routines)	P2	Routine args whose pg_catalog.format_type() spelling differs from the spec GRANT fine and never converge: Observe returns not-exists (grant/reconciler.go:614), Create re-issues the GRANT every reconcile, and delete skips the REVOKE (item 16). Three spelling classes, all verified on postgres:18 both trees: public.-qualified (rendered bare), pg_catalog.-qualified (rendered bare), and names outside [a-z0-9_] such as cred$v2 (rendered double-quoted; the provider lower-cases, so upper-case quoted type names are in the same class). The first two became expressible with crossplane-contrib#436; the $ case is pre-existing. text[] and multi-word names remain unexpressible. Tracked upstream with a fix direction (::regtype comparison) in crossplane-contrib#439.
4, 6, 48, 50	Role	P1	upToDate (role/reconciler.go:437) compares pointers, not values, so every role reports out-of-date from reconcile #2 and Update re-issues ALTER ROLE ... CONNECTION LIMIT forever (issue crossplane-contrib#194). Also: Create and Update quote configurationParameters differently and Observe can't parse what PostgreSQL stores back (6); out-of-band rolconfig drift is detected but never written (48); Create omits CONNECTION LIMIT (50).
41, 42	Database	P2	isTemplate: true makes the CR undeletable — Delete issues a bare DROP DATABASE (database/reconciler.go:311) and PostgreSQL refuses. upToDate also diffs four axes Update can't write, so non-canonical encoding spellings loop forever; owner: DEFAULT, which the field's own doc recommends, is a hard create failure.
43	Extension	P1	spec.forProvider.schema is a no-op end to end — not emitted by Create, not read by Observe, Update is empty. Version drift is diffed but never written. Decide: implement it or remove it (removing later is breaking).
44	Schema	P2	revokePublicOnSchema is write-only — never observed, so out-of-band re-grants to PUBLIC are invisible drift on a security knob. Same class as the already-fixed item 2; commit 93b607b's acl.grantee = 0 clause is the template.
31, 45	clients / all PG CRDs	P1	DSN() (pkg/clients/postgresql/postgresql.go:50) embeds the password in a URL; when it fails to parse, lib/pq echoes the whole DSN — password included — into status.conditions (issue crossplane-contrib#266). Same function: database is concatenated without url.PathEscape, so a crafted name can override sslmode/host. Pre-existing, so not a 0.16.1 blocker, but it's cheap and it's a credential leak.

MySQL

Item	CRD	Pri	Summary
30, 7, 8	Grant	P1	MySQL collapses a complete privilege list to ALL PRIVILEGES; diffPermissions compares that against the user's explicit list and issues REVOKE ALL + re-GRANT every reconcile — a real window with no privileges (issue crossplane-contrib#162). Also: REVOKE ... WITH GRANT OPTION (mysql/grant/reconciler.go:364,380) is invalid MySQL, so deleting such a grant wedges the finalizer. The e2e suite runs MariaDB, which accepts that syntax — that's why it never caught it.
5	User	P1	Same pointer comparison as item 4 (mysql/user/reconciler.go:566): any User with resourceOptions never reports up-to-date.
9	Database	P1	defaultCollation is late-initialised from the server into the spec; a later charset-only change then emits an incompatible COLLATE and Update fails forever.

MSSQL

Item	CRD	Pri	Summary
19, 20, 21	User	P1	Contained-user support shipped in 0.15 with zero tests. The immutability CEL rule has an unset→set hole, so contained: true can be patched onto an existing login-mapped user; Delete then drops the user but not the login, leaving a working server credential behind. The namespaced tree emits USE [db];, which Azure SQL rejects, and database isn't required, so the cluster tree silently creates the user in master.
11, 12	Grant, clients	P1/P3	mssql.QuoteIdentifier (pkg/clients/mssql/mssql.go:130) doesn't escape ], and the grant path interpolates the schema name raw (mssql/grant/reconciler.go:258). Mostly a correctness bug: a schema named My-Schema simply doesn't work.
23	Grant	P2	Update revokes every database-class permission not in the spec, so two Grants on one user fight, and any Grant omitting CONNECT locks the user out. Pre-existing since 2021 — document the one-Grant-per-user constraint, or adopt whatever containment semantics items 16–18 land on.

Cross-cutting

Item	Affects	Pri	Summary
35	every CRD	P2	Connection details are returned from Update only when the password changed, so a ProviderConfig endpoint change never reaches the managed resource's Secret (crossplane-contrib#242) and adopted roles never publish a username (crossplane-contrib#77). Touches every reconciler — schedule it alone, it conflicts with everything.
22	ClusterProviderConfig (postgresql, mssql)	P2	Only the MySQL namespaced config package registers a reconciler for its ClusterProviderConfig, so the PG and MSSQL ones get no finalizer and no status.users — delete one while in use and it goes immediately, breaking every dependent MR.
—	every CRD	P3	Any MR stuck in a Create loop (items 4, 5, 16–18, 41, the routine row above) takes >30 s to delete and can exceed a 60 s kubectl delete --timeout: each re-Create refreshes external-create-succeeded, and crossplane-runtime's managed reconciler requeues inside the 30 s creation grace period before it removes the finalizer. Not a provider defect, but it is what a "delete hangs" report on one of those items looks like. Observed 2026-09-03.
25	provider startup	P3	Every Setup registers a state-metrics recorder whose Start returns the List error, which kingpin.FatalIfError turns into process exit — a CRD not established ~5s after start crash-loops the provider instead of leaving a metric unpopulated. ~5 lines: log instead of return, plus a nil guard on MetricOptions.
27	namespaced CRDs	P3	Marker defects from the tree split: namespaced MySQL Grant doesn't require forProvider, namespaced MSSQL User lost its categories, namespaced MySQL ProviderConfig TLS refs are cross-namespace, and 4 of 6 ProviderConfig CRDs have a stale SECRET-NAME printcolumn. Includes a real nil-deref: Observe panics on a Grant with no user in both trees.
49	all PG CRDs	P2	No adoption or ownership policy: two CRs pointing at one external object produce owner ping-pong (Database, Schema), silently stale credentials (Role), or destructive CASCADE deletes. Needs one documented answer, not six per-resource behaviours.

---

Working on any of these

• Both trees, one PR. pkg/controller/cluster/... and pkg/controller/namespaced/..., plus apis/cluster and apis/namespaced. Diff the two diffs against each other before review.
• Detail is in POTENTIAL_1.16_ISSUES.md — each item there has the proof, the exact SQL or Go, the ruled-out hypotheses, and a fix direction. Read the item before starting; several have a "this looked like a bug and isn't" note that will save a round.
• Probes get promoted, not deleted. Any throwaway test that demonstrates one of these should land as a regression test with the fix.
• Observe must issue zero writes on reconcile chore: bump korthout/backport-action from 1.4.0 to 3.2.0 #2. Most items above are that invariant failing.
• Unit tests use go-sqlmock / table-driven go-cmp; make reviewable before pushing; e2e needs Docker and only one run at a time (the KIND cluster name is shared).

Already fixed — don't re-do these

crossplane-contrib#436 (merged 2026-09-03, 43bb714) — schema-qualified composite types in PostgreSQL Grant routines[].args, e.g. aws_commons._s3_uri_1. E2e-verified on postgres:18, both trees: converges, stable on re-reconcile, delete revokes exactly the targeted overload. Follow-ups (non-converging spellings, unit-test gap) live in crossplane-contrib#439 and the routine row above.

#1 routine-grant argument quoting, #15 grants on views/partitioned tables, #47a Connect on backends without server_version_num, #47b database grants no longer needing a session on the target DB, #2 revokePublicOnDb, #17 for databases (tables/sequences/schemas still open, see above) — all in crossplane-contrib#421. #32 objectType: schema SQL in crossplane-contrib#379. #34 shared login in crossplane-contrib#411. #36 x/net CVE in crossplane-contrib#407. #14/#37 CodeQL version mismatch via the action bump.

[fernandezcuesta]

Regarding why we didn't implement lastPaswordChange (emphasis in Change) for create, as reported in A1 can be read here