Skip to content

restore.sh: take the standby parameter settings from pg_control - #72

Open
souravbiswassanto wants to merge 2 commits into
masterfrom
restore-gating-params
Open

souravbiswassanto wants to merge 2 commits into
masterfrom
restore-gating-params

Conversation

@souravbiswassanto

@souravbiswassanto souravbiswassanto commented Sep 17, 2026

Copy link
Copy Markdown
Member

A PITR restore replays WAL, which makes the recovery instance a standby. PostgreSQL then enforces that max_connections, max_worker_processes, max_locks_per_transaction and max_prepared_transactions are each >= the value recorded in the source's pg_control, or replay aborts:

FATAL:  recovery aborted because of insufficient parameter settings
DETAIL: max_connections = 200 is a lower setting than on the primary server,
        where its value was 500.

Nothing surfaces this. The pod stays Running and the Postgres object sits in Provisioning; the only evidence is in kubectl logs <db>-wal-restorer-N.

Scope changed — read this if you reviewed an earlier version

This PR originally claimed to be the fix, on the grounds that the config secret was not mounted into the wal-restore sidekick at all. That is now handled properly by kubedb/postgres#937, which projects the configSecret and the license into the restorer. Merge #937 first; it is the correctness fix.

The earlier description also argued the config secret could not be mounted, because the restore image was a generic Debian Postgres that cannot load pg_stat_monitor and friends. That argument is obsolete: kubedb/postgres-archiver#123 and kubedb/installer#2443 give the stream9 entries a matching -ext archiver, and an end-to-end PITR against it now works with the secret mounted.

What is left for this PR

One case #937 cannot cover, because it is not a bug in #937:

The config secret holds the CURRENT configuration. Replay is bound by what the source was running when the backup was taken.

So restoring an older backup after the secret has been lowered still aborts. Verified on a k3s cluster: source backed up at max_connections = 500, restore CR pointed at a configSecret saying 200, same repository and timestamp as a restore that otherwise succeeds —

2026-09-18 10:41:04 [23] LOG:  starting point-in-time recovery to 2026-09-18 10:24:10+00
2026-09-18 10:41:04 [23] FATAL:  recovery aborted because of insufficient parameter settings
2026-09-18 10:41:04 [23] DETAIL: max_connections = 200 is a lower setting than on the
                                 primary server, where its value was 500.

pg_control knows the right answer without anyone having to guess it, so read it from there.

This is now an ergonomics / robustness change, not a blocker. The failure it prevents is recoverable by hand — raise the value and re-apply — but the loop is expensive: PostgreSQL reports one violating parameter per attempt, a failed restore is blocked from retrying by its own leftover <name>-config and -auth secrets, and every retry re-downloads the base backup. On a 333 GB database that is hours per iteration.

Approach

pg_controldata is already in the image and reports exactly what the check compares against, so this needs no knowledge of how the source was configured.

Two details for review:

  • The block is appended after the role template, so it takes precedence over /etc/config/user.conf during recovery. That is deliberate — recovery must satisfy the source's values — and affects only the recovery instance; the restored database starts from its own configuration afterwards. Worth a second opinion: I have not tested this PR and #937 together on a cluster, only each separately.
  • It starts with an explicit printf '\n' because the role template does not end with a newline. Without it the first setting is silently absorbed into the template's trailing comment (# icu_validation_level = warningmax_connections = 200) — I hit exactly that during testing.

max_wal_senders is left alone unless the source ran with more than the 90 this script already pins.

Unrelated thing noticed nearby

Two lines below this hunk:

mv /tmp/postgresql.conf "$PGDATA/postgresql.conf"
echo "max_replication_slots = 90" >>/tmp/postgresql.conf

max_replication_slots is appended to /tmp/postgresql.conf after it has been moved away, so it never reaches $PGDATA. Pre-existing, not touched here, but someone should confirm whether it was meant to have an effect.

A PITR restore replays WAL, which makes the recovery instance a standby, so
PostgreSQL enforces that max_connections, max_worker_processes,
max_locks_per_transaction and max_prepared_transactions are each >= the value
recorded in the source's pg_control.

On a KubeDB database those are normally raised through the config secret. That
secret is mounted at /etc/config in the database pod, but the WAL-restore
sidekick gets no such mount, so the role template's
include_if_exists '/etc/config/user.conf' resolves to nothing and recovery
falls back to the built-in defaults of 100 / 8 / 64 / 0. Against any database
whose configuration raises them, replay dies immediately with

  FATAL:  recovery aborted because of insufficient parameter settings
  DETAIL: max_connections = 100 is a lower setting than on the primary
          server, where its value was 200.

and keeps retrying. Nothing surfaces it: the pods stay Running and the
Postgres object sits in Provisioning, so the only evidence is the restorer's
log.

Read the values out of the restored control file instead. pg_controldata is
already in the image and reports exactly what the check compares against, so
this needs no knowledge of how the source was configured and cannot drag in
settings the restore image cannot honour - notably shared_preload_libraries,
which this script deliberately pins to a minimal list because the restore
image does not carry the enterprise extensions. That is also why mounting the
config secret here is the wrong fix.

max_wal_senders is left alone unless the source ran with more than the 90 this
script already pins.

The block is appended after the role template so nothing downstream can lower
the values again, and it starts with an explicit newline because the template
does not end with one - without it the first setting is silently absorbed into
the template's trailing comment line.

Verified on a KubeDB cluster: a source with max_connections 200 and
max_worker_processes 24, restored to a chosen timestamp, previously needed the
values hand-appended inside the restorer pod to get past the parameter check.
With this change the same restore completes unattended and lands exactly on
the target - 3600 of 4200 rows, every row after the target correctly absent,
latest row 25 seconds before it, and the instance promoted out of recovery.

Signed-off-by: souravbiswassanto <saurov@appscode.com>
@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 6de0b132-ef79-452e-9318-b3a358e8e467


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…t is mounted

kubedb/postgres#937 projects the database's configSecret into the wal-restore
sidekick, so the role template's include_if_exists = '/etc/config/user.conf' now
resolves. The comment here claimed it never does, which stops being true the
moment that lands.

The block still has a job, and it is a narrower one: the config secret carries the
CURRENT configuration, while replay is bound by what the source was running when
the backup was taken. Restoring an older backup after the secret has been lowered
still aborts, and nothing surfaces it. Reword to say that instead.

Signed-off-by: souravbiswassanto <saurov@appscode.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant