restore.sh: take the standby parameter settings from pg_control - #72
Open
souravbiswassanto wants to merge 2 commits into
Open
souravbiswassanto wants to merge 2 commits into
souravbiswassanto wants to merge 2 commits into
Conversation
A PITR restore replays WAL, which makes the recovery instance a standby, so
PostgreSQL enforces that max_connections, max_worker_processes,
max_locks_per_transaction and max_prepared_transactions are each >= the value
recorded in the source's pg_control.
On a KubeDB database those are normally raised through the config secret. That
secret is mounted at /etc/config in the database pod, but the WAL-restore
sidekick gets no such mount, so the role template's
include_if_exists '/etc/config/user.conf' resolves to nothing and recovery
falls back to the built-in defaults of 100 / 8 / 64 / 0. Against any database
whose configuration raises them, replay dies immediately with
FATAL: recovery aborted because of insufficient parameter settings
DETAIL: max_connections = 100 is a lower setting than on the primary
server, where its value was 200.
and keeps retrying. Nothing surfaces it: the pods stay Running and the
Postgres object sits in Provisioning, so the only evidence is the restorer's
log.
Read the values out of the restored control file instead. pg_controldata is
already in the image and reports exactly what the check compares against, so
this needs no knowledge of how the source was configured and cannot drag in
settings the restore image cannot honour - notably shared_preload_libraries,
which this script deliberately pins to a minimal list because the restore
image does not carry the enterprise extensions. That is also why mounting the
config secret here is the wrong fix.
max_wal_senders is left alone unless the source ran with more than the 90 this
script already pins.
The block is appended after the role template so nothing downstream can lower
the values again, and it starts with an explicit newline because the template
does not end with one - without it the first setting is silently absorbed into
the template's trailing comment line.
Verified on a KubeDB cluster: a source with max_connections 200 and
max_worker_processes 24, restored to a chosen timestamp, previously needed the
values hand-appended inside the restorer pod to get past the parameter check.
With this change the same restore completes unattended and lands exactly on
the target - 3600 of 4200 rows, every row after the target correctly absent,
latest row 25 seconds before it, and the instance promoted out of recovery.
Signed-off-by: souravbiswassanto <saurov@appscode.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…t is mounted kubedb/postgres#937 projects the database's configSecret into the wal-restore sidekick, so the role template's include_if_exists = '/etc/config/user.conf' now resolves. The comment here claimed it never does, which stops being true the moment that lands. The block still has a job, and it is a narrower one: the config secret carries the CURRENT configuration, while replay is bound by what the source was running when the backup was taken. Restoring an older backup after the secret has been lowered still aborts, and nothing surfaces it. Reword to say that instead. Signed-off-by: souravbiswassanto <saurov@appscode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A PITR restore replays WAL, which makes the recovery instance a standby. PostgreSQL then enforces that
max_connections,max_worker_processes,max_locks_per_transactionandmax_prepared_transactionsare each>=the value recorded in the source'spg_control, or replay aborts:Nothing surfaces this. The pod stays
Runningand thePostgresobject sits inProvisioning; the only evidence is inkubectl logs <db>-wal-restorer-N.Scope changed — read this if you reviewed an earlier version
This PR originally claimed to be the fix, on the grounds that the config secret was not mounted into the wal-restore sidekick at all. That is now handled properly by kubedb/postgres#937, which projects the configSecret and the license into the restorer. Merge #937 first; it is the correctness fix.
The earlier description also argued the config secret could not be mounted, because the restore image was a generic Debian Postgres that cannot load
pg_stat_monitorand friends. That argument is obsolete: kubedb/postgres-archiver#123 and kubedb/installer#2443 give the stream9 entries a matching-extarchiver, and an end-to-end PITR against it now works with the secret mounted.What is left for this PR
One case #937 cannot cover, because it is not a bug in #937:
The config secret holds the CURRENT configuration. Replay is bound by what the source was running when the backup was taken.
So restoring an older backup after the secret has been lowered still aborts. Verified on a k3s cluster: source backed up at
max_connections = 500, restore CR pointed at a configSecret saying200, same repository and timestamp as a restore that otherwise succeeds —pg_controlknows the right answer without anyone having to guess it, so read it from there.This is now an ergonomics / robustness change, not a blocker. The failure it prevents is recoverable by hand — raise the value and re-apply — but the loop is expensive: PostgreSQL reports one violating parameter per attempt, a failed restore is blocked from retrying by its own leftover
<name>-configand-authsecrets, and every retry re-downloads the base backup. On a 333 GB database that is hours per iteration.Approach
pg_controldatais already in the image and reports exactly what the check compares against, so this needs no knowledge of how the source was configured.Two details for review:
/etc/config/user.confduring recovery. That is deliberate — recovery must satisfy the source's values — and affects only the recovery instance; the restored database starts from its own configuration afterwards. Worth a second opinion: I have not tested this PR and #937 together on a cluster, only each separately.printf '\n'because the role template does not end with a newline. Without it the first setting is silently absorbed into the template's trailing comment (# icu_validation_level = warningmax_connections = 200) — I hit exactly that during testing.max_wal_sendersis left alone unless the source ran with more than the 90 this script already pins.Unrelated thing noticed nearby
Two lines below this hunk:
max_replication_slotsis appended to/tmp/postgresql.confafter it has been moved away, so it never reaches$PGDATA. Pre-existing, not touched here, but someone should confirm whether it was meant to have an effect.