Skip to content

Default to LIFO connection reuse (server_round_robin: false) so idle reaping works under sustained load - #22

Closed
micahjz wants to merge 3 commits into
mainfrom
mzirn/pr/pgcat-lifo-idle-reaping
Closed

Default to LIFO connection reuse (server_round_robin: false) so idle reaping works under sustained load#22
micahjz wants to merge 3 commits into
mainfrom
mzirn/pr/pgcat-lifo-idle-reaping

Conversation

@micahjz

@micahjz micahjz commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Flip the base pgcat chart to LIFO connection reuse by setting server_round_robin: false, so idle-connection reaping actually fires under sustained load.

Root cause

pgcat's code default for server_round_robin is true (src/config.rs:437, pinned by the test at src/config.rs:1866; the // False inline comment at line 324 is stale). true maps to QueueStrategy::Fifo (src/pool.rs:493).

In bb8 0.8.6, get() always pop_front() and under FIFO put() does push_back(). That makes the idle deque a true rotating queue: every checkout takes the front connection and returns it to the back. idle_start is reset only on checkin (put()), so under sustained traffic FIFO rotation touches every connection continuously and no connection ever reaches idle_timeout (30s). The reaper runs every 30s, finds nothing older than the threshold, and evicts zero. Result: connections_closed_idle_timeout = 0 while connections_created > 0, and the pool stays pinned at pool_size.

Evidence (env:prod*)

pool txn/s idle closes (6h)
agi_readonly 39,122 0
agi 8,134 0
finance_data 5.4 2,863

Low-traffic pools (finance_data, ~5 txn/s) reap normally because long quiet gaps let connections drift past 30s. High-traffic pools never get that gap. The running -ro pod confirms the effective value: pgcat::config: Server round robin: true.

Faithful bb8 simulation (agi_readonly-like pod)

  • FIFO (current): reaped = 0, idle depth stays ~205 (matches prod)
  • LIFO: reaped = 229, idle depth drains to ~12 (≈ min_idle)

Changes

  1. charts/pgcat/values.yaml — add server_round_robin: false (with explanatory comment).
  2. charts/pgcat/templates/secret.yamlrender the key into pgcat.toml's [general] block. The template previously omitted server_round_robin entirely, which is exactly why prod silently ran the FIFO default. Without this line the values change is inert.
  3. charts/pgcat/Chart.yaml — bump 0.2.5 → 0.2.6.

Non-goals / constraints respected

  • Does not touch server_lifetime (intentionally long to avoid synchronized mass expiry).
  • Does not touch min_pool_size.

Rollout note

This changes the base-chart default for all deployments consuming this chart. LIFO concentrates load on a small hot working set and lets the cold tail age out — desirable here. After rollout, confirm pgcat.pgcat_databases_connections_closed_idle_timeout.count starts firing for agi/agi_readonly and sv_idle drains toward min_pool_size.

@micahjz micahjz closed this Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant