App migration fix - #216
Merged
Merged
Conversation
The backend and frontend Deployments used strategy: Recreate, which stops the old pod before starting the new one, causing a real gap in the middle of every deploy. This traced to a live incident: a RollingUpdate surge pod would fail to mount the ReadWriteOnce uploads PVC while the old pod still held it, which combined with a node hitting DiskPressure during the last backend rollout to strand ~29 pods in ContainerStatusUnknown and extend what should have been a brief blip into real downtime. - senate-uploads PVC switches to ReadWriteMany (the ontap-nas-economy backing storage class supports it); migrate-uploads-to-rwx.sh handles the one-time swap since accessModes is immutable on an existing PVC. - senate-backend Deployment switches to RollingUpdate (maxSurge: 1, maxUnavailable: 0), which fits the project's memory quota exactly. - senate-frontend stays on Recreate for now - a surge pod needs another 1Gi the quota doesn't have room for - documented in the README along with what to ask UNC IT for. - Both Deployments get a startupProbe (shrinks the Recreate/rollout gap by not hiding a fast boot behind a flat initialDelaySeconds) and a preStop hook (gives the Service/Route time to stop routing to a pod before it gets SIGTERM). - README documents the additive-only init_db.py migration approach and the manual expand/contract playbook destructive schema changes still need.
Checked live usage with oc adm top pods: backend and DB were using ~85-95Mi against 512Mi limits each, frontend ~550Mi against 1Gi. That's a lot of committed-but-idle quota, all of it counted against the project's 2.5Gi hard cap regardless of actual use. Lowering BACKEND_MEMORY_LIMIT to 256Mi (still ~3x measured usage) and FRONTEND_MEMORY_LIMIT to 896Mi (~1.6x measured peak) frees exactly enough room for a frontend surge pod to fit too - 512(db) + 256(backend) + 896(frontend) = 1664Mi steady, + up to 896Mi surge = 2560Mi, the exact cap. db's limit is left untouched on purpose: a Postgres OOM-kill is a worse failure mode than a slow web process. senate-frontend switches to RollingUpdate (maxSurge: 1, maxUnavailable: 0) now that it fits, closing the last remaining downtime window on ordinary deploys without waiting on a UNC IT quota increase. Zero slack is left in the quota either way, which the README now documents along with what that means if frontend and backend rollouts ever overlap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes deploy downtime and a related outage caused by the backend/frontend
Deployments using
Recreateinstead ofRollingUpdate. Traced to a realincident: a rollout hit a node DiskPressure condition, and the uploads
PVC being ReadWriteOnce turned that into ~29 pods stuck in
ContainerStatusUnknown and extended downtime instead of a quick blip.
Changes:
surge pod can mount it alongside the still-terminating old pod, instead
of hitting a Multi-Attach error. Added
migrate-uploads-to-rwx.shsinceaccessModesis immutable on anexisting bound PVC — this needs to be run once, live, before this
template is applied.
senate-backendandsenate-frontendDeployments fromRecreatetoRollingUpdate(maxSurge: 1, maxUnavailable: 0), so adeploy never has a window with zero pods serving traffic.
BACKEND_MEMORY_LIMIT(512Mi→256Mi) andFRONTEND_MEMORY_LIMIT(1Gi→896Mi) based on live
oc adm top podsusage (~85Mi and ~550Mirespectively) — frees just enough of the project's fixed 2.5Gi memory
quota for a frontend surge pod to fit too, no UNC IT quota increase
needed.
db's limit is deliberately untouched.startupProbe(polls every 2s instead of a flat 20sinitialDelaySeconds) and apreStophook (lets the Service/Route stoprouting before SIGTERM) to both app Deployments.
playbook (
init_db.pyis additive-only by design — drops/renames/required-no-default columns need manual expand-contract handling), and
the Multi-Attach incident in
deploy/cloudapps/README.md.