Is your feature request related to a problem?
Our CI/CD pipelines have vulnerabilities that could lead to deployment incidents, similar to what occurred with the Glific team. Addressing these gaps is crucial to ensure deployment success and minimize failures.
Describe the solution you'd like
- Ensure that Staging deploys wait for a successful CI run by updating
deploy-staging.yml with workflow_run and adding branch protection on main.
- Integrate a Discord webhook step to notify about staging and production deploy failures.
- Implement health checks for deployments by enabling the ECS deployment circuit breaker and polling the
rolloutState.
- Add Docker image builds in CI for every PR to catch issues earlier.
- Set up ECS staging rehearsals for every merge to main by scaling the staging ECS service and verifying it, with notifications on failures.
Original issue
Our CI CD pipelines can be hardened. There are gaps that need fixing, which will prevent from us from having similar deployment incidents like what Glific team shared recently.
More details here
1. Deploys wait for CI to be green
Staging deploy currently races CI instead of waiting for it. Change
deploy-staging.yml to trigger only after a successful CI run
(workflow_run), add branch protection on main with required checks, and
make create-release.yml confirm the tagged commit passed CI before building.
2. Discord notifications on deploy failures
Add a Discord webhook step (if: failure()) to the staging and production
deploy workflows. Today a red deploy on main is only visible if someone
opens the Actions tab — and Sentry can't catch failures that happen before
the app boots.
3. Verify deployments actually became healthy
- Enable the ECS deployment circuit breaker on our services.
- After
update-service, poll the deployment's rolloutState until
COMPLETED (pass) or FAILED / timeout (fail). A plain curl isn't enough —
the old task can answer with 200 while the new one is failing.
- Bake the git SHA into the image and return it from the health endpoint, so
we can confirm the new code is the one responding.
- On the EC2 staging box, wait for the container healthcheck after
docker compose up -d instead of reporting success when containers start.
4. Build the Docker image in CI on every PR
CI never builds the Dockerfile or validates the compose files, so breakage
there is only discovered during a deploy. Add docker build ./backend and
docker compose config checks to CI.
5. ECS staging rehearsal on every merge to main
Staging runs on EC2 (to save costs), so the production ECS deploy path is
never tested before a release. After the EC2 staging deploy completes
(serial, not parallel — the EC2 path owns migrations), scale the staging ECS
service from 0 to 1 with the new image, verify with the rolloutState poll,
then scale back to 0 (if: always()). One-time prep: remove autoscaling on
the staging service and enable its circuit breaker. A failed rehearsal pings
Discord but does not roll back EC2 staging.
Is your feature request related to a problem?
Our CI/CD pipelines have vulnerabilities that could lead to deployment incidents, similar to what occurred with the Glific team. Addressing these gaps is crucial to ensure deployment success and minimize failures.
Describe the solution you'd like
deploy-staging.ymlwithworkflow_runand adding branch protection onmain.rolloutState.Original issue
Our CI CD pipelines can be hardened. There are gaps that need fixing, which will prevent from us from having similar deployment incidents like what Glific team shared recently.
More details here
1. Deploys wait for CI to be green
Staging deploy currently races CI instead of waiting for it. Change
deploy-staging.ymlto trigger only after a successful CI run(
workflow_run), add branch protection onmainwith required checks, andmake
create-release.ymlconfirm the tagged commit passed CI before building.2. Discord notifications on deploy failures
Add a Discord webhook step (
if: failure()) to the staging and productiondeploy workflows. Today a red deploy on
mainis only visible if someoneopens the Actions tab — and Sentry can't catch failures that happen before
the app boots.
3. Verify deployments actually became healthy
update-service, poll the deployment'srolloutStateuntilCOMPLETED(pass) orFAILED/ timeout (fail). A plain curl isn't enough —the old task can answer with 200 while the new one is failing.
we can confirm the new code is the one responding.
docker compose up -dinstead of reporting success when containers start.4. Build the Docker image in CI on every PR
CI never builds the Dockerfile or validates the compose files, so breakage
there is only discovered during a deploy. Add
docker build ./backendanddocker compose configchecks to CI.5. ECS staging rehearsal on every merge to main
Staging runs on EC2 (to save costs), so the production ECS deploy path is
never tested before a release. After the EC2 staging deploy completes
(serial, not parallel — the EC2 path owns migrations), scale the staging ECS
service from 0 to 1 with the new image, verify with the rolloutState poll,
then scale back to 0 (
if: always()). One-time prep: remove autoscaling onthe staging service and enable its circuit breaker. A failed rehearsal pings
Discord but does not roll back EC2 staging.