Skip to content

feat(api): reclaim orphaned photo objects from S3 daily - #43

Open
MehrshadFb wants to merge 5 commits into
mainfrom
feat/photo-orphan-reconciler
Open

feat(api): reclaim orphaned photo objects from S3 daily#43
MehrshadFb wants to merge 5 commits into
mainfrom
feat/photo-orphan-reconciler

Conversation

@MehrshadFb

@MehrshadFb MehrshadFb commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add PhotoOrphanReconcilerService: walks every page of the photos/ prefix in S3 and deletes objects that no Photo row references
  • Add PhotoOrphanReconcilerScheduler, a daily 03:00 cron (@nestjs/schedule, newly added as a dependency along with ScheduleModule.forRoot())
  • Add S3Service.listObjects() on ListObjectsV2 with continuation-token paging, plus a LIST_FAILED error and unit tests
  • Only keys shaped like photos/{userId}/{eventId}/{photoId} or the legacy photos/{eventId}/{photoId} are candidates, and only when older than the minimum age; anything else under the prefix is skipped, never deleted
  • Config PHOTO_ORPHAN_RECONCILER_ENABLED (off unless set to exactly true) / _BATCH_SIZE (100) / _MIN_OBJECT_AGE_HOURS (24), documented in .env.example, and a new §10 in photos-architecture.md

Why

Postgres is the source of truth, so a row can vanish while its object stays in the bucket: an event or account delete cascades the rows and never touches S3, a manual row delete does the same, and pre-#38 keys still linger. Those bytes never count toward quota, because usage is a SUM over rows, so nothing surfaces them and we pay for them indefinitely.

A row is inserted before its upload URL is minted, so an object can only exist after its row did. That makes "no row" a sound deletion signal, and the 24h minimum age covers clock skew and uploads that landed moments before a scan.

Notes

  • The sweep is opt-in, and that is load-bearing. It decides what to delete from AWS_S3_BUCKET using rows in DATABASE_URL, and those two are only paired in a deployed environment. docker-compose.yml overrides DATABASE_URL to its own empty database while still loading the shared bucket credentials from .env, and local dev does the same, so an on-by-default sweep would delete another environment's live photos on its first 3am tick. Terraform configures no versioning on the bucket, so those deletes are final. Gating on NODE_ENV would not have helped, since compose sets it to production. Set the variable to true only where Postgres owns the bucket
  • Deliberately separate from the stale-PENDING cleanup on feat/photo-pending-cleanup, which starts from Postgres and solves the opposite failure. Separate service, scheduler, and config, as the brief asked. That branch is not on main, so this PR adds @nestjs/schedule itself; expect a small package.json and app.module.ts conflict if it merges first
  • Will also conflict with feat(api): store photo storage limit per user #42, which deletes src/config/photos.config.ts. If feat(api): store photo storage limit per user #42 merges first, keep this file with only the orphan keys and keep photosConfig in the ConfigModule load array
  • One indexed findMany({ s3Key: { in } }) per S3 page rather than a query per key. Nothing beyond the current page is held in memory
  • A row in any status protects its object, PENDING included
  • The batch cap counts failed deletes too and bounds the blast radius of a bad run more than the work: the run reports completed: false and the next one picks up the rest
  • Needs s3:ListBucket, which Terraform already grants

Test plan

  • npx jest --watchman=false (284 tests, 19 suites)
  • npx jest --config ./test/jest-e2e.json --watchman=false (94 tests)
  • npm run typecheck
  • npm run openapi:check
  • npx eslint on changed files, npm run format:check
  • CI passes on PR
  • Confirm the daily run in staging logs (photo.orphan_reconcile.completed) before trusting it in production

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

API unit-test coverage

Metric Coverage
Statements 70.3% (696/990)
Branches 52.14% (292/560)
Functions 74.71% (130/174)
Lines 68.66% (607/884)

Unit suite only; controllers are exercised by the e2e suite.

…onciler

# Conflicts:
#	api/docs/photos-architecture.md
…onciler

# Conflicts:
#	api/.env.example
#	api/docs/photos-architecture.md
#	api/src/config/photos.config.ts
#	api/src/photos/photos.module.ts
…onciler

# Conflicts:
#	api/docs/photos-architecture.md
#	api/src/config/photos.config.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant