feat(api): reclaim orphaned photo objects from S3 daily - #43
Open
MehrshadFb wants to merge 5 commits into
Open
Conversation
API unit-test coverage
Unit suite only; controllers are exercised by the e2e suite. |
…onciler # Conflicts: # api/docs/photos-architecture.md
…onciler # Conflicts: # api/.env.example # api/docs/photos-architecture.md # api/src/config/photos.config.ts # api/src/photos/photos.module.ts
…onciler # Conflicts: # api/docs/photos-architecture.md # api/src/config/photos.config.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PhotoOrphanReconcilerService: walks every page of thephotos/prefix in S3 and deletes objects that noPhotorow referencesPhotoOrphanReconcilerScheduler, a daily 03:00 cron (@nestjs/schedule, newly added as a dependency along withScheduleModule.forRoot())S3Service.listObjects()onListObjectsV2with continuation-token paging, plus aLIST_FAILEDerror and unit testsphotos/{userId}/{eventId}/{photoId}or the legacyphotos/{eventId}/{photoId}are candidates, and only when older than the minimum age; anything else under the prefix is skipped, never deletedPHOTO_ORPHAN_RECONCILER_ENABLED(off unless set to exactlytrue) /_BATCH_SIZE(100) /_MIN_OBJECT_AGE_HOURS(24), documented in.env.example, and a new §10 inphotos-architecture.mdWhy
Postgres is the source of truth, so a row can vanish while its object stays in the bucket: an event or account delete cascades the rows and never touches S3, a manual row delete does the same, and pre-#38 keys still linger. Those bytes never count toward quota, because usage is a
SUMover rows, so nothing surfaces them and we pay for them indefinitely.A row is inserted before its upload URL is minted, so an object can only exist after its row did. That makes "no row" a sound deletion signal, and the 24h minimum age covers clock skew and uploads that landed moments before a scan.
Notes
AWS_S3_BUCKETusing rows inDATABASE_URL, and those two are only paired in a deployed environment.docker-compose.ymloverridesDATABASE_URLto its own empty database while still loading the shared bucket credentials from.env, and local dev does the same, so an on-by-default sweep would delete another environment's live photos on its first 3am tick. Terraform configures no versioning on the bucket, so those deletes are final. Gating onNODE_ENVwould not have helped, since compose sets it toproduction. Set the variable totrueonly where Postgres owns the bucketfeat/photo-pending-cleanup, which starts from Postgres and solves the opposite failure. Separate service, scheduler, and config, as the brief asked. That branch is not onmain, so this PR adds@nestjs/scheduleitself; expect a smallpackage.jsonandapp.module.tsconflict if it merges firstsrc/config/photos.config.ts. If feat(api): store photo storage limit per user #42 merges first, keep this file with only the orphan keys and keepphotosConfigin theConfigModuleload arrayfindMany({ s3Key: { in } })per S3 page rather than a query per key. Nothing beyond the current page is held in memoryPENDINGincludedcompleted: falseand the next one picks up the rests3:ListBucket, which Terraform already grantsTest plan
npx jest --watchman=false(284 tests, 19 suites)npx jest --config ./test/jest-e2e.json --watchman=false(94 tests)npm run typechecknpm run openapi:checknpx eslinton changed files,npm run format:checkphoto.orphan_reconcile.completed) before trusting it in production