Skip to content

add Data Landscape & Egress Estimation stage #46

Merged
bencehezso merged 1 commit into
mainfrom
task/data_landscape_egress_fee_estimations
Jul 20, 2026
Merged

add Data Landscape & Egress Estimation stage #46
bencehezso merged 1 commit into
mainfrom
task/data_landscape_egress_fee_estimations

Conversation

@bencehezso

Copy link
Copy Markdown
Contributor

Summary

Adds an opt-in Stage #7 to the assessment pipeline: --egress on either aws or azure measures data-bearing resources in the assessed scope and estimates the one-time internet egress fee for moving that data out of the provider. Runs after report generation, uses only the existing Reader / ViewOnlyAccess permissions, and cannot fail the main assessment — a Stage 7 failure exits with the new EGRESS code (9) while the standard reports are still produced.

Marked alpha: sizes are point-in-time estimates from provider metrics (capacity metrics can lag ~48 h; provisioned disks count as upper bounds; backup vaults are not sized), and pricing is USD list prices only.

What ships

  • Collectors - core/utils_egress_azure.py sizes Storage Accounts (with blob-tier split), Managed Disks, Snapshots, SQL Databases, Cosmos DB, PostgreSQL/MySQL Flexible Servers, and detects Recovery Services vaults; core/utils_egress_aws.py sizes S3 buckets (with StorageType tier split), EBS volumes, EBS snapshots, RDS instances (with Aurora flagged not-sized), DynamoDB tables, and detects AWS Backup vaults. Azure uses the Monitor REST API directly (no new SDK); AWS uses CloudWatch GetMetricData.
  • Fee engine — tiered pricing with per-resource proration; archive retrieval rates applied to Glacier/Deep Archive/Azure Archive bytes; a "free tier" state when the footprint is below the first zero-price tier.
  • Outputsreports/<ts>/egress.html (dashboard), egress.pdf (executive summary: Summary + Scope + Coverage + Data Landscape + Estimated Costs + Appendix listing the exact prices used), and raw_data/egress_estimate.json.
  • Pricing lives in the dataset — new egresspricing table in datasets/data.db (effective-dated rows, per-row unit/zone/currency), so price updates ship without a code release. Every report folder captures its own point-in-time snapshot via assessment.db, making re-renders fully reproducible and pinning the PDF appendix numbers per report.

Impact on existing behavior

None.

@bencehezso bencehezso self-assigned this Jul 19, 2026
@bencehezso bencehezso added the enhancement New feature or request label Jul 19, 2026
@bencehezso
bencehezso merged commit 7a8d3b1 into main Jul 20, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant