feat: add sequin date-based backfill script and missing backfill indexes#4385
feat: add sequin date-based backfill script and missing backfill indexes#4385epipav wants to merge 1 commit into
Conversation
…indexes Signed-off-by: anilb <epipav@gmail.com>
PR SummaryMedium Risk Overview A Flyway migration creates New Reviewed by Cursor Bugbot for commit ceca499. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
|
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ceca499. Configure here.
| return data | ||
| except (urllib.error.URLError, OSError, json.JSONDecodeError) as e: | ||
| last_err = e | ||
| die(f"cannot reach Sequin API at {self.host}: {last_err}") |
There was a problem hiding this comment.
HTTP 404 never reaches handlers
Medium Severity
SequinApi.get catches urllib.error.URLError, which includes HTTPError, so a 404 from /api/postgres_databases or /api/sinks is treated like a connectivity failure. The script retries HTTP and exits instead of running the HTTPError handlers that fall back to /api/databases or report a missing sinks endpoint.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit ceca499. Configure here.
| ON versions (last_synced_at, id, package_id); | ||
|
|
||
| CREATE INDEX CONCURRENTLY IF NOT EXISTS package_dependencies_updated_at_id_depends_on_id_idx | ||
| ON package_dependencies (updated_at, id, depends_on_id); |
There was a problem hiding this comment.
Concurrent index on partitioned tables
High Severity
The migration runs CREATE INDEX CONCURRENTLY on versions and package_dependencies, which are hash-partitioned parents. On PostgreSQL 14 (the version used for the packages DB in this repo), concurrent index builds on partitioned tables are rejected, so Flyway can fail before either backfill index exists.
Reviewed by Cursor Bugbot for commit ceca499. Configure here.
There was a problem hiding this comment.
Pull request overview
Adds tooling and indexes for date-based Sequin backfills.
Changes:
- Adds an interactive/non-interactive backfill script.
- Adds keyset-pagination indexes for large package tables.
Metadata: PR title is missing the required JIRA key.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
scripts/sequin-backfill.py |
Plans and triggers date-based Sequin backfills. |
backend/src/osspckgs/migrations/V1784745489__sequin_backfill_sort_column_indexes.sql |
Adds backfill sort indexes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| CREATE INDEX CONCURRENTLY IF NOT EXISTS versions_last_synced_at_id_package_id_idx | ||
| ON versions (last_synced_at, id, package_id); |
| CREATE INDEX CONCURRENTLY IF NOT EXISTS package_dependencies_updated_at_id_depends_on_id_idx | ||
| ON package_dependencies (updated_at, id, depends_on_id); |
| class SequinApi: | ||
| def __init__(self, ctx): | ||
| self.token = ctx["api_token"] | ||
| self.host = ctx["hostname"] | ||
| self.base = None # resolved on first request |
No description provided.