DM-55094: Add package version data model and its ConsDB round-trip - #193
Open
mfisherlevine wants to merge 1 commit into
Open
DM-55094: Add package version data model and its ConsDB round-trip#193mfisherlevine wants to merge 1 commit into
mfisherlevine wants to merge 1 commit into
Conversation
mfisherlevine
force-pushed
the
tickets/DM-55094
branch
3 times, most recently
from
July 27, 2026 16:06
eadf90b to
95c82e5
Compare
mfisherlevine
force-pushed
the
tickets/DM-55094
branch
7 times, most recently
from
September 3, 2026 13:45
7b280b0 to
6dabc67
Compare
mfisherlevine
force-pushed
the
tickets/DM-55094
branch
from
September 4, 2026 16:25
6dabc67 to
4976db2
Compare
Rapid analysis records, for every image it dispatches, the versions of the
science packages that determine the AOS results. This adds the read-only
data model for those records: PackageVersions holds the versions in a dict
keyed by package name (so the tracked set can grow without changing the
wire format), and a version set is identified by versionHash - a stable,
order-independent SHA-256 of the versions, a pure function of the data that
needs no state to compute. toDict pins the ConsDB wire shape,
{"hash": ..., "versions": {...}}, with the versions inline so a row is
self-contained.
The model lives here rather than in rapid analysis so that users without
access to that package can consume the data, and it deliberately exposes
only readers: an ordinary summit_utils user can read the versions, never
write them. The writes (and the scraping of the versions themselves) live
in rapid analysis, which builds its blobs from toDict and the
PACKAGE_VERSIONS_TABLE / PACKAGE_VERSIONS_COLUMN constants defined here, so
the two sides cannot drift. Those default to
cdb_<instrument>.exposure_quicklook.package_versions until the ConsDB
schema change lands; both are parameterised (and validated as SQL
identifiers) in case it lands elsewhere.
Three readers cover the common lookups, sharing a _packageVersionsFromCell
helper so the null/object/string tolerance of the JSONB cell parsing stays
consistent:
- readPackageVersionsFromConsDb(client, instrument, dayObs, seqNum), the
base read.
- readPackageVersionsForExposure, taking the dataId straight off an
exposure DimensionRecord instead of its unpacked components.
- readPackageVersionsByHash, resolving a version hash to its versions via a
package_versions->>'hash' filter (LIMIT 1, since the versions are
identical across every row sharing the hash). The hash is validated as a
bare hex digest and lowercased to the stored form before being
interpolated into the query, so an uppercase paste still matches.
The wire shape is pinned by verbatim-decode unit tests, and the read paths
are tested against a mocked ConsDB server via responses, with the emitted
SQL pinned by request matchers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mfisherlevine
force-pushed
the
tickets/DM-55094
branch
from
September 4, 2026 17:04
4976db2 to
6b77d39
Compare
mypy-coverage report✅ All 825 definitions are fully annotated.
Summary
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #193 +/- ##
==========================================
+ Coverage 23.38% 24.12% +0.74%
==========================================
Files 38 39 +1
Lines 7077 7150 +73
==========================================
+ Hits 1655 1725 +70
- Misses 5422 5425 +3
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rapid analysis records, for every image it dispatches, the versions of the science packages that determine the AOS results, plus an overall incrementing version number mapped in a small per-instrument on-disk registry. The data model for that — the PackageVersions dataclass, its JSON serialisation, and the registry read/write — moves here from rubintv_production so that users without access to that package can consume the data.
ConsDB is about to grow a JSONB column for this. writePackageVersionsToConsDb and readPackageVersionsFromConsDb round-trip the pinned blob shape ({"versions": {...}, "version_number": N}) through it, defaulting to cdb_.exposure_quicklook.package_versions until the schema change lands (table and column are parameterised in case it lands elsewhere). loadPackageVersionRegistry and writeBackdatedPackageVersions support back-filling ConsDB for images processed before the column existed, expanding a known overall version number to its full package-version set via the registry.
The blob shape and the deployed registry file format are pinned by verbatim-decode unit tests, and the ConsDB read/write paths are tested against a mocked server via responses, including the write -> read round-trip.