diff --git a/CONTEXT.md b/CONTEXT.md index affd3259..b621b386 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -9,62 +9,66 @@ Mobile and desktop Cyd offer the same Bluesky account-management capabilities an _Avoid_: UI parity, identical clients **Cyd archive**: -A complete, self-contained, versioned snapshot of an account's saved data and media that supported Cyd clients can exchange without losing its meaning. Version 2 is the first supported Cyd archive format; the earlier unversioned mobile prototype is not a Cyd archive compatibility surface. -_Avoid_: HTML export, database backup +An export of one social-platform account's saved data. Each platform defines its own archive contents, behavior, compatibility guarantees, and version history, so references to a particular archive must name the platform. +_Avoid_: archive, platform-neutral archive -**Interchange database**: -The canonical representation of structured account data inside a Cyd archive. It is independent of each client's private runtime storage. -_Avoid_: runtime database, mobile database +**Cyd Bluesky archive**: +A Cyd archive governed by the Bluesky archive contract. Version 2 is the first supported Bluesky archive format; the earlier unversioned mobile prototype is not a compatibility surface. +_Avoid_: Cyd archive v2, platform-neutral v2 archive + +**Bluesky interchange database**: +The canonical representation of structured Bluesky account data inside a Cyd Bluesky archive. It is independent of each client's private runtime storage. +_Avoid_: interchange database, runtime database, mobile database **Bluesky identity**: A Bluesky account identified durably by its DID, even when its handle changes or separate Cyd installations know it by different local identifiers. _Avoid_: handle, Cyd UUID -**Local account**: -A client's local representation of a Bluesky identity, identified within Cyd by a UUID and containing that client's settings and saved data. -_Avoid_: Bluesky identity +**Bluesky local account**: +A client's local representation of a Bluesky identity, identified within Cyd by a UUID and containing that client's Bluesky account settings and Bluesky saved data. +_Avoid_: local account, Bluesky identity -**Archive import**: -An idempotent recovery merge of a Cyd archive into a matching local account, preserving the union of saved data while collapsing records that share stable Bluesky identifiers. It may restore data previously removed through local deletion. -_Avoid_: replace, synchronize +**Bluesky archive import**: +An idempotent recovery merge of a Cyd Bluesky archive into a matching Bluesky local account, preserving the union of Bluesky saved data while collapsing records that share stable Bluesky identifiers. It may restore data previously removed through Bluesky local deletion. +_Avoid_: archive import, replace, synchronize -**Account settings**: -Local preferences governing how Cyd saves and manages a Bluesky identity. An archive can supply defaults for a new local account, but does not silently override an existing local account's preferences or schedules. -_Avoid_: account data, archive state +**Bluesky account settings**: +Local preferences governing how Cyd saves and manages a Bluesky identity. A Cyd Bluesky archive can supply defaults for a new Bluesky local account, but does not silently override an existing Bluesky local account's preferences or schedules. +_Avoid_: account settings, account data, archive state -**Scheduled reminder**: -A prompt to review and start due account-management work. It does not authorize Cyd to perform deletion unattended; clients may use platform-appropriate delivery such as local notifications or server-scheduled push. -_Avoid_: scheduled job, automatic deletion +**Bluesky scheduled reminder**: +A prompt to review and start due Bluesky account-management work. It does not authorize Cyd to perform deletion unattended; clients may use platform-appropriate delivery such as local notifications or server-scheduled push. +_Avoid_: scheduled reminder, scheduled job, automatic deletion -**Connection**: -A local installation's authorization to act on a Bluesky identity. Connections are established separately on each client and are never part of a Cyd archive. -Disconnecting removes authorization without removing the local account or its saved data. -_Avoid_: account data, imported session, local account +**Bluesky connection**: +A local installation's authorization to act on a Bluesky identity. Bluesky connections are established separately on each client and are never part of a Cyd Bluesky archive. +Disconnecting removes authorization without removing the Bluesky local account or its Bluesky saved data. +_Avoid_: connection, account data, imported session, Bluesky local account -**Saved data**: -The account records and complete media Cyd has preserved locally, including material that may later disappear from Bluesky. When a record is selected for saving, its full media is part of the saved data regardless of the record category. -_Avoid_: live feed, HTML export +**Bluesky saved data**: +The Bluesky records and complete media Cyd has preserved locally, including material that may later disappear from Bluesky. When a record is selected for saving, its full media is part of the Bluesky saved data regardless of the record category. +_Avoid_: saved data, live feed, HTML export -**Saved record**: +**Bluesky saved record**: The latest representation of a Bluesky record observed by Cyd at a stable AT URI, together with its observation timestamps and deletion state. It is not a history of every CID revision. -_Avoid_: record revision, live record +_Avoid_: saved record, record revision, live record -**Browse**: -Inspect saved data inside Cyd without requiring a Bluesky connection or network access. -_Avoid_: view on Bluesky, live feed +**Bluesky browse**: +Inspection of Bluesky saved data inside Cyd without requiring a Bluesky connection or network access. +_Avoid_: browse, view on Bluesky, live feed -**Context snapshot**: -The directly referenced author, reply parent, quoted record, external embed, and other captured information required to render a saved record faithfully without recursively preserving the surrounding social graph. -_Avoid_: full thread, live lookup +**Bluesky context snapshot**: +The directly referenced author, reply parent, quoted record, external embed, and other captured information required to render a Bluesky saved record faithfully without recursively preserving the surrounding social graph. +_Avoid_: context snapshot, full thread, live lookup -**Complete backup**: -A saved dataset or Cyd archive containing every expected asset for its selected records. Missing assets are explicit and make the backup incomplete without invalidating the data that was successfully preserved. -_Avoid_: valid archive, successful export +**Complete Bluesky backup**: +Bluesky saved data or a Cyd Bluesky archive containing every expected asset for its selected records. Missing assets are explicit and make the backup incomplete without invalidating the data that was successfully preserved. +_Avoid_: complete backup, valid archive, successful export -**Source deletion**: -Removal of a record or relationship from Bluesky while retaining Cyd's saved copy and deletion state. -_Avoid_: local deletion +**Bluesky source deletion**: +Removal of a record or relationship from Bluesky while retaining its Bluesky saved record and deletion state in Cyd. +_Avoid_: source deletion, Bluesky local deletion -**Local deletion**: -An explicit removal of saved data from Cyd, independent of whether the source still exists on Bluesky. -_Avoid_: source deletion, stop saving +**Bluesky local deletion**: +An explicit removal of Bluesky saved data from Cyd, independent of whether the source still exists on Bluesky. +_Avoid_: local deletion, Bluesky source deletion, stop saving diff --git a/docs/adr/0001-bidirectional-cyd-archive-compatibility.md b/docs/adr/0001-bidirectional-cyd-archive-compatibility.md index ca1e7bd0..e19b4d96 100644 --- a/docs/adr/0001-bidirectional-cyd-archive-compatibility.md +++ b/docs/adr/0001-bidirectional-cyd-archive-compatibility.md @@ -1,3 +1,3 @@ -# Bidirectional Cyd archive compatibility begins with version 2 +# Bidirectional Cyd Bluesky archive compatibility begins with version 2 -Cyd mobile and desktop exchange Cyd archives beginning with the canonical version 2 format. The unversioned mobile prototype has no users and is unsupported by both clients; abandoning it avoids coupling the shared contract to Mobile's historical runtime schema. Each client clearly rejects archive versions it does not support so evolving one application does not silently break portability with the other. +Cyd mobile and desktop exchange Cyd Bluesky archives beginning with the canonical Bluesky version 2 format. Bluesky archive versions are independent of the archive contracts and version histories for Cyd's other social platforms. The unversioned mobile prototype has no users and is unsupported by both clients; abandoning it avoids coupling the shared contract to Mobile's historical runtime schema. Each client clearly rejects Bluesky archive versions it does not support so evolving one application does not silently break portability with the other. diff --git a/docs/adr/0002-canonical-archive-interchange-database.md b/docs/adr/0002-canonical-archive-interchange-database.md index f038760b..1802b8a5 100644 --- a/docs/adr/0002-canonical-archive-interchange-database.md +++ b/docs/adr/0002-canonical-archive-interchange-database.md @@ -1,3 +1,3 @@ -# Canonical archive interchange database +# Canonical Bluesky interchange database -The `data.db` inside version 2 and later Cyd archives is a canonical interchange database rather than a copy of either client's runtime database. Mobile and desktop translate between private storage and this versioned schema, allowing their implementations and migrations to evolve independently; neither client maintains an adapter for the unused unversioned mobile prototype. +The `data.db` inside Bluesky version 2 and later Cyd Bluesky archives is a canonical Bluesky interchange database rather than a copy of either client's runtime database. Mobile and desktop translate between private storage and this versioned schema, allowing their implementations and migrations to evolve independently; neither client maintains an adapter for the unused unversioned mobile prototype. diff --git a/docs/adr/0004-merge-archive-imports-idempotently.md b/docs/adr/0004-merge-archive-imports-idempotently.md index fabe17b9..dd39cb5a 100644 --- a/docs/adr/0004-merge-archive-imports-idempotently.md +++ b/docs/adr/0004-merge-archive-imports-idempotently.md @@ -1,3 +1,3 @@ # Merge archive imports idempotently -Importing a Cyd archive merges it with the matching local account rather than replacing local data. Records match on stable Bluesky identifiers such as AT URIs, richer or newer representations can update older ones, data absent from the archive is retained, and importing the same archive repeatedly has no further effect; this supports safe cross-device transfer without sacrificing data collected independently on either device. Because import is an explicit recovery action, it may restore locally deleted records, and its preview reports that restoration rather than relying on permanent deletion tombstones that would suppress older backups. +A Bluesky archive import merges a Cyd Bluesky archive with the matching Bluesky local account rather than replacing Bluesky saved data. Bluesky saved records match on stable identifiers such as AT URIs, richer or newer representations can update older ones, Bluesky saved data absent from the Cyd Bluesky archive is retained, and importing the same archive repeatedly has no further effect; this supports safe cross-device transfer without sacrificing data collected independently on either device. Because Bluesky archive import is an explicit recovery action, it may restore records removed through Bluesky local deletion, and its preview reports that restoration rather than relying on permanent deletion tombstones that would suppress older backups. diff --git a/docs/adr/0005-exclude-credentials-from-archives.md b/docs/adr/0005-exclude-credentials-from-archives.md index 9085237d..d0a5a0bc 100644 --- a/docs/adr/0005-exclude-credentials-from-archives.md +++ b/docs/adr/0005-exclude-credentials-from-archives.md @@ -1,3 +1,3 @@ # Exclude credentials from Cyd archives -Cyd archives never contain OAuth tokens, private keys, session state, or other reusable credentials. Imported accounts are immediately available for offline browsing, but each installation must establish its own Bluesky connection before it can refresh data or perform account actions, preventing a portable archive from becoming an account-control credential. +Cyd Bluesky archives never contain OAuth tokens, private keys, session state, or other reusable credentials. Imported accounts are immediately available for offline browsing, but each installation must establish its own Bluesky connection before it can refresh data or perform account actions, preventing a portable archive from becoming an account-control credential. diff --git a/docs/adr/0008-own-the-archive-contract-in-cyd.md b/docs/adr/0008-own-the-archive-contract-in-cyd.md index 1ceda80c..75985b34 100644 --- a/docs/adr/0008-own-the-archive-contract-in-cyd.md +++ b/docs/adr/0008-own-the-archive-contract-in-cyd.md @@ -1,3 +1,3 @@ # Own the shared archive contract in cyd -The normative Cyd archive specification, format-version history, and canonical compatibility fixtures live in the `cyd` repository. Mobile and desktop may maintain implementation-specific types, but both must validate against this shared contract so compatibility is not inferred from either application's current runtime schema. +The normative Cyd Bluesky archive specification, Bluesky format-version history, and canonical compatibility fixtures live in the `cyd` repository. Other social platforms have independent archive contracts and version histories. Mobile and desktop may maintain implementation-specific types, but both must validate against this shared Bluesky contract so compatibility is not inferred from either application's current runtime schema. diff --git a/docs/adr/0010-make-each-archive-self-contained.md b/docs/adr/0010-make-each-archive-self-contained.md index 4551b450..d8873e8e 100644 --- a/docs/adr/0010-make-each-archive-self-contained.md +++ b/docs/adr/0010-make-each-archive-self-contained.md @@ -1,3 +1,3 @@ -# Make each Cyd archive self-contained +# Make each Cyd Bluesky archive self-contained -Every Cyd archive is a complete snapshot of the local account's saved data and media at export time rather than an incremental delta that depends on earlier archives. Individual archives may be larger, but any one of them remains sufficient for recovery or cross-device transfer and can be merged idempotently on import. +Every Cyd Bluesky archive is a complete snapshot of a Bluesky local account's Bluesky saved data at export time rather than an incremental delta that depends on earlier Cyd Bluesky archives. Individual archives may be larger, but any one of them remains sufficient for recovery or cross-device transfer through Bluesky archive import. diff --git a/docs/adr/0011-ship-v2-archives-without-encryption.md b/docs/adr/0011-ship-v2-archives-without-encryption.md index 66a85ecc..e96f325e 100644 --- a/docs/adr/0011-ship-v2-archives-without-encryption.md +++ b/docs/adr/0011-ship-v2-archives-without-encryption.md @@ -1,3 +1,3 @@ -# Ship version 2 archives without encryption +# Ship Bluesky version 2 archives without encryption -Version 2 Cyd archives are plaintext ZIP packages while the initial cross-platform contract is established. Passphrase encryption is deferred to a later format version; clients must not imply that version 2 archives protect sensitive saved data such as chats, and exports should communicate that limitation clearly. +Cyd Bluesky archives at version 2 are plaintext ZIP packages while the initial cross-client Bluesky contract is established. Passphrase encryption is deferred to a later Bluesky format version; clients must not imply that Bluesky version 2 archives protect sensitive Bluesky saved data such as chats, and exports should communicate that limitation clearly. diff --git a/docs/adr/0012-validate-v2-archive-integrity-before-import.md b/docs/adr/0012-validate-v2-archive-integrity-before-import.md index fab66651..e70cf2a1 100644 --- a/docs/adr/0012-validate-v2-archive-integrity-before-import.md +++ b/docs/adr/0012-validate-v2-archive-integrity-before-import.md @@ -1,3 +1,3 @@ -# Validate version 2 archive integrity before import +# Validate Bluesky version 2 archive integrity before import -Version 2 archives include a manifest listing every payload file's path, byte size, and SHA-256 digest, and export captures a point-in-time-consistent database and media snapshot. Import validates the complete manifest before mutating local data and applies the merge transactionally, preventing corrupt or partial packages from producing partially imported accounts. +Bluesky version 2 archives include a manifest listing every payload file's path, byte size, and SHA-256 digest, and export captures a point-in-time-consistent database and media snapshot. Bluesky archive import validates the complete manifest before mutating Bluesky saved data and applies the merge transactionally, preventing corrupt or partial packages from producing partially imported Bluesky local accounts. diff --git a/docs/adr/0019-store-the-latest-observed-record.md b/docs/adr/0019-store-the-latest-observed-record.md index 375431b3..80893904 100644 --- a/docs/adr/0019-store-the-latest-observed-record.md +++ b/docs/adr/0019-store-the-latest-observed-record.md @@ -1,3 +1,3 @@ # Store the latest observed record -Version 2 represents the latest observation of each Bluesky record at its stable AT URI, including observation timestamps and deletion state, rather than retaining every historical CID revision. Import selects the newer observation and does not replace populated fields with absent data; complete revision history is outside the initial parity scope. +Bluesky version 2 represents the latest observation of each Bluesky record at its stable AT URI, including observation timestamps and deletion state, rather than retaining every historical CID revision. Import selects the newer observation and does not replace populated fields with absent data; complete revision history is outside the initial parity scope. diff --git a/docs/adr/0020-do-not-use-the-archive-filename-for-validation.md b/docs/adr/0020-do-not-use-the-archive-filename-for-validation.md index d035a808..1ddf26e0 100644 --- a/docs/adr/0020-do-not-use-the-archive-filename-for-validation.md +++ b/docs/adr/0020-do-not-use-the-archive-filename-for-validation.md @@ -1,3 +1,3 @@ # Do not use the archive filename for validation -Version 2 archive filenames are friendly suggestions and may be changed by users without affecting import. Clients determine identity, format support, and integrity from validated internal metadata, schema, and manifest content rather than a handle-bearing filename; the unused unversioned mobile prototype receives no filename-based compatibility exception. +Bluesky version 2 archive filenames are friendly suggestions and may be changed by users without affecting import. Clients determine platform, identity, format support, and integrity from validated internal metadata, schema, and manifest content rather than a handle-bearing filename; the unused unversioned mobile prototype receives no filename-based compatibility exception. diff --git a/docs/adr/0024-deduplicate-media-within-each-account.md b/docs/adr/0024-deduplicate-media-within-each-account.md index 3a0e8ab7..8f7c2ebc 100644 --- a/docs/adr/0024-deduplicate-media-within-each-account.md +++ b/docs/adr/0024-deduplicate-media-within-each-account.md @@ -1,3 +1,3 @@ -# Deduplicate media within each local account +# Deduplicate media within each Bluesky local account -Desktop stores media content-addressably within each UUID-keyed local account, and saved records reference those assets rather than owning duplicate files. Version 2 archives likewise package each unique asset once and map record references through the interchange database; cross-account deduplication is deferred so deleting or exporting one account does not depend on another account's storage. +Desktop stores media content-addressably within each UUID-keyed Bluesky local account, and Bluesky saved records reference those assets rather than owning duplicate files. Bluesky version 2 archives likewise package each unique asset once and map record references through the Bluesky interchange database; cross-account deduplication is deferred so deleting or exporting one Bluesky local account does not depend on another Bluesky local account's storage. diff --git a/docs/adr/0026-export-portable-account-data-not-operational-state.md b/docs/adr/0026-export-portable-account-data-not-operational-state.md index 440686fa..fe962f95 100644 --- a/docs/adr/0026-export-portable-account-data-not-operational-state.md +++ b/docs/adr/0026-export-portable-account-data-not-operational-state.md @@ -1,3 +1,3 @@ -# Export portable account data, not operational state +# Export portable Bluesky saved data, not operational state -Version 2 archives include account identity and captured profile, saved records and media, source-deletion state, asset-completeness state, and portable save/delete rule defaults. They exclude credentials, schedules, pending or historical jobs, logs, analytics, error reports, temporary files, caches, and UI state because those are client-local operations whose transfer could duplicate work or trigger unintended behavior. +Bluesky version 2 archives include the Bluesky identity and captured profile, Bluesky saved records and media, Bluesky source-deletion state, asset-completeness state, and portable Bluesky account settings. They exclude credentials, schedules, pending or historical jobs, logs, analytics, error reports, temporary files, caches, and UI state because those are client-local operations whose transfer could duplicate work or trigger unintended behavior. diff --git a/docs/adr/0028-roll-out-archive-readers-before-writers.md b/docs/adr/0028-roll-out-archive-readers-before-writers.md index 12bb2120..33bb7421 100644 --- a/docs/adr/0028-roll-out-archive-readers-before-writers.md +++ b/docs/adr/0028-roll-out-archive-readers-before-writers.md @@ -1,3 +1,3 @@ # Roll out archive readers before writers -Archive format changes ship readers before writers: Mobile and Desktop first gain and verify version 2 import, and version 2 export is enabled only after compatible readers are released. Version 2 is the first supported archive format; neither client imports or exports the unused unversioned mobile prototype. +Bluesky archive format changes ship readers before writers: Mobile and Desktop first gain and verify Bluesky version 2 import, and Bluesky version 2 export is enabled only after compatible readers are released. Version 2 is the first supported Bluesky archive format; neither client imports or exports the unused unversioned mobile prototype. diff --git a/docs/archive/bluesky/v2/README.md b/docs/archive/bluesky/v2/README.md new file mode 100644 index 00000000..342e21b2 --- /dev/null +++ b/docs/archive/bluesky/v2/README.md @@ -0,0 +1,179 @@ +# Cyd Bluesky archive format version 2 + +This directory is the normative contract for version 2 of the cross-client Cyd +Bluesky archive. It applies only to Bluesky data; other social platforms have +independent archive contracts and version histories. “Must”, “must not”, +“required”, “should”, and “may” are used in their RFC 2119 sense. + +## Package + +A Bluesky v2 archive is a plaintext, self-contained ZIP file. Its filename and +file extension are not authoritative. Readers identify its archive family, +platform, and platform-specific version from `metadata.json`. All ZIP entry +names are UTF-8, `/`-separated, normalized relative paths. An archive has +exactly this layout: + +```text +metadata.json +manifest.json +data.db +media/sha256//<64-lowercase-hex-digest> +``` + +Directories may be explicit or implicit. Every other entry is forbidden. +Entries must be regular files or directories: symlinks, hard links, devices, +absolute paths, drive-qualified paths, `.`/`..` segments, backslashes, +duplicate normalized names, and encrypted ZIP entries are forbidden. + +`metadata.json` is UTF-8 JSON with this exact shape (unknown fields are +reserved and must be ignored): + +```json +{ + "format": "cyd-archive", + "platform": "bluesky", + "version": 2, + "createdAt": "2026-01-15T12:00:00.000Z", + "accountDid": "did:plc:examplealice", + "accountUuid": "018d5f7a-9b3c-7d10-8a2e-1f4c6b8d0e12", + "completeness": "complete" +} +``` + +`format`, `platform`, and `version` form the format discriminator. Version 2 +belongs to the Bluesky archive version namespace; it does not define version 2 +for X, Facebook, Mastodon, or any other platform. Readers must reject a +different platform rather than interpreting it with this schema. + +Times use RFC 3339 UTC with milliseconds. `completeness` is `complete` only +when every expected asset is available; otherwise it is `incomplete`. The +metadata format, platform, version, identity, UUID, creation time, and +completeness must equal the single row in `data.db.archive`. + +## Integrity manifest + +`manifest.json` is UTF-8 JSON: + +```json +{ + "algorithm": "sha256", + "payloads": [ + { "path": "data.db", "bytes": 1234, "sha256": "…64 lowercase hex…" } + ] +} +``` + +It lists every regular payload except itself exactly once, sorted by path. +Each entry contains its normalized path, uncompressed byte length, and SHA-256 +of its bytes. Readers must validate the complete manifest before exposing a +Bluesky archive import preview or mutating Bluesky saved data. `metadata.json` +and `data.db` are payloads. +Media paths must end in their digest and their bytes must match that digest. + +Readers extract into an isolated staging directory while validating entry +type, path, declared and expanded sizes, entry counts, free space, and digest. +They must not follow links or write outside staging. There is no small fixed +archive-size limit; clients may require explicit confirmation at documented +resource thresholds. Failure or cancellation leaves the live Bluesky local +account unchanged. + +## Bluesky interchange database + +`data.db` is SQLite 3 and must implement [schema.sql](schema.sql) exactly for +Bluesky v2. It is a Bluesky interchange model, never a copy of a client's +runtime database. Text is UTF-8. Booleans are integers constrained to `0` or +`1`. JSON columns contain canonical JSON values rather than client-specific +serialized objects. + +The model has these semantic groups: + +- `archive`, `identity`, and `profiles`: Bluesky local account UUID, durable + DID, current profile, and captured historical author profiles. +- `records`, `selections`, `record_subjects`, and `record_context`: the latest + observed form of posts/reposts/likes/bookmarks, the selected relationship's + target record, plus direct reply-parent, quote, external, and author context. + Context is bounded; it does not recursively capture threads. +- `conversations`, `conversation_members`, and `messages`: direct-message + context, membership, and messages. +- `relationships`: follows, blocks, and mutes with Bluesky source-deletion state. +- `assets` and `record_assets`: every expected image, preview, thumbnail, and + full video, including unavailable assets and their reason. +- `portable_settings`: portable Bluesky account setting defaults. + +Stable AT URIs identify records; DIDs identify Bluesky identities. `cid` is +the latest observed CID, not revision history. `observed_at`, +`source_deleted_at`, and captured profiles preserve observation and deletion +state. An unavailable asset has no digest or archive path, remains referenced, +and makes the archive incomplete. Every available asset has a unique digest, +byte count, media type, and content-addressed payload. + +All five selection categories—`posts`, `reposts`, `likes`, `bookmarks`, and +`chats`—must be representable. Category selection pulls its required direct +context and media with it. Bluesky archive import merges records by stable +identifiers, retains Bluesky saved data absent from the Cyd Bluesky archive, +prefers newer non-empty observations, and is idempotent. It may restore data +removed through Bluesky local deletion. + +## Completeness and consistency + +Export is a point-in-time-consistent snapshot of the selected Bluesky saved +data. Each Cyd Bluesky archive stands alone; Bluesky v2 has no delta or +predecessor mechanism. +A structurally valid archive may be incomplete. Missing expected media must be +represented as `unavailable` or `missing`, never silently omitted. The archive +and metadata completeness is `complete` if and only if all asset rows are +`available` and every referenced payload validates. + +## Privacy boundary + +Only portable Bluesky saved data belongs in Bluesky v2. Cyd Bluesky archives +must not contain OAuth or other credentials, private keys, sessions, schedules, +pending/running/history jobs, logs, analytics, diagnostics, error reports, +caches, temporary data, filesystem paths, or UI state. Portable Bluesky account +settings are defaults, not authority to schedule or execute work, and do not +overwrite an existing Bluesky local account unless the user explicitly chooses +to adopt them. + +## Version behavior + +Desktop imports and exports Bluesky v2 only. It rejects the historical +unversioned Bluesky mobile format explicitly as `unsupported legacy format`; +it never guesses a version from a filename. It rejects another platform as +`unsupported archive platform` and a valid internal Bluesky version greater +than 2 as `unsupported newer archive version`. It rejects invalid Bluesky v2 +content as an invalid or corrupt archive, which is distinct from platform and +version rejection. Desktop Bluesky writers must emit only Bluesky v2. Readers +must ship and pass the canonical semantic fixtures before Bluesky v2 writing +is release-enabled. + +## Canonical fixtures + +`fixtures/complete.cyd` and `fixtures/incomplete.cyd` are canonical semantic +examples generated by `scripts/generate.py`. They cover every category, +captured/current profiles, reply/quote/external context, relationships, +observations, deletion state, chats, settings, images, previews, video +thumbnails, and a playable full video payload. The PNG payloads have complete +chunk framing and valid CRCs; the MP4 is an FFmpeg-generated test pattern with +movie metadata and sample tables. The incomplete fixture differs by one +expected unavailable full-video asset. + +`fixtures/semantic-expectations.json` defines the normalized identity, profiles, +records, selections, record subjects, direct context, conversations, messages, +relationships, observation and deletion state, assets and their digests, +asset roles, portable Bluesky account settings, completeness, and version +behavior. A consumer must compare all of those semantics, not only row counts +or category names. + +Consumers compare normalized database meaning, asset digests, completeness, +and rejection outcomes—not ZIP bytes, entry order, or SQLite page layout. + +## Bluesky version history + +| Version | Status | Meaning | +| ------------------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------- | +| Unversioned (“v1”) | Unsupported | Historical Mobile-only Bluesky archive format; never a cross-client contract. | +| 2 | Current | First Cyd Bluesky archive contract supported for Desktop/Mobile interchange. Plaintext ZIP and canonical SQLite Bluesky interchange database. | + +Future Bluesky versions must append an entry here and define explicit reader +behavior. Version meaning must never be inferred from a filename or applied to +another social platform. diff --git a/docs/archive/bluesky/v2/assets/README.md b/docs/archive/bluesky/v2/assets/README.md new file mode 100644 index 00000000..507c3e11 --- /dev/null +++ b/docs/archive/bluesky/v2/assets/README.md @@ -0,0 +1,12 @@ +# Canonical media sources + +`full-video.mp4` is a 10-second, 1280×720 FFmpeg test pattern generated without +third-party source material: + +```sh +ffmpeg -f lavfi -i testsrc=duration=10:size=1280x720:rate=30 \ + -pix_fmt yuv420p full-video.mp4 +``` + +The fixture generator creates the three one-pixel PNG sources itself with +distinct RGBA values, complete PNG chunk framing, and valid CRCs. diff --git a/docs/archive/bluesky/v2/assets/full-video.mp4 b/docs/archive/bluesky/v2/assets/full-video.mp4 new file mode 100644 index 00000000..eb6ab69a Binary files /dev/null and b/docs/archive/bluesky/v2/assets/full-video.mp4 differ diff --git a/docs/archive/bluesky/v2/fixtures/complete.cyd b/docs/archive/bluesky/v2/fixtures/complete.cyd new file mode 100644 index 00000000..9a8316f9 Binary files /dev/null and b/docs/archive/bluesky/v2/fixtures/complete.cyd differ diff --git a/docs/archive/bluesky/v2/fixtures/incomplete.cyd b/docs/archive/bluesky/v2/fixtures/incomplete.cyd new file mode 100644 index 00000000..90bd17e6 Binary files /dev/null and b/docs/archive/bluesky/v2/fixtures/incomplete.cyd differ diff --git a/docs/archive/bluesky/v2/fixtures/semantic-expectations.json b/docs/archive/bluesky/v2/fixtures/semantic-expectations.json new file mode 100644 index 00000000..e817cf77 --- /dev/null +++ b/docs/archive/bluesky/v2/fixtures/semantic-expectations.json @@ -0,0 +1,445 @@ +{ + "archiveFormat": { + "format": "cyd-archive", + "platform": "bluesky", + "version": 2 + }, + "desktopBlueskyVersionBehavior": { + "unversionedV1": "reject_unsupported_legacy_format", + "blueskyV2Import": "accept", + "blueskyV2Export": "write_only_bluesky_v2", + "otherPlatform": "reject_unsupported_archive_platform", + "newerVersion": "reject_unsupported_newer_version" + }, + "commonSemantics": { + "archive": { + "createdAt": "2026-01-15T12:00:00.000Z", + "accountDid": "did:plc:canonicalalice", + "accountUuid": "018d5f7a-9b3c-7d10-8a2e-1f4c6b8d0e12" + }, + "identity": { + "did": "did:plc:canonicalalice", + "currentProfileId": "profile-alice-current" + }, + "profiles": [ + { + "id": "profile-alice-captured", + "did": "did:plc:canonicalalice", + "handle": "alice.test", + "displayName": "Alice Captured", + "description": "captured author", + "avatarAssetId": null, + "bannerAssetId": null, + "capturedAt": "2025-12-01T09:00:00.000Z" + }, + { + "id": "profile-alice-current", + "did": "did:plc:canonicalalice", + "handle": "alice.example", + "displayName": "Alice Current", + "description": "current profile", + "avatarAssetId": "asset-image", + "bannerAssetId": null, + "capturedAt": "2026-01-15T12:00:00.000Z" + }, + { + "id": "profile-bob", + "did": "did:plc:canonicalbob", + "handle": "bob.example", + "displayName": "Bob", + "description": "direct context author", + "avatarAssetId": null, + "bannerAssetId": null, + "capturedAt": "2026-01-15T12:00:00.000Z" + } + ], + "records": [ + { + "uri": "at://did:plc:canonicalalice/app.bsky.feed.like/like", + "cid": "bafy-like", + "recordType": "app.bsky.feed.like", + "authorProfileId": "profile-alice-captured", + "indexedAt": null, + "createdAt": "2026-01-03T10:00:00.000Z", + "firstObservedAt": "2026-01-03T10:00:00.000Z", + "observedAt": "2026-01-15T12:00:00.000Z", + "sourceDeletedAt": null, + "text": null, + "facets": [], + "payload": {} + }, + { + "uri": "at://did:plc:canonicalalice/app.bsky.feed.post/post", + "cid": "bafy-post", + "recordType": "app.bsky.feed.post", + "authorProfileId": "profile-alice-captured", + "indexedAt": null, + "createdAt": "2026-01-01T10:00:00.000Z", + "firstObservedAt": "2026-01-01T10:00:00.000Z", + "observedAt": "2026-01-15T12:00:00.000Z", + "sourceDeletedAt": null, + "text": "Own post with full video", + "facets": [], + "payload": {} + }, + { + "uri": "at://did:plc:canonicalalice/app.bsky.feed.repost/repost", + "cid": "bafy-repost", + "recordType": "app.bsky.feed.repost", + "authorProfileId": "profile-alice-captured", + "indexedAt": null, + "createdAt": "2026-01-02T10:00:00.000Z", + "firstObservedAt": "2026-01-02T10:00:00.000Z", + "observedAt": "2026-01-15T12:00:00.000Z", + "sourceDeletedAt": null, + "text": null, + "facets": [], + "payload": {} + }, + { + "uri": "at://did:plc:canonicalalice/app.bsky.graph.listitem/bookmark", + "cid": "bafy-bookmark", + "recordType": "app.cyd.bookmark", + "authorProfileId": "profile-alice-captured", + "indexedAt": null, + "createdAt": "2026-01-04T10:00:00.000Z", + "firstObservedAt": "2026-01-04T10:00:00.000Z", + "observedAt": "2026-01-15T12:00:00.000Z", + "sourceDeletedAt": null, + "text": null, + "facets": [], + "payload": {} + }, + { + "uri": "at://did:plc:canonicalbob/app.bsky.feed.post/quote", + "cid": "bafy-quote", + "recordType": "app.bsky.feed.post", + "authorProfileId": "profile-bob", + "indexedAt": null, + "createdAt": "2025-12-29T10:00:00.000Z", + "firstObservedAt": "2025-12-29T10:00:00.000Z", + "observedAt": "2026-01-15T12:00:00.000Z", + "sourceDeletedAt": "2026-01-10T00:00:00.000Z", + "text": "Quoted record later deleted at source", + "facets": [], + "payload": {} + }, + { + "uri": "at://did:plc:canonicalbob/app.bsky.feed.post/reply", + "cid": "bafy-reply", + "recordType": "app.bsky.feed.post", + "authorProfileId": "profile-bob", + "indexedAt": null, + "createdAt": "2025-12-30T10:00:00.000Z", + "firstObservedAt": "2025-12-30T10:00:00.000Z", + "observedAt": "2026-01-15T12:00:00.000Z", + "sourceDeletedAt": null, + "text": "Direct reply parent", + "facets": [], + "payload": {} + } + ], + "selections": [ + { + "category": "bookmarks", + "subjectId": "at://did:plc:canonicalalice/app.bsky.graph.listitem/bookmark", + "selectedAt": "2026-01-15T12:00:00.000Z" + }, + { + "category": "chats", + "subjectId": "convo-canonical", + "selectedAt": "2026-01-15T12:00:00.000Z" + }, + { + "category": "likes", + "subjectId": "at://did:plc:canonicalalice/app.bsky.feed.like/like", + "selectedAt": "2026-01-15T12:00:00.000Z" + }, + { + "category": "posts", + "subjectId": "at://did:plc:canonicalalice/app.bsky.feed.post/post", + "selectedAt": "2026-01-15T12:00:00.000Z" + }, + { + "category": "reposts", + "subjectId": "at://did:plc:canonicalalice/app.bsky.feed.repost/repost", + "selectedAt": "2026-01-15T12:00:00.000Z" + } + ], + "recordSubjects": [ + { + "relationshipUri": "at://did:plc:canonicalalice/app.bsky.feed.like/like", + "subjectRecordUri": "at://did:plc:canonicalalice/app.bsky.feed.post/post" + }, + { + "relationshipUri": "at://did:plc:canonicalalice/app.bsky.feed.repost/repost", + "subjectRecordUri": "at://did:plc:canonicalbob/app.bsky.feed.post/quote" + }, + { + "relationshipUri": "at://did:plc:canonicalalice/app.bsky.graph.listitem/bookmark", + "subjectRecordUri": "at://did:plc:canonicalalice/app.bsky.feed.post/post" + } + ], + "recordContext": [ + { + "recordUri": "at://did:plc:canonicalalice/app.bsky.feed.post/post", + "kind": "external", + "contextRecordUri": null, + "contextProfileId": null, + "external": { + "title": "Canonical link", + "uri": "https://example.com/context" + } + }, + { + "recordUri": "at://did:plc:canonicalalice/app.bsky.feed.post/post", + "kind": "quote", + "contextRecordUri": "at://did:plc:canonicalbob/app.bsky.feed.post/quote", + "contextProfileId": "profile-bob", + "external": null + }, + { + "recordUri": "at://did:plc:canonicalalice/app.bsky.feed.post/post", + "kind": "reply_parent", + "contextRecordUri": "at://did:plc:canonicalbob/app.bsky.feed.post/reply", + "contextProfileId": "profile-bob", + "external": null + } + ], + "conversations": [ + { + "id": "convo-canonical", + "rev": "1", + "firstObservedAt": "2026-01-15T12:00:00.000Z", + "observedAt": "2026-01-15T12:00:00.000Z", + "sourceDeletedAt": null + } + ], + "conversationMembers": [ + { + "conversationId": "convo-canonical", + "profileId": "profile-alice-captured" + }, + { + "conversationId": "convo-canonical", + "profileId": "profile-bob" + } + ], + "messages": [ + { + "id": "message-canonical", + "conversationId": "convo-canonical", + "senderProfileId": "profile-bob", + "sentAt": "2026-01-05T10:00:00.000Z", + "observedAt": "2026-01-15T12:00:00.000Z", + "sourceDeletedAt": null, + "text": "Portable direct message", + "facets": [], + "payload": {} + } + ], + "relationships": [ + { + "uri": "at://did:plc:canonicalalice/app.bsky.graph.follow/bob", + "kind": "follow", + "actorDid": "did:plc:canonicalalice", + "subjectDid": "did:plc:canonicalbob", + "createdAt": "2026-01-15T12:00:00.000Z", + "observedAt": "2026-01-15T12:00:00.000Z", + "sourceDeletedAt": null + }, + { + "uri": "mute:did:plc:canonicalbob", + "kind": "mute", + "actorDid": "did:plc:canonicalalice", + "subjectDid": "did:plc:canonicalbob", + "createdAt": "2026-01-15T12:00:00.000Z", + "observedAt": "2026-01-15T12:00:00.000Z", + "sourceDeletedAt": "2026-01-12T00:00:00.000Z" + } + ], + "recordAssets": [ + { + "ownerType": "profile", + "ownerId": "profile-alice-current", + "assetId": "asset-image", + "role": "avatar", + "position": 0 + }, + { + "ownerType": "record", + "ownerId": "at://did:plc:canonicalalice/app.bsky.feed.post/post", + "assetId": "asset-image", + "role": "content", + "position": 0 + }, + { + "ownerType": "record", + "ownerId": "at://did:plc:canonicalalice/app.bsky.feed.post/post", + "assetId": "asset-video-full", + "role": "content", + "position": 1 + }, + { + "ownerType": "record", + "ownerId": "at://did:plc:canonicalalice/app.bsky.feed.post/post", + "assetId": "asset-preview", + "role": "preview", + "position": 0 + }, + { + "ownerType": "record", + "ownerId": "at://did:plc:canonicalalice/app.bsky.feed.post/post", + "assetId": "asset-thumbnail", + "role": "thumbnail", + "position": 0 + } + ], + "portableSettings": [ + { "key": "delete_follows", "value": false }, + { "key": "delete_posts", "value": { "olderThanDays": 30 } }, + { "key": "save_bookmarks", "value": true }, + { "key": "save_chats", "value": true }, + { "key": "save_likes", "value": true }, + { "key": "save_posts", "value": true }, + { "key": "save_reposts", "value": true } + ] + }, + "fixtures": { + "complete.cyd": { + "completeness": "complete", + "assets": [ + { + "id": "asset-image", + "kind": "image", + "mediaType": "image/png", + "byteCount": 70, + "sha256": "6c55d7bbccd73bb135e8e7c7161b4be3cef97142313f2fba304e73b3329ecc3d", + "archivePath": "media/sha256/6c/6c55d7bbccd73bb135e8e7c7161b4be3cef97142313f2fba304e73b3329ecc3d", + "availability": "available", + "unavailableReason": null, + "sourceUrl": "https://cdn.example/asset-image", + "width": 1, + "height": 1, + "altText": "canonical image" + }, + { + "id": "asset-preview", + "kind": "preview", + "mediaType": "image/png", + "byteCount": 70, + "sha256": "74280ef143147745086279ee96a1a35f8821c5f03785b46476537c24e3990847", + "archivePath": "media/sha256/74/74280ef143147745086279ee96a1a35f8821c5f03785b46476537c24e3990847", + "availability": "available", + "unavailableReason": null, + "sourceUrl": "https://cdn.example/asset-preview", + "width": 1, + "height": 1, + "altText": "canonical preview" + }, + { + "id": "asset-thumbnail", + "kind": "thumbnail", + "mediaType": "image/png", + "byteCount": 70, + "sha256": "c7fe85dd5a8b22c8c3295535bb3665617f778c9d81058f3b8adc1eae69db99ec", + "archivePath": "media/sha256/c7/c7fe85dd5a8b22c8c3295535bb3665617f778c9d81058f3b8adc1eae69db99ec", + "availability": "available", + "unavailableReason": null, + "sourceUrl": "https://cdn.example/asset-thumbnail", + "width": 1, + "height": 1, + "altText": "canonical thumbnail" + }, + { + "id": "asset-video-full", + "kind": "video", + "mediaType": "video/mp4", + "byteCount": 149576, + "sha256": "f7a5be429b0d633d22038d97f5930c95e8403643fbb5b4e7274ee5ec16018cde", + "archivePath": "media/sha256/f7/f7a5be429b0d633d22038d97f5930c95e8403643fbb5b4e7274ee5ec16018cde", + "availability": "available", + "unavailableReason": null, + "sourceUrl": "https://cdn.example/asset-video-full", + "width": 1280, + "height": 720, + "altText": "canonical video" + } + ] + }, + "incomplete.cyd": { + "completeness": "incomplete", + "assets": [ + { + "id": "asset-image", + "kind": "image", + "mediaType": "image/png", + "byteCount": 70, + "sha256": "6c55d7bbccd73bb135e8e7c7161b4be3cef97142313f2fba304e73b3329ecc3d", + "archivePath": "media/sha256/6c/6c55d7bbccd73bb135e8e7c7161b4be3cef97142313f2fba304e73b3329ecc3d", + "availability": "available", + "unavailableReason": null, + "sourceUrl": "https://cdn.example/asset-image", + "width": 1, + "height": 1, + "altText": "canonical image" + }, + { + "id": "asset-preview", + "kind": "preview", + "mediaType": "image/png", + "byteCount": 70, + "sha256": "74280ef143147745086279ee96a1a35f8821c5f03785b46476537c24e3990847", + "archivePath": "media/sha256/74/74280ef143147745086279ee96a1a35f8821c5f03785b46476537c24e3990847", + "availability": "available", + "unavailableReason": null, + "sourceUrl": "https://cdn.example/asset-preview", + "width": 1, + "height": 1, + "altText": "canonical preview" + }, + { + "id": "asset-thumbnail", + "kind": "thumbnail", + "mediaType": "image/png", + "byteCount": 70, + "sha256": "c7fe85dd5a8b22c8c3295535bb3665617f778c9d81058f3b8adc1eae69db99ec", + "archivePath": "media/sha256/c7/c7fe85dd5a8b22c8c3295535bb3665617f778c9d81058f3b8adc1eae69db99ec", + "availability": "available", + "unavailableReason": null, + "sourceUrl": "https://cdn.example/asset-thumbnail", + "width": 1, + "height": 1, + "altText": "canonical thumbnail" + }, + { + "id": "asset-video-full", + "kind": "video", + "mediaType": "video/mp4", + "byteCount": null, + "sha256": null, + "archivePath": null, + "availability": "unavailable", + "unavailableReason": "source video was unavailable at export", + "sourceUrl": "https://cdn.example/asset-video-full", + "width": null, + "height": null, + "altText": null + } + ] + } + }, + "excluded": [ + "credentials", + "oauth", + "schedules", + "jobs", + "logs", + "analytics", + "diagnostics", + "errorReports", + "caches", + "temporaryData", + "uiState" + ] +} diff --git a/docs/archive/bluesky/v2/schema.sql b/docs/archive/bluesky/v2/schema.sql new file mode 100644 index 00000000..97dcd947 --- /dev/null +++ b/docs/archive/bluesky/v2/schema.sql @@ -0,0 +1,144 @@ +PRAGMA application_id = 0x43594232; +PRAGMA user_version = 2; +PRAGMA foreign_keys = ON; + +CREATE TABLE archive ( + format TEXT NOT NULL CHECK (format = 'cyd-archive'), + platform TEXT NOT NULL CHECK (platform = 'bluesky'), + version INTEGER NOT NULL CHECK (version = 2), + created_at TEXT NOT NULL, + account_did TEXT NOT NULL, + account_uuid TEXT NOT NULL, + completeness TEXT NOT NULL CHECK (completeness IN ('complete', 'incomplete')) +); + +CREATE TABLE identity ( + did TEXT PRIMARY KEY, + current_profile_id TEXT NOT NULL +); + +CREATE TABLE profiles ( + id TEXT PRIMARY KEY, + did TEXT NOT NULL, + handle TEXT, + display_name TEXT, + description TEXT, + avatar_asset_id TEXT REFERENCES assets(id), + banner_asset_id TEXT REFERENCES assets(id), + captured_at TEXT NOT NULL +); + +CREATE TABLE records ( + uri TEXT PRIMARY KEY, + cid TEXT, + record_type TEXT NOT NULL, + author_profile_id TEXT NOT NULL REFERENCES profiles(id), + indexed_at TEXT, + created_at TEXT NOT NULL, + first_observed_at TEXT NOT NULL, + observed_at TEXT NOT NULL, + source_deleted_at TEXT, + text TEXT, + facets_json TEXT, + payload_json TEXT NOT NULL +); + +CREATE TABLE selections ( + category TEXT NOT NULL CHECK (category IN ('posts', 'reposts', 'likes', 'bookmarks', 'chats')), + subject_id TEXT NOT NULL, + selected_at TEXT NOT NULL, + PRIMARY KEY (category, subject_id) +); + +CREATE TABLE record_subjects ( + relationship_uri TEXT PRIMARY KEY REFERENCES records(uri), + subject_record_uri TEXT NOT NULL REFERENCES records(uri) +); + +CREATE TABLE record_context ( + record_uri TEXT NOT NULL REFERENCES records(uri), + kind TEXT NOT NULL CHECK (kind IN ('reply_parent', 'quote', 'external')), + context_record_uri TEXT, + context_profile_id TEXT REFERENCES profiles(id), + external_json TEXT, + PRIMARY KEY (record_uri, kind) +); + +CREATE TABLE conversations ( + id TEXT PRIMARY KEY, + rev TEXT, + first_observed_at TEXT NOT NULL, + observed_at TEXT NOT NULL, + source_deleted_at TEXT +); + +CREATE TABLE conversation_members ( + conversation_id TEXT NOT NULL REFERENCES conversations(id), + profile_id TEXT NOT NULL REFERENCES profiles(id), + PRIMARY KEY (conversation_id, profile_id) +); + +CREATE TABLE messages ( + id TEXT PRIMARY KEY, + conversation_id TEXT NOT NULL REFERENCES conversations(id), + sender_profile_id TEXT NOT NULL REFERENCES profiles(id), + sent_at TEXT NOT NULL, + observed_at TEXT NOT NULL, + source_deleted_at TEXT, + text TEXT, + facets_json TEXT, + payload_json TEXT NOT NULL +); + +CREATE TABLE relationships ( + uri TEXT PRIMARY KEY, + kind TEXT NOT NULL CHECK (kind IN ('follow', 'block', 'mute')), + actor_did TEXT NOT NULL, + subject_did TEXT NOT NULL, + created_at TEXT, + observed_at TEXT NOT NULL, + source_deleted_at TEXT +); + +CREATE TABLE assets ( + id TEXT PRIMARY KEY, + kind TEXT NOT NULL CHECK (kind IN ('image', 'preview', 'thumbnail', 'video')), + media_type TEXT NOT NULL, + byte_count INTEGER CHECK (byte_count IS NULL OR byte_count >= 0), + sha256 TEXT UNIQUE CHECK (sha256 IS NULL OR (length(sha256) = 64 AND sha256 = lower(sha256))), + archive_path TEXT UNIQUE, + availability TEXT NOT NULL CHECK (availability IN ('available', 'missing', 'unavailable')), + unavailable_reason TEXT, + source_url TEXT, + width INTEGER, + height INTEGER, + alt_text TEXT, + CHECK ( + (availability = 'available' AND byte_count IS NOT NULL AND sha256 IS NOT NULL AND archive_path IS NOT NULL AND unavailable_reason IS NULL) + OR + (availability != 'available' AND byte_count IS NULL AND sha256 IS NULL AND archive_path IS NULL AND unavailable_reason IS NOT NULL) + ) +); + +CREATE TABLE record_assets ( + owner_type TEXT NOT NULL CHECK (owner_type IN ('record', 'message', 'profile')), + owner_id TEXT NOT NULL, + asset_id TEXT NOT NULL REFERENCES assets(id), + role TEXT NOT NULL CHECK (role IN ('content', 'preview', 'thumbnail', 'avatar', 'banner')), + position INTEGER NOT NULL CHECK (position >= 0), + PRIMARY KEY (owner_type, owner_id, role, position) +); + +CREATE TABLE portable_settings ( + key TEXT PRIMARY KEY CHECK (key IN ( + 'save_posts', 'save_reposts', 'save_likes', 'save_bookmarks', 'save_chats', + 'delete_posts', 'delete_reposts', 'delete_likes', 'delete_bookmarks', + 'delete_chats', 'delete_follows' + )), + value_json TEXT NOT NULL +); + +CREATE INDEX records_author ON records(author_profile_id); +CREATE INDEX records_observed ON records(observed_at); +CREATE INDEX messages_conversation ON messages(conversation_id, sent_at); +CREATE INDEX assets_availability ON assets(availability); diff --git a/docs/archive/bluesky/v2/scripts/generate.py b/docs/archive/bluesky/v2/scripts/generate.py new file mode 100755 index 00000000..1d39ce01 --- /dev/null +++ b/docs/archive/bluesky/v2/scripts/generate.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python3 +"""Generate the semantic Bluesky v2 fixtures. ZIP/SQLite bytes are not the contract.""" + +from __future__ import annotations + +import hashlib +import json +import sqlite3 +import struct +import tempfile +import zipfile +import zlib +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +SCHEMA = (ROOT / "schema.sql").read_text(encoding="utf-8") +FIXTURES = ROOT / "fixtures" +CREATED = "2026-01-15T12:00:00.000Z" +DID = "did:plc:canonicalalice" +UUID = "018d5f7a-9b3c-7d10-8a2e-1f4c6b8d0e12" + + +def png_chunk(kind: bytes, payload: bytes) -> bytes: + checksum = zlib.crc32(kind + payload) & 0xFFFFFFFF + return struct.pack(">I", len(payload)) + kind + payload + struct.pack(">I", checksum) + + +def one_pixel_png(red: int, green: int, blue: int) -> bytes: + signature = b"\x89PNG\r\n\x1a\n" + header = struct.pack(">IIBBBBB", 1, 1, 8, 6, 0, 0, 0) + scanline = bytes((0, red, green, blue, 255)) + return ( + signature + + png_chunk(b"IHDR", header) + + png_chunk(b"IDAT", zlib.compress(scanline)) + + png_chunk(b"IEND", b"") + ) + + +IMAGE = one_pixel_png(35, 116, 171) +PREVIEW = one_pixel_png(80, 160, 80) +THUMBNAIL = one_pixel_png(200, 120, 40) +FULL_VIDEO = (ROOT / "assets" / "full-video.mp4").read_bytes() + + +def add_asset( + db: sqlite3.Connection, + staging: Path, + asset_id: str, + kind: str, + media_type: str, + content: bytes | None, + reason: str | None = None, +) -> None: + if content is None: + db.execute( + "INSERT INTO assets (id, kind, media_type, availability, unavailable_reason, source_url) VALUES (?, ?, ?, 'unavailable', ?, ?)", + (asset_id, kind, media_type, reason, f"https://cdn.example/{asset_id}"), + ) + return + digest = hashlib.sha256(content).hexdigest() + archive_path = f"media/sha256/{digest[:2]}/{digest}" + output = staging / archive_path + output.parent.mkdir(parents=True, exist_ok=True) + output.write_bytes(content) + db.execute( + "INSERT INTO assets (id, kind, media_type, byte_count, sha256, archive_path, availability, source_url, width, height, alt_text) VALUES (?, ?, ?, ?, ?, ?, 'available', ?, ?, ?, ?)", + ( + asset_id, + kind, + media_type, + len(content), + digest, + archive_path, + f"https://cdn.example/{asset_id}", + 1 if kind != "video" else 1280, + 1 if kind != "video" else 720, + f"canonical {kind}", + ), + ) + + +def populate(db: sqlite3.Connection, staging: Path, complete: bool) -> None: + completeness = "complete" if complete else "incomplete" + db.execute( + "INSERT INTO archive VALUES ('cyd-archive', 'bluesky', 2, ?, ?, ?, ?)", + (CREATED, DID, UUID, completeness), + ) + profiles = [ + ("profile-alice-current", DID, "alice.example", "Alice Current", "current profile", CREATED), + ("profile-alice-captured", DID, "alice.test", "Alice Captured", "captured author", "2025-12-01T09:00:00.000Z"), + ("profile-bob", "did:plc:canonicalbob", "bob.example", "Bob", "direct context author", CREATED), + ] + db.executemany( + "INSERT INTO profiles (id, did, handle, display_name, description, captured_at) VALUES (?, ?, ?, ?, ?, ?)", + profiles, + ) + db.execute("INSERT INTO identity VALUES (?, ?)", (DID, "profile-alice-current")) + + add_asset(db, staging, "asset-image", "image", "image/png", IMAGE) + add_asset(db, staging, "asset-preview", "preview", "image/png", PREVIEW) + add_asset(db, staging, "asset-thumbnail", "thumbnail", "image/png", THUMBNAIL) + add_asset( + db, + staging, + "asset-video-full", + "video", + "video/mp4", + FULL_VIDEO if complete else None, + "source video was unavailable at export", + ) + db.execute( + "UPDATE profiles SET avatar_asset_id = 'asset-image' WHERE id = 'profile-alice-current'" + ) + + record_rows = [ + ("at://did:plc:canonicalalice/app.bsky.feed.post/post", "bafy-post", "app.bsky.feed.post", "profile-alice-captured", "2026-01-01T10:00:00.000Z", None, "Own post with full video"), + ("at://did:plc:canonicalbob/app.bsky.feed.post/reply", "bafy-reply", "app.bsky.feed.post", "profile-bob", "2025-12-30T10:00:00.000Z", None, "Direct reply parent"), + ("at://did:plc:canonicalbob/app.bsky.feed.post/quote", "bafy-quote", "app.bsky.feed.post", "profile-bob", "2025-12-29T10:00:00.000Z", "2026-01-10T00:00:00.000Z", "Quoted record later deleted at source"), + ("at://did:plc:canonicalalice/app.bsky.feed.repost/repost", "bafy-repost", "app.bsky.feed.repost", "profile-alice-captured", "2026-01-02T10:00:00.000Z", None, None), + ("at://did:plc:canonicalalice/app.bsky.feed.like/like", "bafy-like", "app.bsky.feed.like", "profile-alice-captured", "2026-01-03T10:00:00.000Z", None, None), + ("at://did:plc:canonicalalice/app.bsky.graph.listitem/bookmark", "bafy-bookmark", "app.cyd.bookmark", "profile-alice-captured", "2026-01-04T10:00:00.000Z", None, None), + ] + db.executemany( + "INSERT INTO records (uri, cid, record_type, author_profile_id, created_at, first_observed_at, observed_at, source_deleted_at, text, facets_json, payload_json) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, '[]', '{}')", + [(uri, cid, record_type, profile, created, created, CREATED, deleted, text) for uri, cid, record_type, profile, created, deleted, text in record_rows], + ) + post, reply, quote, repost, like, bookmark = [row[0] for row in record_rows] + selections = [ + ("posts", post), + ("reposts", repost), + ("likes", like), + ("bookmarks", bookmark), + ("chats", "convo-canonical"), + ] + db.executemany("INSERT INTO selections VALUES (?, ?, ?)", [(category, subject, CREATED) for category, subject in selections]) + db.executemany( + "INSERT INTO record_subjects VALUES (?, ?)", + [(repost, quote), (like, post), (bookmark, post)], + ) + db.executemany( + "INSERT INTO record_context VALUES (?, ?, ?, ?, ?)", + [ + (post, "reply_parent", reply, "profile-bob", None), + (post, "quote", quote, "profile-bob", None), + (post, "external", None, None, '{"title":"Canonical link","uri":"https://example.com/context"}'), + ], + ) + db.execute("INSERT INTO conversations VALUES ('convo-canonical', '1', ?, ?, NULL)", (CREATED, CREATED)) + db.executemany( + "INSERT INTO conversation_members VALUES ('convo-canonical', ?)", + [("profile-alice-captured",), ("profile-bob",)], + ) + db.execute( + "INSERT INTO messages VALUES ('message-canonical', 'convo-canonical', 'profile-bob', ?, ?, NULL, 'Portable direct message', '[]', '{}')", + ("2026-01-05T10:00:00.000Z", CREATED), + ) + db.executemany( + "INSERT INTO relationships VALUES (?, ?, ?, ?, ?, ?, ?)", + [ + ("at://did:plc:canonicalalice/app.bsky.graph.follow/bob", "follow", DID, "did:plc:canonicalbob", CREATED, CREATED, None), + ("mute:did:plc:canonicalbob", "mute", DID, "did:plc:canonicalbob", CREATED, CREATED, "2026-01-12T00:00:00.000Z"), + ], + ) + db.executemany( + "INSERT INTO record_assets VALUES (?, ?, ?, ?, ?)", + [ + ("profile", "profile-alice-current", "asset-image", "avatar", 0), + ("record", post, "asset-image", "content", 0), + ("record", post, "asset-preview", "preview", 0), + ("record", post, "asset-thumbnail", "thumbnail", 0), + ("record", post, "asset-video-full", "content", 1), + ], + ) + db.executemany( + "INSERT INTO portable_settings VALUES (?, ?)", + [ + ("save_posts", "true"), + ("save_reposts", "true"), + ("save_likes", "true"), + ("save_bookmarks", "true"), + ("save_chats", "true"), + ("delete_posts", '{"olderThanDays":30}'), + ("delete_follows", "false"), + ], + ) + + +def generate(name: str, complete: bool) -> None: + with tempfile.TemporaryDirectory(prefix=f"cyd-bluesky-v2-{name}-") as temporary: + staging = Path(temporary) + metadata = { + "format": "cyd-archive", + "platform": "bluesky", + "version": 2, + "createdAt": CREATED, + "accountDid": DID, + "accountUuid": UUID, + "completeness": "complete" if complete else "incomplete", + } + (staging / "metadata.json").write_text( + json.dumps(metadata, indent=2, ensure_ascii=False) + "\n", encoding="utf-8" + ) + db = sqlite3.connect(staging / "data.db") + db.executescript(SCHEMA) + populate(db, staging, complete) + db.commit() + result = db.execute("PRAGMA integrity_check").fetchone()[0] + assert result == "ok", result + db.close() + + payload_paths = sorted( + file.relative_to(staging).as_posix() + for file in staging.rglob("*") + if file.is_file() + ) + payloads = [] + for relative in payload_paths: + content = (staging / relative).read_bytes() + payloads.append( + { + "path": relative, + "bytes": len(content), + "sha256": hashlib.sha256(content).hexdigest(), + } + ) + (staging / "manifest.json").write_text( + json.dumps({"algorithm": "sha256", "payloads": payloads}, indent=2) + "\n", + encoding="utf-8", + ) + + with zipfile.ZipFile(FIXTURES / f"{name}.cyd", "w", zipfile.ZIP_DEFLATED, compresslevel=9) as archive: + for relative in sorted(payload_paths + ["manifest.json"]): + info = zipfile.ZipInfo(relative, (2026, 1, 15, 12, 0, 0)) + info.compress_type = zipfile.ZIP_DEFLATED + info.external_attr = 0o100644 << 16 + archive.writestr(info, (staging / relative).read_bytes(), compresslevel=9) + + +if __name__ == "__main__": + FIXTURES.mkdir(parents=True, exist_ok=True) + generate("complete", complete=True) + generate("incomplete", complete=False) diff --git a/src/bluesky_archive_v2/__tests__/canonicalFixtures.test.ts b/src/bluesky_archive_v2/__tests__/canonicalFixtures.test.ts new file mode 100644 index 00000000..a4a6456b --- /dev/null +++ b/src/bluesky_archive_v2/__tests__/canonicalFixtures.test.ts @@ -0,0 +1,417 @@ +import fs from "fs"; +import os from "os"; +import path from "path"; + +import Database from "better-sqlite3"; +import { afterEach, describe, expect, test } from "vitest"; +import unzipper from "unzipper"; + +const contractRoot = path.resolve("docs/archive/bluesky/v2"); +const fixtureRoot = path.join(contractRoot, "fixtures"); +const temporaryDirectories: string[] = []; + +type JsonObject = Record; + +type SemanticExpectations = { + archiveFormat: JsonObject; + desktopBlueskyVersionBehavior: Record; + commonSemantics: JsonObject; + fixtures: Record< + "complete.cyd" | "incomplete.cyd", + { completeness: string; assets: JsonObject[] } + >; +}; + +function readExpectations(): SemanticExpectations { + return JSON.parse( + fs.readFileSync( + path.join(fixtureRoot, "semantic-expectations.json"), + "utf8", + ), + ) as SemanticExpectations; +} + +function rows(database: Database.Database, sql: string): JsonObject[] { + return database.prepare(sql).all() as JsonObject[]; +} + +function parseJsonFields(values: JsonObject[], fields: string[]): JsonObject[] { + return values.map((value) => ({ + ...value, + ...Object.fromEntries( + fields.map((field) => [ + field, + value[field] === null ? null : JSON.parse(value[field] as string), + ]), + ), + })); +} + +function normalizeSemantics(database: Database.Database): JsonObject { + const archive = database + .prepare( + `SELECT created_at AS createdAt, account_did AS accountDid, + account_uuid AS accountUuid FROM archive`, + ) + .get() as JsonObject; + const identity = database + .prepare("SELECT did, current_profile_id AS currentProfileId FROM identity") + .get() as JsonObject; + + return { + archive, + identity, + profiles: rows( + database, + `SELECT id, did, handle, display_name AS displayName, + description, avatar_asset_id AS avatarAssetId, + banner_asset_id AS bannerAssetId, captured_at AS capturedAt + FROM profiles ORDER BY id`, + ), + records: parseJsonFields( + rows( + database, + `SELECT uri, cid, record_type AS recordType, + author_profile_id AS authorProfileId, indexed_at AS indexedAt, + created_at AS createdAt, first_observed_at AS firstObservedAt, + observed_at AS observedAt, source_deleted_at AS sourceDeletedAt, + text, facets_json AS facets, payload_json AS payload + FROM records ORDER BY uri`, + ), + ["facets", "payload"], + ), + selections: rows( + database, + `SELECT category, subject_id AS subjectId, selected_at AS selectedAt + FROM selections ORDER BY category`, + ), + recordSubjects: rows( + database, + `SELECT relationship_uri AS relationshipUri, + subject_record_uri AS subjectRecordUri + FROM record_subjects ORDER BY relationship_uri`, + ), + recordContext: parseJsonFields( + rows( + database, + `SELECT record_uri AS recordUri, kind, + context_record_uri AS contextRecordUri, + context_profile_id AS contextProfileId, external_json AS external + FROM record_context ORDER BY record_uri, kind`, + ), + ["external"], + ), + conversations: rows( + database, + `SELECT id, rev, first_observed_at AS firstObservedAt, + observed_at AS observedAt, source_deleted_at AS sourceDeletedAt + FROM conversations ORDER BY id`, + ), + conversationMembers: rows( + database, + `SELECT conversation_id AS conversationId, profile_id AS profileId + FROM conversation_members ORDER BY conversation_id, profile_id`, + ), + messages: parseJsonFields( + rows( + database, + `SELECT id, conversation_id AS conversationId, + sender_profile_id AS senderProfileId, sent_at AS sentAt, + observed_at AS observedAt, source_deleted_at AS sourceDeletedAt, + text, facets_json AS facets, payload_json AS payload + FROM messages ORDER BY id`, + ), + ["facets", "payload"], + ), + relationships: rows( + database, + `SELECT uri, kind, actor_did AS actorDid, subject_did AS subjectDid, + created_at AS createdAt, observed_at AS observedAt, + source_deleted_at AS sourceDeletedAt + FROM relationships ORDER BY uri`, + ), + recordAssets: rows( + database, + `SELECT owner_type AS ownerType, owner_id AS ownerId, + asset_id AS assetId, role, position + FROM record_assets ORDER BY owner_type, owner_id, role, position`, + ), + portableSettings: parseJsonFields( + rows( + database, + `SELECT key, value_json AS value + FROM portable_settings ORDER BY key`, + ), + ["value"], + ), + }; +} + +function normalizeAssets(database: Database.Database): JsonObject[] { + return rows( + database, + `SELECT id, kind, media_type AS mediaType, byte_count AS byteCount, + sha256, archive_path AS archivePath, availability, + unavailable_reason AS unavailableReason, source_url AS sourceUrl, + width, height, alt_text AS altText + FROM assets ORDER BY id`, + ); +} + +function crc32(bytes: Buffer): number { + let checksum = 0xffffffff; + for (const byte of bytes) { + checksum ^= byte; + for (let bit = 0; bit < 8; bit += 1) { + checksum = (checksum >>> 1) ^ (0xedb88320 & -(checksum & 1)); + } + } + return (checksum ^ 0xffffffff) >>> 0; +} + +function expectValidPng(bytes: Buffer): void { + expect(bytes.subarray(0, 8)).toEqual( + Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]), + ); + const chunkTypes: string[] = []; + let offset = 8; + while (offset < bytes.length) { + const length = bytes.readUInt32BE(offset); + const typeStart = offset + 4; + const dataEnd = typeStart + 4 + length; + const chunkEnd = dataEnd + 4; + expect(chunkEnd).toBeLessThanOrEqual(bytes.length); + const typeAndData = bytes.subarray(typeStart, dataEnd); + chunkTypes.push(typeAndData.subarray(0, 4).toString("ascii")); + expect(bytes.readUInt32BE(dataEnd)).toBe(crc32(typeAndData)); + offset = chunkEnd; + } + expect(offset).toBe(bytes.length); + expect(chunkTypes[0]).toBe("IHDR"); + expect(chunkTypes).toContain("IDAT"); + expect(chunkTypes.at(-1)).toBe("IEND"); +} + +function expectPlayableMp4Structure(bytes: Buffer): void { + const boxes = new Map(); + let offset = 0; + while (offset < bytes.length) { + const size = bytes.readUInt32BE(offset); + const type = bytes.subarray(offset + 4, offset + 8).toString("ascii"); + expect(size).toBeGreaterThanOrEqual(8); + expect(offset + size).toBeLessThanOrEqual(bytes.length); + boxes.set(type, bytes.subarray(offset, offset + size)); + offset += size; + } + expect(offset).toBe(bytes.length); + expect([...boxes.keys()]).toEqual( + expect.arrayContaining(["ftyp", "mdat", "moov"]), + ); + const movie = boxes.get("moov")!; + expect(movie.includes(Buffer.from("mvhd"))).toBe(true); + expect(movie.includes(Buffer.from("trak"))).toBe(true); + expect(movie.includes(Buffer.from("stbl"))).toBe(true); +} + +async function openFixture(name: "complete" | "incomplete") { + const archivePath = path.join(fixtureRoot, `${name}.cyd`); + const directory = await unzipper.Open.file(archivePath); + const outputPath = fs.mkdtempSync( + path.join(os.tmpdir(), `cyd-bluesky-v2-${name}-`), + ); + temporaryDirectories.push(outputPath); + await directory.extract({ path: outputPath }); + const database = new Database(path.join(outputPath, "data.db"), { + readonly: true, + }); + return { database, directory, outputPath }; +} + +afterEach(() => { + for (const directory of temporaryDirectories.splice(0)) { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +describe("canonical Cyd Bluesky archive v2 fixtures", () => { + test.each(["complete", "incomplete"] as const)( + "%s fixture matches every normalized semantic expectation", + async (fixtureName) => { + const { database } = await openFixture(fixtureName); + const expectations = readExpectations(); + const fixtureExpectations = expectations.fixtures[`${fixtureName}.cyd`]; + + expect(normalizeSemantics(database)).toEqual( + expectations.commonSemantics, + ); + expect(normalizeAssets(database)).toEqual(fixtureExpectations.assets); + expect( + database.prepare("SELECT completeness FROM archive").pluck().get(), + ).toBe(fixtureExpectations.completeness); + database.close(); + }, + ); + + test("complete fixture contains valid PNGs and a structured playable MP4", async () => { + const { database, directory } = await openFixture("complete"); + const entries = new Map( + directory.files.map((entry) => [entry.path, entry]), + ); + const assets = rows( + database, + "SELECT kind, archive_path AS archivePath FROM assets ORDER BY kind", + ); + + for (const asset of assets) { + const bytes = await entries.get(asset.archivePath as string)!.buffer(); + if (asset.kind === "video") { + expectPlayableMp4Structure(bytes); + } else { + expectValidPng(bytes); + } + } + database.close(); + }); + + test.each(["complete", "incomplete"] as const)( + "%s fixture represents every portable category and direct context", + async (fixtureName) => { + const { database } = await openFixture(fixtureName); + + expect( + database + .prepare("SELECT category FROM selections ORDER BY category") + .pluck() + .all(), + ).toEqual(["bookmarks", "chats", "likes", "posts", "reposts"]); + expect( + database + .prepare("SELECT kind FROM record_context ORDER BY kind") + .pluck() + .all(), + ).toEqual(["external", "quote", "reply_parent"]); + expect( + database.prepare("SELECT kind FROM assets ORDER BY kind").pluck().all(), + ).toEqual(["image", "preview", "thumbnail", "video"]); + expect( + database.prepare("SELECT COUNT(*) FROM relationships").pluck().get(), + ).toBeGreaterThan(0); + expect( + database.prepare("SELECT COUNT(*) FROM messages").pluck().get(), + ).toBeGreaterThan(0); + expect( + database.prepare("SELECT COUNT(*) FROM record_subjects").pluck().get(), + ).toBe(3); + expect( + database + .prepare( + "SELECT COUNT(*) FROM profiles WHERE avatar_asset_id IS NOT NULL", + ) + .pluck() + .get(), + ).toBeGreaterThan(0); + database.close(); + }, + ); + + test.each([ + ["complete", "complete", 0], + ["incomplete", "incomplete", 1], + ] as const)( + "%s fixture describes asset completeness honestly", + async (fixtureName, expectedState, unavailableCount) => { + const { database } = await openFixture(fixtureName); + expect( + database.prepare("SELECT completeness FROM archive").pluck().get(), + ).toBe(expectedState); + expect( + database + .prepare( + "SELECT COUNT(*) FROM assets WHERE availability != 'available'", + ) + .pluck() + .get(), + ).toBe(unavailableCount); + database.close(); + }, + ); + + test.each(["complete", "incomplete"] as const)( + "%s fixture manifest covers every payload with correct size and digest", + async (fixtureName) => { + const { directory } = await openFixture(fixtureName); + const entries = new Map( + directory.files.map((entry) => [entry.path, entry]), + ); + const manifest = JSON.parse( + (await entries.get("manifest.json")!.buffer()).toString("utf8"), + ) as { + payloads: Array<{ path: string; bytes: number; sha256: string }>; + }; + const crypto = await import("crypto"); + + expect( + manifest.payloads.map(({ path: payloadPath }) => payloadPath), + ).toEqual( + [...entries.keys()].filter((entry) => entry !== "manifest.json").sort(), + ); + for (const payload of manifest.payloads) { + const bytes = await entries.get(payload.path)!.buffer(); + expect(payload.bytes).toBe(bytes.byteLength); + expect(crypto.createHash("sha256").update(bytes).digest("hex")).toBe( + payload.sha256, + ); + } + }, + ); + + test("metadata and database scope version 2 to Bluesky", async () => { + const { database, directory } = await openFixture("complete"); + const metadataEntry = directory.files.find( + (entry) => entry.path === "metadata.json", + ); + const metadata = JSON.parse( + (await metadataEntry!.buffer()).toString("utf8"), + ) as Record; + + expect(metadata).toEqual({ + format: "cyd-archive", + platform: "bluesky", + version: 2, + createdAt: "2026-01-15T12:00:00.000Z", + accountDid: "did:plc:canonicalalice", + accountUuid: "018d5f7a-9b3c-7d10-8a2e-1f4c6b8d0e12", + completeness: "complete", + }); + expect( + database + .prepare( + `SELECT format, platform, version, created_at AS createdAt, + account_did AS accountDid, account_uuid AS accountUuid, + completeness FROM archive`, + ) + .get(), + ).toEqual(metadata); + expect(database.pragma("application_id", { simple: true })).toBe( + 0x43594232, + ); + database.close(); + }); + + test("semantic expectations make Desktop Bluesky version support explicit", () => { + const expectations = readExpectations(); + + expect(expectations.archiveFormat).toEqual({ + format: "cyd-archive", + platform: "bluesky", + version: 2, + }); + expect(expectations.desktopBlueskyVersionBehavior).toEqual({ + unversionedV1: "reject_unsupported_legacy_format", + blueskyV2Import: "accept", + blueskyV2Export: "write_only_bluesky_v2", + otherPlatform: "reject_unsupported_archive_platform", + newerVersion: "reject_unsupported_newer_version", + }); + }); +});