Skip to content

docs(dedup): clarify survivor completeness scope #708

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wesm merged 1 commit into from
Sep 2, 2026
+11 −11
Merged

docs(dedup): clarify survivor completeness scope #708

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/diagrams/survivor-selection-concept.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -394,7 +394,7 @@ <h2>Stage 2 — Priority list</h2>
<div class="tier-num">03</div>
<div>
<div class="tier-name">More attachments</div>
<div class="tier-hint">when normalized MIME matches, prefer the more complete set</div>
<div class="tier-hint">only when every eligible copy has matching normalized MIME</div>
</div>
<div class="tier-side"><span class="tier-tag">payload</span></div>
</div>
Expand All @@ -403,7 +403,7 @@ <h2>Stage 2 — Priority list</h2>
<div class="tier-num">04</div>
<div>
<div class="tier-name">Attachment-presence signal</div>
<div class="tier-hint">when counts tie, preserve a source-declared attachment</div>
<div class="tier-hint">within that equivalent set, preserve a source-declared attachment</div>
</div>
<div class="tier-side"><span class="tier-tag">payload</span></div>
</div>
Expand All @@ -412,7 +412,7 @@ <h2>Stage 2 — Priority list</h2>
<div class="tier-num">05</div>
<div>
<div class="tier-name">Larger payload</div>
<div class="tier-hint">when normalized MIME matches, prefer more stored bytes</div>
<div class="tier-hint">within that equivalent set, prefer more stored bytes</div>
</div>
<div class="tier-side"><span class="tier-tag">payload</span></div>
</div>
Expand Down
12 changes: 6 additions & 6 deletions docs/internal/accounts-identities-collections-dedup/spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -313,9 +313,9 @@ preference runs in this order:

1. Source preference (when `--prefer` is configured, or the default
order: `gmail,imap,mbox,emlx,hey`).
2. Complete original payload — has raw MIME, then, when normalized
raw MIME matches, more attachments, an attachment-presence signal,
and a larger original payload.
2. Complete original payload — has raw MIME, then, only when every
eligible copy has the same normalized raw MIME hash, more attachments,
an attachment-presence signal, and a larger original payload.
3. Source metadata quality — provider IDs, threading info, presence
of Message-ID.
4. Richer label or folder metadata.
Expand All @@ -324,9 +324,9 @@ preference runs in this order:

Earlier rules win outright. Later rules apply only when all earlier
ones tie. Attachment and payload-size metadata are authoritative only
when both rows have raw MIME and their normalized MIME hashes match;
a shared Message-ID alone is insufficient. The exact policy is visible
in dry-run output.
when every eligible copy has raw MIME and all normalized MIME hashes
match; a shared Message-ID alone is insufficient. The exact policy is
visible in dry-run output.

The public [Deduplication](../../usage/deduplication.md) guide carries
the rendered survivor-selection diagram.
Expand Down
4 changes: 2 additions & 2 deletions docs/usage/deduplication.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,10 @@ Survivor selection is deterministic and explainable, and the reasoning is printe
7. Earlier archive timestamp.
8. A stable row ID, as the final tie-breaker.

Earlier rules win outright; later rules apply only when all earlier ones tie. The attachment-count, attachment-presence, and payload-size rules apply only when both copies have raw MIME and their normalized MIME hashes match. A shared `Message-ID` alone cannot make those payload-completeness signals authoritative. The survivor inherits the union of labels from the copies it replaces, and backfills raw MIME from a non-survivor if it was missing the original payload.
Earlier rules win outright; later rules apply only when all earlier ones tie. The attachment-count, attachment-presence, and payload-size rules apply only when every eligible copy has raw MIME and all their normalized MIME hashes match. A shared `Message-ID` alone cannot make those payload-completeness signals authoritative. The survivor inherits the union of labels from the copies it replaces, and backfills raw MIME from a non-survivor if it was missing the original payload.

<figure data-lightbox style="margin: 1.5rem 0; text-align: center;">
<img src="/assets/generated/concepts/survivor-selection-concept.png" alt="Survivor selection runs the sent-copy eligibility filter first, then a priority list: source preference, raw MIME, more attachments, an attachment-presence signal, a larger payload, richer labels, earlier archive time, and finally a stable row ID." loading="lazy" style="width: 100%; display: block;" />
<img src="/assets/generated/concepts/survivor-selection-concept.png" alt="Survivor selection filters to eligible sent copies first, then considers source preference and raw MIME. Only when every eligible copy has matching normalized MIME does it compare attachments, attachment presence, and payload size before labels, archive time, and stable row ID." loading="lazy" style="width: 100%; display: block;" />
</figure>

## Choosing a Scope
Expand Down