fix(media): skip byte-identical writes so a synced vault stops forking - #134
Open
VGonPa wants to merge 1 commit into
Open
fix(media): skip byte-identical writes so a synced vault stops forking#134VGonPa wants to merge 1 commit into
VGonPa wants to merge 1 commit into
Conversation
`generate` re-mirrors the entire media tree into the vault on every run and the nightly job runs it nightly, so the same bytes were rewritten thousands of times a night. On iCloud Drive a rewrite is a modification to reconcile, and a clash with a not-yet-uploaded version is resolved by keeping BOTH — the loser as a `name N.ext` sibling. The x-knowledge vault had accumulated 8.41 GB across 50,020 such copies, every one byte-identical to the file next to it. Two write paths, two distinct signatures: - `generate._mirror_file` — `shutil.copy2` truncates in place, so the inode survives and `copystat` restores mtime; only ctime records the write. This is the path that filled the vault. - `media._write_bytes` — `os.replace` installs a fresh inode on every call, including no-op rewrites during a full refresh (which rebuilds item state and so re-downloads every photo, defeating the item-level idempotency). Both now compare before writing, size first so the common case settles from metadata without faulting in cloud-only bytes. A stat/read failure falls through to the write: a redundant copy is cheap, a missing embed is not. Verified against the live vault — 300 real files re-mirrored, 0 touched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Qué pasaba
learnings/x-knowledge/_media/en el vault pesaba 9,1 GB. De eso, 8,41 GB eran copias de conflicto de iCloud — 50.020 ficherosnombre N.ext, todos byte-idénticos al fichero de al lado, ninguno referenciado por ninguna nota. El contenido real son 0,55 GB.No era vídeo:
digest-videodescarta los bytes como debe. Era la misma imagen guardada hasta 17 veces.Por qué
generatere-mirrorea el árbol de media entero al vault en cada ejecución, y el job nocturno lanzageneratecada noche. Miles de reescrituras de bytes idénticos por noche. Para iCloud cada reescritura es una modificación que reconciliar, y si choca con una versión que aún no ha subido, conserva las dos: la perdedora comonombre N.ext.Los duplicados se agrupan en 9 fechas — las de los refresh grandes (25-jun, 4/5/6-jul, 12-ago), no las noches sueltas. Un pase masivo satura el sync y bifurca en bloque.
El fix
Dos rutas de escritura, con firmas observables distintas:
generate._mirror_fileshutil.copy2(trunca en sitio)copystat)media._write_bytesos.replaceLa primera es la que llenó el vault. Ambas comparan ahora antes de escribir, con el tamaño como corto-circuito para no materializar bytes que sólo viven en la nube. Un fallo de
stat/lectura cae a escribir: una copia redundante es barata, un embed roto no.download_allya era idempotente a nivel de ítem, pero un full refresh reconstruye el estado y re-descarga todas las fotos — por eso la idempotencia de arriba no bastaba.Tests
TDD, con el RED verificado en cada caso. El primer intento contra
_mirror_filepasó sin el fix: asertaba sobre inode y mtime, quecopy2no altera. Corregido ast_ctime_ns, que es el único campo que registra la escritura — y el que un cliente de sync interpreta como "esto ha cambiado"._mirror_file: no toca un destino idéntico · reescribe uno que divergió · reescribe cuando el tamaño coincide pero los bytes no_write_bytes: no toca un fichero idéntico · reescribe si el contenido cambió · no se confunde con un.parthuérfano al ladoSuite completa: 1633 passed, 1 xfailed.
scripts/check.sh: las 11 puertas en verde, coverage 94%.Verificación contra el vault real
300 ficheros reales del vault re-mirrorados con el código nuevo → 0 tocados. iCloud no vería nada que reconciliar.
Los 8,41 GB ya se han recuperado en local (el disco iba al 94%), verificando antes fichero a fichero: 50.020/50.020 confirmados byte-idénticos por md5 contra su base, 0 grupos sin fichero base, 0 referencias de notas rotas después.
🤖 Generated with Claude Code