Separate IndexedDB database from object store, and keep the value structured - #317
Merged
Conversation
…uctured
The database and object store names were joined into a single field as
f"{database_name}.{obj_store_name}", which cannot reliably be split back apart:
real profiles hold database names containing dots, such as Bitwarden's
storage.bw.offline. They are now separate fields, and the key's IndexedDB type
and raw bytes are recorded alongside them, since str() of a number and of a
numeric string are identical and a binary key had nowhere to go.
The value was stringified at parse time, so the structure a page actually stored
was gone before anything could read it. The record now holds the deserialized
object and renders the string form on demand. Keeping both roughly doubles
IndexedDB memory, measured on one test profile at 21 MB of strings against 28 MB
of objects and 49 MB for the pair, so only the object is kept; nothing reads a
rendered value more than once per run.
Rendering is a per-browser override, so Chrome keeps its repr and Firefox its
truncated JSON, and neither output moved. A record with no value now renders
blank instead of the string None. ccl returns None both for an entry carrying no
value bytes, which is what a deletion tombstone looks like, and for a serialized
JS null; the literal None read as recovered data and could not be told apart
from a value that really was the string "None".
Adds an Object Store column to the Storage sheet, and object_store, key_type and
key_raw to the sqlite storage table and to JSONL records. key_raw is hex because
the JSONL encoder renders bytes as UTF-8 with replacement characters, which
would corrupt a binary key beyond recovery.
RyanDFIR
changed the base branch from
hash-cache-and-filesystem-bodies
to
main
September 5, 2026 23:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #316, because the Storage sheet column numbers here are computed on top of the File SHA256 column that PR adds. Review that one first.
The database and object store names were joined into one field as f"{database_name}.{obj_store_name}", which cannot reliably be split back apart. Not hypothetical: bf4sa_2025_bob-1 has 19 rows whose database name is Bitwarden's storage.bw.offline. They are separate fields now, and the key's IndexedDB type and raw bytes are recorded too, since str() of a number and of a numeric string look identical and a binary key had nowhere to go.
The value was stringified at parse time, so the structure a page stored was gone before anything could read it. The record now holds the deserialized object and renders the string on demand. Keeping both roughly doubles IndexedDB memory, measured on one profile at 21 MB of strings against 28 MB of objects and 49 MB for the pair, so only the object is kept. Nothing reads a rendered value more than once per run.
A record with no value used to render the string None. ccl returns None both for an entry carrying no value bytes, which is what a deletion tombstone looks like, and for a serialized JS null; the literal None read as recovered data and could not be told apart from a value that really was the string "None". Those cells are blank now.
Rendering is a per-browser override, so Chrome keeps its repr and Firefox its truncated JSON, and no existing output moved. New output is an Object Store column on the Storage sheet, plus object_store, key_type and key_raw in the sqlite storage table and JSONL records. key_raw is hex because the JSONL encoder renders bytes as UTF-8 with replacement, which would corrupt a binary key.
Checked against bf4sa_2025_bob-1 and its 35,235 IndexedDB rows: no row still carries a glued database name, every row carries a deserialized value, and no rendering differs from the old str() form. All six corpus baselines are unchanged. The blank-value fix was verified against cellebrite.ctf_2021, where all 238 deleted rows now render blank and no row contains the literal string None.