Skip database_metadata cleanly and report the schema version it carries - #347
Merged
RyanDFIR merged 1 commit intoSep 7, 2026
Conversation
The Site Characteristics store holds a `database_metadata` record whose value is
the store's bare schema version, not a SiteDataProto. The skip for it was
already there, but the version check guarding it called .encode() on
`item['value']`, which is bytes and so has no .encode(). On any store not at the
expected version -- b'2' is in the wild -- that raised AttributeError before
reaching the `continue`, and the record surfaced through the outer handler as
Exception parsing SiteDataProto ({'key': b'database_metadata', 'value': b'2', ...})
AttributeError: 'bytes' object has no attribute 'encode'
It fires on healthy profiles, so it is pure noise, and noise that trains the
reader to skim past errors in the log. Nothing was actually lost: the record was
skipped either way, just via the exception handler rather than the branch
written for it.
The version is now decoded rather than encoded, and surfaced as parser context
instead of discarded: at the expected version it is logged at info, and anything
else warns that the parser was written against a different one, since a schema
change is exactly what would make the output quietly wrong. The expected version
moves to a module-level constant next to SECONDARY_CACHE_DIRS.
The old message read "Trying to parse anyway" immediately before a `continue`,
which reads as a contradiction. It meant the rest of the store, not this record;
the new wording says so.
Three tests, all failing on main.
Owner
|
Thanks! Code looks good. However,
Is this true? I have not seen a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #332.
What was actually wrong
The
database_metadataskip the issue asks for is already there — but it is unreachable on exactly the profiles that hit this, because the version check guarding it raises first:item['value']isbytes.byteshas no.encode(). So on any store whose version is notb'1'—b'2'is what the corpus carries — that line raisesAttributeErrorbefore reaching thecontinue, and the record falls through to the outer handler asException parsing SiteDataProto. That is the traceback in the issue, and it is why it looks like the record is being parsed as a proto when the code plainly says to skip it.So this is the str/bytes mix-up the issue's closing note asks for a glance at. It is not near the problem; it is the problem.
Nothing was ever lost — the record was skipped either way, just through the exception handler instead of the branch written for it.
The fix
continue.SECONDARY_CACHE_DIRS.Trying to parse anyway.immediately before acontinue, which reads as a contradiction — it meant the rest of the store, not this record.Verification
Three tests driving
get_site_characteristicswith a stubbedget_ldb_records, coveringb'1',b'2'and an undecodable value. All three fail onmain:Full suite on this branch: 242 passed, 2 skipped, 80 subtests passed.
I did not have the
magnet.ctf_2020corpus dataset to hand, so the reproduction here is the synthetic record from the issue's own log line rather than a corpus run — the record shape is copied verbatim from it.