Skip to content

Skip database_metadata cleanly and report the schema version it carries - #347

Merged
RyanDFIR merged 1 commit into
RyanDFIR:mainfrom
dchaudhari7177:fix/332-site-characteristics-database-metadata
Sep 7, 2026
Merged

RyanDFIR merged 1 commit into
RyanDFIR:mainfrom
dchaudhari7177:fix/332-site-characteristics-database-metadata

Conversation

@dchaudhari7177

Copy link
Copy Markdown
Contributor

Closes #332.

What was actually wrong

The database_metadata skip the issue asks for is already there — but it is unreachable on exactly the profiles that hit this, because the version check guarding it raises first:

if item['key'] == b'database_metadata':
    if item['value'] != b'1':
        log.warning(f' - Expected type 1; got type {item["value"].encode()}. ...')
    continue

item['value'] is bytes. bytes has no .encode(). So on any store whose version is not b'1'b'2' is what the corpus carries — that line raises AttributeError before reaching the continue, and the record falls through to the outer handler as Exception parsing SiteDataProto. That is the traceback in the issue, and it is why it looks like the record is being parsed as a proto when the code plainly says to skip it.

So this is the str/bytes mix-up the issue's closing note asks for a glance at. It is not near the problem; it is the problem.

Nothing was ever lost — the record was skipped either way, just through the exception handler instead of the branch written for it.

The fix

  • Decode rather than encode, so the branch reaches its continue.
  • Surface the schema version as parser context rather than discarding it, as the issue suggests: info at the expected version, a warning otherwise. A schema change is exactly what would make this parser's output quietly wrong, so it is worth saying out loud.
  • The expected version becomes a module-level constant next to SECONDARY_CACHE_DIRS.
  • Reword the message. The old text read Trying to parse anyway. immediately before a continue, which reads as a contradiction — it meant the rest of the store, not this record.

Verification

Three tests driving get_site_characteristics with a stubbed get_ldb_records, covering b'1', b'2' and an undecodable value. All three fail on main:

FAILED test_the_expected_version_is_skipped_and_reported
FAILED test_an_unexpected_version_warns_without_raising
FAILED test_an_undecodable_version_still_does_not_raise

Full suite on this branch: 242 passed, 2 skipped, 80 subtests passed.

I did not have the magnet.ctf_2020 corpus dataset to hand, so the reproduction here is the synthetic record from the issue's own log line rather than a corpus run — the record shape is copied verbatim from it.

The Site Characteristics store holds a `database_metadata` record whose value is
the store's bare schema version, not a SiteDataProto. The skip for it was
already there, but the version check guarding it called .encode() on
`item['value']`, which is bytes and so has no .encode(). On any store not at the
expected version -- b'2' is in the wild -- that raised AttributeError before
reaching the `continue`, and the record surfaced through the outer handler as

    Exception parsing SiteDataProto ({'key': b'database_metadata', 'value': b'2', ...})
    AttributeError: 'bytes' object has no attribute 'encode'

It fires on healthy profiles, so it is pure noise, and noise that trains the
reader to skim past errors in the log. Nothing was actually lost: the record was
skipped either way, just via the exception handler rather than the branch
written for it.

The version is now decoded rather than encoded, and surfaced as parser context
instead of discarded: at the expected version it is logged at info, and anything
else warns that the parser was written against a different one, since a schema
change is exactly what would make the output quietly wrong. The expected version
moves to a module-level constant next to SECONDARY_CACHE_DIRS.

The old message read "Trying to parse anyway" immediately before a `continue`,
which reads as a contradiction. It meant the rest of the store, not this record;
the new wording says so.

Three tests, all failing on main.
@RyanDFIR

RyanDFIR commented Sep 7, 2026

Copy link
Copy Markdown
Owner

Thanks! Code looks good. However,

b'2' is what the corpus carries

Is this true? I have not seen a 2 in the wild. The code works regardless though.

@RyanDFIR
RyanDFIR merged commit 24f2957 into RyanDFIR:main Sep 7, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skip the database_metadata bookkeeping record in the Site Characteristics parser

2 participants