Skip to content

pip install pyhindsight produces a package that crashes on first use #331

Description

@RyanDFIR

ccl_chromium_reader is a hard runtime dependency that exists only as a git repo, and PyPI
rejects uploads whose metadata contains direct-URL dependencies. So it cannot be declared
in pyproject.toml, and a plain pip install pyhindsight yields a package that fails with
a raw ModuleNotFoundError the first time it is used, with no hint at the fix.

This is the worst first-run experience in the project. Note the frozen binaries already
ship ccl inside the exe, so the released artifact is effectively vendored already. Only the
pip path is broken.

There is a second, separate reason to care, which is control over the library's I/O.
ccl_chromium_reader reports records it cannot decode with bare print(), 43 calls across
7 modules. On Windows a redirected stdout is cp1252, and printing undecodable record text
into that stream from inside a running parser raises, at which point the parser's own
except reports its artifact as unreadable. That cost 954 Session Storage records across
three of nine test profiles before a stdout guard was added. The guard defuses all 43 calls
and stays regardless as defence against any library, but it is a workaround: those calls
should be log.warning(). Hindsight also monkeypatches the sessionstorage module's logger
at runtime because the module sets its own logger to None, which is the same problem
solved the same unsatisfying way.

Options, in preference order:

  1. Ask CCL to publish to PyPI. It already builds a clean wheel, so it is trivial on their
    end, and dependencies then just gets ccl-chromium-reader>=0.3. This closes the
    distribution half and does nothing for the control half.
  2. Vendor it. MIT permits it with attribution, and the repo already vendors Chromium protos.
    A copy under pyhindsight/_vendor/ plus a shim that registers the vendored package under
    the real name means no call-site changes are needed. Add a sync script in the manner of
    regen-protos.sh.
  3. Minimum hardening regardless of the choice: guard the top-level import with a try/except
    that prints the exact pip install git+... command instead of a raw traceback.

If vendoring, the discipline matters more than the copy. Vendor unmodified first at a
recorded commit and land each local change as its own reviewable commit, so "what have we
changed from upstream" stays answerable. File every local patch upstream, because the
failure mode of vendoring is a silent private fork where nothing ever gets reported. Keep
the reported version honest, since Hindsight logs the ccl module version and a report may
rest on it. And accept that drift is the real cost: Chrome's formats move constantly and
ccl tracks them, so a vendored copy that is not synced is worse than a pin.

For sizing, measured against the currently pinned commit: 22 Python files, 7,957 lines,
861 KB installed. MIT, with a per-file copyright header on 15 of the 22 files; the 7
without should be confirmed as covered by the repository LICENSE before relying on the
per-file headers.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugdependenciesPull requests that update a dependency file

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions