Skip to content

Local dev venv (macOS, reused .venv) collects 10 fewer tests than CI's hash-pinned Linux install (6572 vs 6582) #287

Description

@cdeust

Description

While verifying PR #284 (stdio drain-on-EOF fix), the CI `Test (Python 3.12)`
job's "Check advertised test count" step failed because `pytest --collect-only -q`
on the CI runner (ubuntu-latest, `pip install --require-hashes -r
requirements/ci-postgresql.txt`) collects 6582 tests, while the same
command against the locally-reused `.venv` (macOS, built earlier via `uv
sync`) collects only 6572 — a delta of exactly 10.

Confirmed this is not a Python-version artifact: the `Test (Python 3.13)`
leg on the same CI run reports `6577 passed, 5 skipped` = 6582 total,
matching the 3.12 leg's collected count exactly. So the divergence is
between "CI's Linux + hash-pinned `requirements/ci-postgresql.txt`
environment" and "this local macOS dev venv", not between Python versions.

Ruled out during investigation:

  • Setting `CI=true DATABASE_URL=postgresql://cortex:cortex@localhost:5432/cortex`
    (CI's exact env vars) locally does NOT change the local count (still 6572) —
    so it is not gated on those variables.
  • `psycopg` (3.3.3) and `sqlite-vec` (0.1.9) are both installed locally, and
    `sqlite3.Connection.enable_load_extension` works on the local interpreter
    (uv-managed `cpython-3.12.12-macos-aarch64-none`) — so the "macOS lacks
    extension-capable Python" pattern (documented in an earlier lesson) does not
    directly apply to this interpreter.
  • A likely lead not yet chased to ground: `pgvector` is 0.4.2 locally vs.
    the CHANGELOG's stated lock-pinned 0.5.0 ("the version whose psycopg
    loader change `pg_store.vector_to_bytes` is written for") — the local venv
    was built at an earlier point in time and may have drifted from `uv.lock`.
    Several `tests_py/infrastructure/test_pg
    *.py` files use
    `pytest.importorskip`, which (unlike a runtime `skip`) removes a module's
    tests from `--collect-only`'s count entirely on import failure — a
    plausible mechanism for a "same code, different collected count" gap, but
    not confirmed against this exact version delta.

Reproduction

```

Local (macOS, reused venv):

cd Cortex && pytest --collect-only -q | tail -1

=> 6572 tests collected

CI (ubuntu-latest, Test (Python 3.12) job, run 30521851911):

=> "collected 6582 tests" (Check advertised test count step)

```

Acceptance criteria

  • Identify the exact 10 test items CI collects that the reused local
    macOS venv does not (or vice versa if the direction is reversed).
  • Determine whether the cause is the `pgvector` version drift (0.4.2 vs.
    0.5.0), a different missing/extra package, or a platform-conditional
    collection path.
  • Either fix the local venv rebuild instructions (CONTRIBUTING.md) to
    guarantee parity with `requirements/ci-postgresql.txt`, or fix the
    test-collection gate so it is not silently sensitive to this kind of
    drift (e.g. an explicit test asserting the two known extras produce the
    same collected set, mirroring the existing pyright-env-parity test).

Non-blocking note

PR #284 works around this by using CI's own verified collected count (6582)
directly, since CI is the environment the doc-claims gate is actually
enforcing parity with; this issue tracks the underlying local/CI parity gap
itself, which pre-dates that PR and is outside its blast radius.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions