Make system-uchardet a Meson feature option (auto/enabled/disabled)

[wbarnha]

Implements the feature-flag suggestion from the review of #56 (#56 (comment)).

Builds on top of @mgorny's #56 (that commit is included here, unchanged) and refines the system-uchardet switch from a hard boolean into a Meson feature option, so one switch serves more environments.

What changed

• meson.options — type: 'boolean' → type: 'feature', value: false → value: 'disabled'. Also fixes the description typo (chardet → uchardet).

• src/cchardet/meson.build — three-way selection:

  -Dsystem-uchardet=	Behaviour
  disabled (default)	Always build the bundled copy — unchanged behaviour for the published wheels.
  enabled	Require the system library; hard fail if missing or too old. What distro packaging wants.
  auto	Use the system library if new enough, else fall back to the bundled copy. Handy for source installs.

  Passing the feature to dependency('uchardet', required: system_uchardet) gives the enabled (hard-fail) vs auto (quiet fallback) distinction for free. The existing has_function('uchardet_get_n_candidates') probe still rejects a found-but-too-old system library, and now only error()s under enabled — auto falls through to bundled.

• README.md — a "Building against a system uchardet" section for packagers: the three states, the pip install . --config-settings=setup-args=-Dsystem-uchardet=enabled invocation, and the git-uchardet requirement. The # TODO to swap has_function() for a version: constraint once upstream tags a release is kept.

Verification (local)

meson setup exercised for every state (submodule checked out, no system uchardet present):

• disabled (default) and auto → configure the bundled copy; full build + test suite pass: 125 passed, 1 skipped.
• enabled with no system library → configuration fails (exit 1) as intended: ERROR: Dependency "uchardet" not found.

The whole meson.build parses (so the too-old-error() branch is syntax-checked), and the bundled extension compiles and links cleanly.

Not included

• CI coverage for the system path (suggestion Add linux aarch64 wheel support #2 in the review): a Linux job that builds uchardet from git, installs it, then builds with -Dsystem-uchardet=enabled and runs the tests. I left it out to keep this PR focused and locally-verifiable — happy to add it as a follow-up commit here or a separate PR, whichever you prefer.

Credit to @mgorny for the original declare_dependency refactor and system-uchardet support in #56; this supersedes that PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BnTZQEXqpedyrFAgwRnbsA

---

Generated by Claude Code

[wbarnha]

Measured what this branch does against current master, since it predates the 3.0.1 fixes.

Summary: with -Dsystem-uchardet=enabled this reproduces the defect 3.0.0 was yanked for, and CI stays green.

Method

Merged this branch into master (8dbf2fb) — it merges cleanly, no conflict. Built the system library from the pinned submodule commit (06029ec) so the overlay is the only variable between the two builds, then built with -Dsystem-uchardet=enabled and confirmed ldd reports libuchardet.so.0. Corpus is benchmarks/make_nonutf8_corpus.py, median of 3 runs.

Build	non-UTF-8 MB/s	UTF-8 mislabel
bundled 3.0.1	3.10	0.0%
system-uchardet	1.44	16.2%
v2.2.1	5.39	0.0%

The system path links upstream's nsMBCSGroupProber, so the encoding-only overlay from #62 is not compiled in and the generic language pass is back.

Nothing catches it

pytest src/tests gives 130 passed, 1 skipped on the system build — identical to the bundled build. test-system-uchardet.yml runs only pytest, so this job goes green while shipping the regression.

What does survive

The UTF-8 fast path and the chunked-feeding fix for the long-input SIGABRT live in _cchardet.pyx, not the overlay, so both are retained — verified: UTF-8 throughput is unchanged and 644 KiB of multibyte input does not crash. The damage is confined to non-UTF-8 detection.

A fix that turned out not to work

An engine-independent guard in the Cython layer — reject a UTF-8 verdict the bytes contradict, fall through to the next candidate — looked like the cheap repair. Tested against the live library: of the 268 mislabeled documents, only 50 (19%) have a correct lower candidate. The overlay changes candidate generation, not just which candidate is picked, so there is no cheap substitute.

Caveat on the numbers

make_nonutf8_corpus.py builds documents from random mid-word slices and mixes languages within a single document, which is adversarial for language-model-based detection. Treat 16.2% as a corpus-specific rate rather than a field rate. That said, reporting bytes as UTF-8 which do not decode as UTF-8 is a hard error regardless of corpus realism, and the bundled build scores 0.0% on identical input.

Suggested path

1. Rebase. The base here is feee038, before the overlay, the performance fix and the 3.0.1 release — the "125 passed" figure in the description is 130 on current master.
2. Add the correctness gate to test-system-uchardet.yml: run the non-UTF-8 benchmark with --max-utf8-mislabel-rate 0.02. It fails today, which is the point — it makes the defect visible instead of green.
3. Warn packagers in the README table that enabled currently produces a build with known-worse non-UTF-8 detection.
4. The real unblock is upstream. This is an upstream defect rather than a cChardet one: the stock v3.0.0 wheel measures the same 16.2%. nsUTF8Prober never rejects invalid byte sequences and its 0.5 confidence floor always clears CANDIDATE_THRESHOLD, so upstream leans on the language pass to discard it. Once uchardet carries the UTF-8 validation the overlay performs, this can gate on a version: constraint — exactly what the existing # TODO anticipates.

Separately: I have two patches for uchardet's throughput problem (freedesktop uchardet #38, ~23x on the pathological case, no change in detection output). They do not help here — they fix speed, not the mislabel. The correctness fix is a second, independent upstream contribution.

---

Generated by Claude Code