Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 11 additions & 9 deletions corpus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,14 @@ The model compiler derives each single-byte encoding's case-folding byte map
through iconv, applies the detector's whitespace normalization, and emits 64
sorted detector-ready trigrams into `src/encoding/models/generated.ts`. It also
learns Laplace-smoothed high-byte distributions from the training split. The
evaluator uses byte likelihood to rank candidates whose trigram hit rates are
statistically competitive at a 95% margin. `models:evaluate` reports every
held-out test document, while `models:verify` checks reproducibility and fails if
any encoding or language result is incorrect. A different encoding label is
accepted only when both encodings decode the actual held-out bytes identically;
the report distinguishes these byte-equivalent results from exact matches.
CP949 is excluded because its
multibyte recogniser requires character frequency data rather than byte
trigrams.
compiler evaluator and library runtime share the normalization, trigram,
byte-likelihood, confidence, candidate-prefiltering, and tie-breaking
implementation in `src/encoding/sbcs-scoring.ts`. Candidates whose trigram hit
rates are statistically competitive at a 95% margin are ranked by byte
likelihood and deterministic encoding-family preferences. `models:evaluate`
reports every held-out test document, while `models:verify` checks
reproducibility and fails if any encoding or language result is incorrect. A
different encoding label is accepted only when both encodings decode the actual
held-out bytes identically; the report distinguishes these byte-equivalent
results from exact matches. CP949 is excluded because its multibyte recogniser
requires character frequency data rather than byte trigrams.
Loading