normtext

Deterministic, composable text normalization for Go: the mechanical "clean the text before you sketch it" layer that similarity and search tooling usually punts to the caller.

Built as the preprocessing companion to semblance (deterministic text-similarity sketching):

import (
    "github.com/ophymx/normtext"
    "github.com/ophymx/semblance"
)

clean := normtext.ForShingling()
sim := semblance.Similarity(clean(a), clean(b))

Install

go get github.com/ophymx/normtext

Requires Go 1.25+. Dependencies: golang.org/x/text (and golang.org/x/net if you import the markup sub-package).

Your pipeline is part of your frozen configuration

normtext output feeds stored similarity signatures: a change in normalization is a change in every signature computed downstream, exactly like changing a MinHash seed. normtext therefore treats pipeline definitions as frozen surface:

• Presets never change behavior under their name. An improved preset ships as a new name (ForShingling2); the old name keeps working and keeps its output forever.
• Pipelines carry a storable identity. Callers persisting signatures should record it alongside the sketch parameters:

p := normtext.ForShinglingPipeline()
p.ID() // "normtext/1:stripinvisible|nfc|lower|straightenpunct|collapsespace"

Custom chains get the same treatment via normtext.NewPipeline and normtext.Named, plus golden tests on your side. Six months from now, "which normalization produced these signatures?" should be a lookup, not an archaeology dig.

Transforms

Each transform is a func(string) string, pure and safe for concurrent use. Input that would not change is returned unchanged — same backing array, zero allocations for clean ASCII (asserted in tests).

Transform	What it does
NFC / NFKC	Unicode canonical / compatibility composition
Lower	Unicode lowercasing (language-neutral)
Fold	full case folding for caseless matching (ß → ss)
CollapseSpace	whitespace runs → one space, trim ends
Newlines	CR, CRLF, NEL, LS, PS → \n
StripDiacritics	café → cafe (NFD, drop marks, NFC)
StripInvisible	remove zero-width chars, bidi controls, BOM, soft hyphens, non-whitespace controls
StraightenPunct	smart quotes/dashes/ellipsis → ASCII
FoldConfusables	UTS #39 skeleton: homoglyph spoofs (Cyrillic "раураl") fold to their prototypes ("paypal")
MaskPattern	regex → placeholder for URLs, emails, numbers

Compose with Chain/Apply, or use a frozen preset:

• ForShingling — the conservative default before word shingles: stripinvisible | nfc | lower | straightenpunct | collapsespace.
• Aggressive — recall-maximizing fuzzy matching: adds NFKC, case folding, diacritic stripping, and confusable folding. Note that UTS #39 skeletons rewrite some ASCII (m → rn, 1 → l): Aggressive output is for matching, never display.

Sub-packages

Optional imports; the root package depends on none of them.

Package	What it gives you
message	email/NNTP structural stripping: quoted replies + attribution (StripQuotes), signatures (StripSignature, plus Usenet decorative-rule sign-offs via StripSignatureRules), header blocks (StripHeaders)
markup	StripTags: HTML/XML → normalized plain text via x/net/html tokenization (entities decoded, block boundaries → newlines)
code	SplitIdentifiers: camelCase/PascalCase/snake_case → space-separated lowercase words

A reply-corpus pipeline, end to end:

p := normtext.NewPipeline(
    message.StripHeadersStep(),
    message.StripQuotesStep(),
    message.StripSignatureStep(),
    normtext.StripInvisibleStep(),
    normtext.NFCStep(),
    normtext.LowerStep(),
    normtext.StraightenPunctStep(),
    normtext.CollapseSpaceStep(),
)
sig := minhasher.Sketch(shingle.Words(p.Apply(post), 3))
// store sig together with p.ID()

Determinism

Same input + same pipeline → same output, on every platform, in every process. The one honest dependency is the Unicode data version, which comes from two places:

• golang.org/x/text — NFC/NFKC, casing, folding. x/text ships tables for more than one Unicode generation and picks one at build time with go1.N build constraints, so the active generation is selected jointly by the x/text version in go.mod and the building Go toolchain. An upgrade of either that crosses a generation is a signature-affecting change — a dedicated test probes generation-distinguishing code points so it fails CI visibly instead of drifting silently.
• Tables generated into this repository — Unicode categories (Mn, Cf) and the UTS #39 confusables table (which x/text does not ship). These shift only on a deliberate, reviewed regeneration commit.

Two facts keep that drift surface small. Unicode's stability policies make generation changes strictly additive — newly assigned code points gain behavior, assigned ones never change (measured 15.0.0 → 17.0.0: 158 code points differ, every one newly assigned in Unicode 16/17 — new letters and marks, case pairs, and compatibility-mapped symbols; zero changes to existing behavior) — so text containing only code points assigned in the active generation keeps its normalization forever. And pure-ASCII paths, whitespace, line separators, punctuation mappings, and the message package's rules are frozen in source: unconditionally stable. Behavior is pinned by golden tests (testdata/*.tsv), regenerated only via go test -update as a reviewed diff.

Invalid UTF-8 is never repaired but always handled deterministically; each transform's godoc says exactly how.

Non-goals

No stemming, stopwords, or language detection (mechanical code-point rules only); no boilerplate/readability extraction; no encoding detection (input is UTF-8 — see x/text/encoding); no tokenization for search (that's semblance's side of the boundary); nothing model-based, no I/O.

License

MIT