chore: bump tokenizers from 0.21.4 to 0.23.1

[dependabot]

Bumps tokenizers from 0.21.4 to 0.23.1.

Release notes

Sourced from tokenizers's releases.

Release v0.23.1

TL;DR

tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on pyo3 0.27 without free-threaded support). 0.23.1 is the version where everything actually goes out the door together: full Node multi-platform wheels for the first time in years, Python 3.14 (regular and free-threaded 3.14t), full type hints for every Python class, and a stack of measurable perf wins on the BPE / added-vocab hot paths.

There is no functional 0.23.0 published — we tag 0.23.1 directly so users don't accidentally pull a never-shipped version.

---

🚨 Breaking changes

• Drop Python 3.9 (#1952) — requires-python = ">=3.10"; 3.9 users stay on 0.22.x.
• add_tokens normalizes content at insertion (#1995) — re-saved tokenizer.json may differ in the added_tokens block. Existing files load unchanged.
• Type stubs are precise (#1928, #1997) — methods that returned Any now return real types; mypy --strict may surface previously-hidden errors. Stub layout also moved from tokenizers/<sub>/__init__.pyi to tokenizers/<sub>.pyi. This breaks the surface of some of the processors like RobertaProcessign's __init__ .
• 3.14t-only: setters/getters return PyResult<T> because of Arc<RwLock<Tokenizer>>; a poisoned lock surfaces as PyException instead of a panic.

---

⚡ Performance — measured locally on this Mac, not lifted from PRs

Run with cargo bench --bench <name> -- --save-baseline v0_22_2 on v0.22.2, then --baseline v0_22_2 on v0.23.1. Numbers are point-in-time wall clock on a single laptop; relative deltas are what matters, absolute numbers will differ on CI hardware.

Added-vocabulary deserialize — the headline win (#1995, #1999)

bench: improve added_vocab_deserialize to reflect real-world workloads (#2000) is now representative of how transformers actually loads tokenizer.json files. The combined effect of daachorse for the matching automaton plus the normalize-on-insert refactor is enormous on this workload:

benchmark	v0.22.2	v0.23.1	change
100k tokens, special, no norm	~410 ms	248 ms	−40%
100k tokens, non-special, no norm	~7.1 s	273 ms	−96%
100k tokens, special, NFKC	~395 ms	235 ms	−40%
100k tokens, non-special, NFKC	~7.4 s	290 ms	−96%
400k tokens, special, no norm	~15 s	980 ms	−94%

Real-world impact: loading a Llama-3-style tokenizer with a large set of added tokens dropped from "noticeable pause" to "instant".

BPE encode

benchmark	v0.22.2	v0.23.1	change
BPE GPT2 encode batch, no cache	530 ms	446 ms	−16%
BPE GPT2 encode batch (cached)	690 ms	685 ms	noise
BPE GPT2 encode (single)	1.95 s	1.94 s	noise
BPE Train (small)	32.6 ms	31.5 ms	−3%
BPE Train (big)	1.01 s	988 ms	−2%

The BPE per-thread cache PR (#2028) shows much larger wins on highly-parallel workloads (+47–62% at 88+ threads on a server box, per the PR's own measurements on Vera). Single-thread batch numbers above are flat or slightly improved because cache-hit overhead was already low without contention.

Llama-3 encode

... (truncated)

Commits

• 7f1623b Bump version to 0.23.1
• bbe43ad ci: release workflow fixes (node + python) (#2043)
• ab0c5d8 Fix node release (#2034)
• decd8e0 bindings/python: free-threaded Python (3.14t) support (#2041)
• 3992692 update for release (#2033)
• bcdd25b BPE cache: per-thread read-through cache to avoid RwLock atomics on hits (#2028)
• 618eb38 Bump follow-redirects in /tokenizers/examples/unstable_wasm/www (#2024)
• b6b1688 chore: bump doc-builder SHA for PR upload workflow (#2025)
• 19015d6 fix: use uvx --with cairosvg instead of uv pip install --system (#2021)
• efbcc68 Ci benchmarks (#2019)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Labels

The following labels could not be found: dependencies. Please create it before Dependabot can add it to a pull request.

Please fix the above issues or remove invalid values from dependabot.yml.

[git-mind-pr-guardian]

GitMind PR Review

{'pr_url': '#6', 'repo': 'Ayyankhan101/Transformer-In-Rust', 'pr_num': '6', 'title': 'chore: bump tokenizers from 0.21.4 to 0.23.1', 'author': 'dependabot[bot]', 'additions': 10, 'deletions': 4, 'changed_files': 2, 'review': '## Summary\nThis PR bumps the tokenizers dependency from version 0.21.4 to 0.23.1, which is a relatively significant update. The changes seem to be mostly related to the updated dependencies and their checksums. Overall, the update appears to be straightforward and should not introduce any major issues.\n\n## Issues\nNo significant issues found.\n\n## Verdict\n✅ APPROVE\nThe update to tokenizers version 0.23.1 seems to be a standard dependency update with no apparent issues or breaking changes, so the PR can be approved.'}

---

Auto-generated by GitMind — AI-powered code analysis.

[Ayyankhan101]

Superseded by #16, which takes this bump together with the other thirteen and verifies them against current master. All 10 CI checks pass there; the failures shown here were inherited from the pre-fix base commit, not caused by the bump.

[dependabot]

OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting @dependabot ignore this major version or @dependabot ignore this minor version. You can also ignore all major, minor, or patch releases for a dependency by adding an ignore condition with the desired update_types to your config file.

If you change your mind, just re-open this PR and I'll resolve any conflicts on it.