feat: bindings for WebAssembly

[SBrandeis]

TL;DR

A new bindings library using https://github.com/wasm-bindgen/wasm-bindgen to generate a npm package that can be used in a browser context

Goal would be to publish it as a npm package so users can simply npm install tokenizers-web (or any other name) and use tokenize locally in their browser apps.

import { Tokenizer } from "tokenizers-web";

using tokenizer = Tokenizer.from_pretrained("openai/gpt-oss-120b");

const ids = tokenizer.encode("Hello world!");
console.log(ids); // Uint32Array [15496, 995, 0]
console.log(tokenizer.decode_tokens(ids)); // ["Hello", "Ġworld", "!"]
console.log(tokenizer.decode(ids)); // "Hello world!"

TODO

Release package

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Nice!
Examples are cool, but imo a readme + pointing to the example space makes our changes minimal / better!

Let's try to update ptr_hash instead the maintainer is great and always responsive ! if possible !

[ArthurZucker]

we have from_file and from_pretrained in other APIs

[ArthurZucker]

still to adress

[SBrandeis]

Browsers don't expose the user's file system so from_file is a bit moot here
To load from the user's file system you would use <input type="file"> and a handle like that:

input.onchange = async () => {
  const tokenizer = Tokenizer.from_json(await input.files[0].text());
};

from_pretrained has been implemented, however

[daaain]

I've been compiling the 1.0 crates to wasm for my browser tokenizer playground and hit the same things, so a few notes from that:

1. SIMD is compiled out. bitcannon's wasm kernel is behind cfg(target_feature = "simd128"), and .cargo/config.toml doesn't set it, so this build uses the scalar pre-tokenizer. Adding '-C', 'target-feature=+simd128' to the rustflags turns it on. But now that I measured it, in Node the flag makes no visible encode difference across six models, so maybe moot point.
2. decode doc vs default: the doc for skip_special_tokens says false when not specified (src/lib.rs:63), but the code does unwrap_or(true) (:71). The test comment says skipping is intended, so I think the doc is what's wrong.
3. node-release.yml: the diff moves contents: read back to the workflow level, which looks like it undoes Scope GITHUB_TOKEN permissions per job #2448. Maybe a rebase leftover?
4. Per-token decoded text. decode_tokens gives vocab spellings (Ġworld), but a UI that labels each token needs decoded text ( world). Would a decode_each(ids) fit here? Mine is ~5 lines: daaain@b873c34. Returning one string plus offsets would be faster; I can measure that as a follow-up.
5. Gemma 2 doesn't load (a core bug, not in this PR): the fix is in a separate PR, Load byte-fallback BPE vocabs that lack some <0xNN> codes #2461.
6. cargo fmt --check fails on src/lib.rs and src/options.rs (trailing whitespace, one long line).

On ptr_hash: I'm happy to open the upstream PR Arthur suggested. Since the timings only feed trace!, taking timestamps only when log_enabled!(Trace) would avoid adding a web-time dependency.

Update: PR created RagnarGrootKoerkamp/PtrHash#41

Update 2: merged and released in PtrHash 2.1.2

[SBrandeis]

Thanks for porting the fix upstream in RagnarGrootKoerkamp/PtrHash#41 @daaain

[SBrandeis]

Returning one string plus offsets would be faster

Shipping offset tracking is in the pipeline for v1, fyi

[daaain]

Thanks for porting the fix upstream in RagnarGrootKoerkamp/PtrHash#41

No worries, it was a simple fix!

BTW I got this branch's WASM version fully working in my online tokenizer and it's fast!

Here's a readme about the bits I needed to do (and the script I used), with the PtrHash fix in and SIMD flag added, it's pretty much there (just the Gemma 2 fix and the decode_each method missing): https://github.com/daaain/online-llm-tokenizer/tree/main/scripts/tokenizers-wasm

There's a transformers.js parity checker in there too if you're interested, but that might not be this branch's remit

[ArthurZucker]

LGTM otherwise, trusting you entirely on the js / web but biding is alright! Le't sminimize files, code, etc if possilbe to have an as thin layer as possible!

[ArthurZucker]

IMO if we can point to a space, I'd rather have that 👀 or an externa tokenizers example repo?

[daaain]

I'm happy to volunteer my vanilla JS repo to be an example and to update to the canonical usage once this branch is merged and released!

[ArthurZucker]

still to adress