Pass `revision` through the file-listing and config paths by astefanutti · Pull Request #1751 · huggingface/transformers.js

astefanutti · 2026-08-18T10:35:45Z

What

pipeline() accepts a revision, and the weights are indeed fetched from it, but four functions on the way to the file listing drop it and fall back to main:

pipeline() passes only {device, dtype} to get_pipeline_files(), though it has revision, cache_dir and local_files_only in scope
get_files() destructures a fixed option set, so anything it does not name is lost
get_model_files() does the same, then calls get_config(modelId, { config }) — dropping the revision get_config is written to accept
get_tokenizer_files() / get_processor_files() take no options at all and hardcode {} into get_file_metadata()
loadTokenizer() has options and does not pass them to get_tokenizer_files()

Each of these now forwards the pretrained options it does not name, so the whole chain reads a single revision.

How it shows up

Loading a model pinned to a commit in the browser, every file resolves twice — once at the pinned sha, once at @main:

https://huggingface.co/<repo>/resolve/<sha>/config.json
https://huggingface.co/<repo>/resolve/main/config.json            <- listing/config path
https://huggingface.co/<repo>/resolve/<sha>/tokenizer_config.json
https://huggingface.co/<repo>/resolve/main/tokenizer_config.json  <- listing path

Since the browser cache is keyed on the resolved URL, the @main copies are cached and never read again. They are small, but they are permanent, and they are confusing to anyone auditing what a pinned load actually downloaded.

The stray entries are the mild symptom. The listing can also disagree with the revision being loaded: if main has moved on — different dtypes available, a processor added, a session renamed — get_files() reports files that do not exist at the pinned commit, and get_config() reads a config that does not describe the weights being loaded.

Tests

Two cases in tests/utils/model_registry.test.js, modelled on the existing get_available_dtypes revision test:

get_files forwards revision/cache_dir to every metadata lookup, tokenizer and processor included
get_model_files reads the config from the requested revision

Both fail on main and pass with this change. pnpm test for tests/utils is green (293 passing), as are prettier --check and tsc --build.

`pipeline()` accepts a `revision`, and the weights are fetched from it, but four functions on the way to the file listing dropped it and fell back to `main`: - `pipeline()` gave `get_pipeline_files()` only `{device, dtype}` - `get_files()` destructured a fixed option set, so anything it did not name was lost - `get_model_files()` did the same, then called `get_config(modelId, {config})` - `get_tokenizer_files()` / `get_processor_files()` took no options and passed `{}` to `get_file_metadata()` The visible effect when loading a pinned model in the browser is a second set of requests at `@main`: `config.json` and `tokenizer_config.json` are fetched twice, under two different cache keys, and the copy the app never uses stays in the cache. Beyond the stray entries, the listing can also disagree with the revision being loaded — a repo whose `main` has since changed dtypes or gained a processor lists files that do not exist at the pinned commit. Each function now forwards the pretrained options it does not name, so the whole chain reads one revision. Tested with the existing `get_available_dtypes` revision test as the model: both new cases fail on `main` and pass here.

astefanutti · 2026-08-18T10:37:39Z

Reproduction with the built dist, instrumenting fetch and grouping requests by /resolve/<rev>/, loading Xenova/all-MiniLM-L6-v2 pinned to 751bff37:

Before

main  — 2 requests
    config.json
    tokenizer_config.json

PINNED 751bff37  — 4 requests
    config.json
    tokenizer.json
    tokenizer_config.json
    onnx/model_quantized.onnx

After

PINNED 751bff37  — 5 requests
    config.json
    tokenizer_config.json
    tokenizer.json
    tokenizer_config.json
    onnx/model_quantized.onnx

config.json is fetched twice before the change, once per revision, and the main copy is the one the listing and get_config() read. In a browser those two land in the cache under @main keys and are never read again.

HuggingFaceDocBuilderDev · 2026-08-19T13:51:10Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copilot

Pull request overview

Ensures model file discovery uses the same pinned revision and cache options as loading.

Changes:

Forwards pretrained options through model, tokenizer, and processor discovery.
Passes revision options from pipeline and tokenizer loading paths.
Adds regression tests for option propagation.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`packages/transformers/tests/utils/model_registry.test.js`	Tests revision forwarding.
`packages/transformers/src/utils/model_registry/get_tokenizer_files.js`	Forwards tokenizer metadata options.
`packages/transformers/src/utils/model_registry/get_processor_files.js`	Forwards processor metadata options.
`packages/transformers/src/utils/model_registry/get_pipeline_files.js`	Documents revision options.
`packages/transformers/src/utils/model_registry/get_model_files.js`	Forwards options when loading configuration.
`packages/transformers/src/utils/model_registry/get_files.js`	Propagates options across discovery.
`packages/transformers/src/tokenization_utils.js`	Passes tokenizer loading options.
`packages/transformers/src/pipelines.js`	Passes revision options into file discovery.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

astefanutti · 2026-08-19T16:51:19Z

+        cache_dir,
+        local_files_only,
+        revision,


Fixed in f5ae594 — the metadata pass now receives the same { cache_dir, local_files_only, revision } as the listing call. Since the memoization key matches the listing chain's probes, the tokenizer_config.json/preprocessor_config.json lookups are now reused instead of re-requested at main.

astefanutti · 2026-08-19T16:51:21Z

 * @returns {Promise<string[]>} An array of file names that will be loaded
 */
-export async function get_tokenizer_files(modelId) {
+export async function get_tokenizer_files(modelId, options = {}) {


Fixed in b18374e — ModelRegistry.get_tokenizer_files now accepts and forwards PretrainedOptions, with a regression test covering the public facade call.

astefanutti · 2026-08-19T16:51:23Z

 * @returns {Promise<string[]>} Array of processor file names (empty if no processor)
 */
-export async function get_processor_files(modelId) {
+export async function get_processor_files(modelId, options = {}) {


Fixed in b18374e, same commit as the tokenizer facade — options are forwarded and the public call is covered by a regression test.

The per-file metadata pass six lines below the fixed `get_pipeline_files` call still probed with empty options: for a pinned revision every expected file's metadata was read from `main`, and `cache_dir` and `local_files_only` were ignored. The empty-options memoization key also kept the two probes the listing chain had already made (tokenizer_config.json, preprocessor_config.json) from being reused, so those were requested twice. Forwarding the options reads the metadata from the pinned revision and lets those probes dedupe.

`get_tokenizer_files` and `get_processor_files` are exported through the `ModelRegistry` facade, not as bare helpers, and the facade silently dropped a second argument — so public callers passing `{ revision }` still had the existence checks read `main`. Forward the options and cover the public calls. Also document `revision`/`cache_dir`/`local_files_only` on the `get_files`, `get_pipeline_files`, and `get_model_files` facades: the published types are generated from this JSDoc, so TypeScript callers were rejected for passing options the runtime already forwards.

nico-martin

Hi @astefanutti, thank you so much for looking into this! This looks good to me. The revision now flows consistently through discovery, metadata, config, tokenizer, processor, and model loading, and the tests cover the important helper paths. No changes are requested; an end-to-end URL test would just be a non-blocking follow-up.

nico-martin · 2026-08-19T13:59:19Z

{ cache_dir, local_files_only, revision } should be passed here too.

nico-martin self-assigned this Aug 19, 2026

nico-martin requested a balanced review from Copilot August 19, 2026 14:03

Copilot started reviewing on behalf of nico-martin August 19, 2026 14:04 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

astefanutti added 2 commits August 19, 2026 16:43

nico-martin approved these changes Aug 24, 2026

View reviewed changes

Navigation Menu

Uh oh!

Pass `revision` through the file-listing and config paths - #1751

Pass `revision` through the file-listing and config paths#1751
astefanutti wants to merge 3 commits into
huggingface:mainfrom
astefanutti:fix/propagate-revision-file-listing

astefanutti commented Aug 18, 2026

Uh oh!

astefanutti commented Aug 18, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Aug 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

astefanutti Aug 19, 2026

Uh oh!

astefanutti Aug 19, 2026

Uh oh!

astefanutti Aug 19, 2026

Uh oh!

nico-martin left a comment

Uh oh!

nico-martin Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

astefanutti commented Aug 18, 2026

What

How it shows up

Tests

Uh oh!

astefanutti commented Aug 18, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Aug 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

astefanutti Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

astefanutti Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

astefanutti Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

nico-martin left a comment

Choose a reason for hiding this comment

Uh oh!

nico-martin Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants