Skip to content

fix: invalidate chunks when chunkers change - #528

Open
mikemikimike wants to merge 1 commit into
agentscope-ai:mainfrom
mikemikimike:codex/reme-524-chunker-fingerprint
Open

fix: invalidate chunks when chunkers change#528
mikemikimike wants to merge 1 commit into
agentscope-ai:mainfrom
mikemikimike:codex/reme-524-chunker-fingerprint

Conversation

@mikemikimike

Copy link
Copy Markdown

Summary

Invalidates persisted file chunks whenever the configured file chunker output shape changes.

What changed

  • Adds stable configuration fingerprints to the built-in file chunkers.
  • Versions LocalFileStore chunk persistence with the active chunker fingerprint.
  • Versions FAISS and Zvec derived indexes with the same suffix.
  • Preserves the legacy v1 filename behavior for standalone stores without an application context.
  • Adds regressions for configuration changes, equivalent configuration stability, and legacy snapshot exclusion with graph repair.

Verification

  • PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest -p pytest_asyncio.plugin tests/unit/test_chunk_store_fingerprint.py tests/unit/test_file_store_consistency.py tests/unit/test_default_file_chunker.py tests/unit/test_json_file_chunker.py tests/unit/test_jsonl_file_chunker.py -q --deselect tests/unit/test_default_file_chunker.py::test_parse_empty_file
  • Result: 169 passed, 12 skipped, 1 deselected
  • ruff check passed
  • python -m compileall -q reme passed
  • git diff --check passed

The one deselected test asserts Windows backslash formatting against a POSIX-normalized path and is unrelated to this change.

Closes #524.

@jinliyl
jinliyl requested a review from ployts September 8, 2026 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Persisted chunk store is not invalidated when the chunker changes — stale "full-outline" chunks survive upgrades and are never re-chunked

1 participant