Skip to content

Split short coCondenser documents into paired spans - #1

Merged
capemox merged 1 commit into
mainfrom
fix/cocondenser-short-doc-spans
Aug 16, 2026
Merged

Split short coCondenser documents into paired spans#1
capemox merged 1 commit into
mainfrom
fix/cocondenser-short-doc-spans

Conversation

@capemox

@capemox capemox commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Summary

  • split a tokenized document in half when it is too short to produce two maximum-length chunks
  • retain the existing error boundary for documents with fewer than two tokens
  • add regression coverage for both cases

Validation

  • uv run ruff check .
  • uv run mypy --python-version 3.10 src/pretense
  • uv run pytest -q (109 passed)

This intentionally does not address token-ID span preparation; that independent Transformers 5 compatibility issue is handled in a separate PR.

@capemox
capemox force-pushed the fix/cocondenser-short-doc-spans branch from fa61092 to ffcb97f Compare August 16, 2026 19:26
@capemox
capemox merged commit 12d3a0f into main Aug 16, 2026
4 checks passed
@capemox
capemox deleted the fix/cocondenser-short-doc-spans branch August 16, 2026 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant