Fix tokens split across content streams - #76
Open
rztaylor wants to merge 1 commit into
Open
Conversation
rztaylor
marked this pull request as ready for review
July 20, 2026 07:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Text extraction can loop indefinitely when a page uses an array of content streams and a PDF object starts in one stream and finishes in the next.
This was found while processing an otherwise readable PDF whose page
/Contentsentry contains two streams. Other PDF implementations completed the document, butPage.GetPlainTextin this package consumed CPU without returning.Synthetic test case
The added regression test constructs a minimal, one-page, six-object PDF entirely in memory. The page declares:
The first content stream starts a
TJarray:The second stream completes that array and invokes the operator:
The expected extracted text is
Hello world. No real-world or user-provided PDF is included in the test.Evidence of failure
On
masterat5959a4027728, with only the regression test added, this command does not complete normally:It terminates at the test timeout with the active stack in:
This reproduces the non-progress condition without the original document.
Root cause analysis
Interpretrecognizes that a PDF stream value may be an array, but it currently creates a newbuffer(lexer) for each array element. The first buffer reaches end-of-stream whilereadArrayis still parsing the incompleteTJarray. Because the remainder of the array is only available through the next buffer, the current parser cannot make progress.A page content-stream array is one logical content sequence: its streams are interpreted in order as though concatenated. Lexical state therefore needs to be preserved across each stream boundary.
Proposed fix
For an array value, collect the ordered stream readers into one
io.MultiReader, then create onebufferover that combined reader. A single stream continues to use its existing reader. The stack and interpreter behavior are otherwise unchanged.This lets
readArraycontinue into the second stream, after which the regression test completes and extractsHello world.Validation
go test ./...passes with this change.go test -v ./..., andgo build -v ./...on Go 1.24.