Skip to content

Fix tokens split across content streams - #76

Open
rztaylor wants to merge 1 commit into
ledongthuc:masterfrom
rztaylor:fix/content-stream-array-tokenization
Open

Fix tokens split across content streams#76
rztaylor wants to merge 1 commit into
ledongthuc:masterfrom
rztaylor:fix/content-stream-array-tokenization

Conversation

@rztaylor

@rztaylor rztaylor commented Jul 20, 2026

Copy link
Copy Markdown

Problem

Text extraction can loop indefinitely when a page uses an array of content streams and a PDF object starts in one stream and finishes in the next.

This was found while processing an otherwise readable PDF whose page /Contents entry contains two streams. Other PDF implementations completed the document, but Page.GetPlainText in this package consumed CPU without returning.

Synthetic test case

The added regression test constructs a minimal, one-page, six-object PDF entirely in memory. The page declares:

/Contents [4 0 R 5 0 R]

The first content stream starts a TJ array:

BT /F1 12 Tf 20 100 Td [(Hello)

The second stream completes that array and invokes the operator:

( world)] TJ ET

The expected extracted text is Hello world. No real-world or user-provided PDF is included in the test.

Evidence of failure

On master at 5959a4027728, with only the regression test added, this command does not complete normally:

go test -run TestInterpretContinuesTokensAcrossContentStreams -count=1 -timeout=2s

It terminates at the test timeout with the active stack in:

buffer.readArray
buffer.readObject
Interpret
Page.GetPlainText
TestInterpretContinuesTokensAcrossContentStreams

This reproduces the non-progress condition without the original document.

Root cause analysis

Interpret recognizes that a PDF stream value may be an array, but it currently creates a new buffer (lexer) for each array element. The first buffer reaches end-of-stream while readArray is still parsing the incomplete TJ array. Because the remainder of the array is only available through the next buffer, the current parser cannot make progress.

A page content-stream array is one logical content sequence: its streams are interpreted in order as though concatenated. Lexical state therefore needs to be preserved across each stream boundary.

Proposed fix

For an array value, collect the ordered stream readers into one io.MultiReader, then create one buffer over that combined reader. A single stream continues to use its existing reader. The stack and interpreter behavior are otherwise unchanged.

This lets readArray continue into the second stream, after which the regression test completes and extracts Hello world.

Validation

  • Confirmed the new focused test times out on the unpatched revision.
  • Confirmed the focused test passes with this change.
  • Confirmed go test ./... passes with this change.
  • The repository has no additional contribution or PR-template instructions; its CI runs dependency download, go test -v ./..., and go build -v ./... on Go 1.24.

@rztaylor
rztaylor marked this pull request as ready for review July 20, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant