Fix: Harden malformed input - #78
Open
gage-marshall wants to merge 7 commits into
Open
Conversation
The cross-reference table is preallocated and indexed from values taken
straight out of the file, none of which were checked.
- /Size in an xref stream sized make([]xref, size) directly. A negative
value panicked; a large one exhausted memory. Because a mid-range value
reaches the allocator rather than the length check, this surfaced as a
Go out-of-memory error, which is fatal and cannot be recovered.
- /Index and classic subsection headers name object numbers used to index
the table. A two-element /Index of [4000000000 1] grew it by ~128GB on
one entry of input.
- Growing the table's capacity does not necessarily grow its length past
the index being written, so table[x] could still be out of range. The
xref stream path was missing the reslice the classic table path has.
- A negative trailer /Size reached table[:size] as a negative bound.
- A /W field width was used as a slice bound without a sign check, and
summed into the size of the read buffer without a magnitude check.
The table is no longer bounded by the file length: an xref stream is
compressed, so a small file can legitimately describe far more objects than
it has bytes. Instead the preallocation is capped on its own, with the table
still growing to whatever the file really contains, and object numbers are
bounded separately.
Also in the same untrusted path: startxref and /Prev offsets are range
checked before being handed to the lexer, which reports an unreadable stream
by panicking; a FlateDecode /Columns is checked before it sizes a row
buffer; object stream /Extends chains and nesting are bounded, since a cycle
there recursed until the stack was exhausted; and the object stream pair
loop stops at the end of the data instead of trusting the declared /N.
NewReaderEncrypted now converts the lexer's panics into errors. The lexer
signals malformed input by panicking, and every caller opening an untrusted
file had to recover for itself or crash.
Fixes the trailing-newline underflow too: the hand-rolled strip loop read
buf[len(buf)-1] after emptying buf, because && binds tighter than ||, so a
file ending in newlines indexed buf[-1]. The following TrimRight already
does that job correctly, so the loop is simply gone.
readByte reports end of input as a synthetic '\n' once allowEOF is set, and readHexString treated that as whitespace to skip. An unterminated hex string in a content stream or object stream therefore looped forever, consuming a core until the process was killed. Both whitespace-skipping paths now stop at end of input. seekForward computed a buffer position that could be negative, for a seek to a negative offset or to one behind the data still held in the buffer. That left pos negative, which indexed buf out of bounds on the next read rather than at the seek itself.
The loop read s[i+1] on the final iteration of an odd-length string. The callers that guard length (Value.Text, TextFromUTF16) made this look safe, but cmap.Decode passes bfchar and bfrange replacement strings taken verbatim from the file, so a replacement of an odd number of bytes panicked. A trailing odd byte cannot form a UTF-16 code unit, so it is dropped.
Content stream operators:
- Q with an empty graphics stack indexed gstack[-1]. A content stream may
hold more Q than q, and Page.Content has no error return, so this
panicked out of a public API on a one-byte content stream.
- Tm in walkTextBlocks read args[4] and args[5] without checking how many
operands were supplied.
ToUnicode CMaps:
- Entry counts were declared by the file and trusted. "268435456
beginbfchar endbfchar" appended 2.7e8 empty entries, about 8.6GB, from a
76-byte stream. A count is now only honoured if the operands to fill it
were actually supplied, which bounds the work by real input.
- A codespace range's width indexes cmap.space[4], so a 5-byte range was
out of range.
- Scaling a bfrange destination read b[len(b)-1] on an empty string.
- Interpret's own panics (mismatched begin/end, def without a dict) escaped
readCmap into Page.Content, which cannot report them. A malformed cmap
now degrades to no cmap, as a failed parse already did.
Cyclic references. /Parent, /Kids, /First and /Next are object references, so
a file can point them back at themselves. Walking them looped forever, and
the outline case recursed until the goroutine stack was exhausted, which is a
fatal error a caller cannot recover from. All four traversals are bounded.
GetTextByColumn and GetTextByRow recovered into unnamed results, so the
deferred function only reassigned its own locals: a panic surfaced as
(nil, nil) and the error was silently dropped. Naming the results fixes it.
Reader.GetPlainText and Reader.GetStyledTexts now recover too. Both call into
object resolution and Page.Content, which panic on malformed input, so they
could not honour their error returns.
Page.Content is documented as panicking, since it has no way to report a
malformed content stream and callers handling untrusted files need to know.
101 assertions across 17 tests. Every malformed case was confirmed to panic, hang, or allocate without bound before the corresponding fix, and each runs under a timeout so a regression that reintroduces an infinite loop fails rather than hanging CI. The suite also pins the behaviour the bounds must not break: a well-formed PDF, one that keeps its objects in an object stream indexed by a cross-reference stream, a well-formed multi-entry CMap, a table larger than the preallocation cap, and a compressed xref stream describing 100000 objects in under 8KB, which is the case that rules out bounding the table by file length. There were no tests in the repository before this, so the CI job that runs go test ./... had nothing to run.
Updated the README to reflect the new repository URL and usage instructions for the PDF library.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Harden parsing against malformed and hostile PDFs
A PDF is untrusted input. This package reads sizes, counts, offsets and object
numbers straight out of the file and uses them to allocate slices, index them,
and seek, mostly without checking them first. A file of a few hundred bytes can
therefore panic the process, spin a core forever, or drive a multi-gigabyte
allocation.
The starting point was
make([]xref, size)inreadXrefStream, where/Sizeis used unchecked. Auditing outward from there turned up the same missing-guard
pattern in 20 places across
read.go,lex.go,page.goandtext.go. Everyone below was reproduced against
masterbefore being fixed.This continues work already in the history:
d295ac1"fix infinity loop",1e65caa"avoid infinite loop on missing page",#11"fix-out-of-memory", andtries to close out the class rather than one more instance of it.
Why it matters
Three of these are worse than an ordinary panic, because a caller cannot defend
against them with
recover:/Sizein the mid range (e.g.1e9)/Firstin an outlinefatal error: stack overflowThe rest are panics, which matter because they escape APIs that look safe.
NewReaderdid not recover at all, so simply opening a file could take downthe caller.
Page.Contenthas no error return and still panicked.What's here
Cross-reference table (
read.go)/Size 4611686018427387904makeslice: len out of range; ~1e9-> OOM/Size -1make/Index [4000000000 1]/Index [5 1]with/Size 0index out of range [5] with length 54000000000 1subsection header/Size -1slice bounds out of range [:-1]/W [-1 2 1]slice bounds out of range [:-1]/W [2e9 2e9 2e9]make([]byte, wtotal)One of these deserves a specific note: growing a slice's capacity past an
index does not necessarily grow its length past it, so
table[x]could stillbe out of range after the grow loop. The classic-table path has the reslice that
handles this; the xref-stream path was missing it. That is the
/Index [5 1]row.The table is deliberately not bounded by the file length. That was my first
attempt and it is wrong: an xref stream is compressed, so a small file can
legitimately describe far more objects than it has bytes.
TestCompressedXrefStreamManyObjectsis the counterexample that rules it out: 100,000 objects in under 8 KB, which a
length-derived bound rejects. Instead the preallocation is capped on its own
(the table still grows to whatever the file really contains) and object numbers
are bounded separately.
File structure (
read.go)The trailing-newline strip loop read
buf[len(buf)-1]after emptyingbuf:&&binds tighter than||, so once the loop emptiesbufthe third operandstill evaluates and indexes
buf[-1]. A ~100-byte file ending in newlinespanics. The next line's
bytes.TrimRight(buf, "\r\n\t ")already does this jobcorrectly, so the loop is simply deleted.
startxrefand/Prevoffsets are now range checked before reaching the lexer,which reports a permanently unreadable stream by panicking. A negative
startxrefalso producedslice bounds out of range [:4101] with capacity 4096inside
reload.NewReaderEncryptednow converts the lexer's panics into errors. The lexersignals malformed input by panicking (
buffer.errorf), which is fineinternally, but it meant every caller opening an untrusted file had to recover
for itself or crash.
Lexer (
lex.go)readBytereturns a synthetic'\n'at end of input onceallowEOFis set, andreadHexStringtreated that as whitespace to skip � so<ABwith no closing>looped forever.
seekForwardcould leavebuf.posnegative, which indexed outof bounds on the next read rather than at the seek.
Content streams, CMaps, outlines (
page.go,text.go)Qwith an empty graphics stackindex out of range [-1]out ofPage.Content268435456 beginbfcharindex out of range [4] with length 4(cmap.spaceis[4])()index out of range [-1]index out of range [3]inutf16Decode/Parent,/Kids,/Next/FirstCMap entry counts were declared by the file and trusted. A count is now honoured
only if the operands to fill it were actually supplied, which bounds the work by
real input rather than by an arbitrary limit.
Interpret's own panics (mismatched begin/end,def without open dict) alsoescaped
readCmapintoPage.Content. A malformed cmap now degrades to no cmap,which is what a failed parse already did.
Two bugs found along the way
GetTextByColumnandGetTextByRowrecovered into unnamed results:The deferred function only reassigned its own locals, so a panic surfaced as
(nil, nil), caller sees no error and no data. Naming the results fixes it.Reader.GetPlainTextandReader.GetStyledTextsnever recovered at all, thoughboth call into resolution and
Page.Content, so they could not honour theirerror returns either.
Compatibility
No exported signature changes. Behaviour changes only for input that previously
crashed, hung, or silently returned nothing.
Two deliberately conservative choices, to avoid breaking files that work today:
Tmoperand check inwalkTextBlocksis a lower bound (< 6), not!= 6.argsis built from the whole stack, so a stream leaving extra operands behindkeeps working exactly as before.
resolvecarries its recursion depth as a parameter rather than onReader.A
Readeronly reads immutable state once opened, so several goroutines canread from one concurrently; a counter on the struct would have quietly
introduced a data race.
TestConcurrentReadspins this.The limits are named constants at the top of
read.goandpage.go, each with acomment explaining what it bounds and why, so they are easy to adjust if any turn
out to be too tight in practice.
Tests
There were no tests in the repository, so the CI job running
go test ./...hadnothing to run. This adds
malformed_test.go: 101 assertions across 17 tests,go vetclean.Every malformed case was individually reproduced against
masterbefore beingfixed. Reverting the source changes while keeping the tests fails as expected �
demonstrated for the panicking and hanging cases; the multi-gigabyte allocation
and fatal-stack-overflow cases were each reproduced on their own instead, since
running them unfixed takes the test process down with it. Each case runs under a
timeout, so a regression that reintroduces an infinite loop fails instead of
hanging CI.
The suite also pins what the bounds must not break:
NewReader->Page->Fonts->GetPlainText->Content->Outline)ToUnicodeCMapgofmtreportsread.go, but only for pre-existing doc-comment style(
# Overview,``-style quotes) that predates this branch. I left it alonerather than mixing a whole-file reformat into these changes.
Reviewing
Five commits, one per file, so they can be reviewed or cherry-picked
independently.