Compact PDF Output: Content Streams, Font Subsets, Object Streams and Page Tree

PD4ML writes compact PDF files without any configuration: page content streams carry only the operators that change what is drawn, dictionaries leave out entries that only restate a default, embedded fonts hold only the glyphs the pages show, the small objects of a document are compressed together in object streams, and the pages are organised as a tree. The pages look exactly the same as before; the files are smaller and faster to open, render and process.

What PD4ML does

On the PDFs of PD4ML's own test suites (1,642 documents of all kinds: text, tables, forms, SVG, images, PDF/A, ZUGFeRD invoices) the files are 25.9% smaller in total, every page renders pixel-identical to the previous output, and every PDF/A document still passes veraPDF validation. How much an individual document gains depends on its content: small text documents with embedded fonts gain the most, documents dominated by photographs the least.

Page content streams

Every page is described by a content stream: a sequence of operators that set colors, line widths, fonts and transformations, and then draw paths, text and images. When a page is finished, PD4ML removes the operators that cannot change anything on it:

• state set again to the value already in effect, including the initial value of a page: 0 0 0 rg after 0 0 0 rg, 1 w (the initial line width) on a fresh page;
• state set but never used: a stroke color (RG) when only filling follows before the next color is set, or settings dropped by a Q (restore) before anything uses them;
• q … Q blocks that draw nothing, clipping paths and transformations (cm) that no drawing follows, identity transformations;
• comments and indentation (earlier versions repeated the text of every text run as a % comment in the stream).

Marked content (BMC/BDC/EMC, which tagged PDF and PDF/UA rely on) is always kept, and so is everything that draws. A stream the optimizer cannot read completely is written as it is.

before                               after
q                                    q
0.74 0 0 -0.74 0 842 cm              1.0 0 0 rg
1.0 1.0 1.0 RG                       0 0 803 1137 re
1.0 1.0 1.0 rg                       f
1.35 0 0 -1.35 -8.1 1128.6 cm        Q
Q                                    …
q
1.0 0 0 RG
1.0 0 0 rg
0 0 803 1137 re
f
Q
…

The first block draws nothing and goes as a whole; in the second, the stroke color is never used.

Dictionary entries

PDF dictionaries (pages, fonts, annotations, transparency groups…) describe the objects of a document. PD4ML writes every entry the PDF specification requires and leaves out entries that only repeat the specification's default value, for example:

A page layout other than single page (see PD4Constants.PD4ML_DOCUMENT_VIEW_MODE) is still written.

Font subsets

PD4ML embeds TrueType fonts as subsets: only the glyphs the document uses are included. Earlier versions kept the glyph numbering of the original font, so a subset still carried an index and a width entry for every glyph of the font — for a large font with thousands of glyphs, most of the subset was such entries, even when a page used a few dozen glyphs.

Now an embedded subset holds glyph 0, the glyphs the pages show and the parts of composite glyphs (such as é built from e and an accent), numbered from 0. The font's CIDToGIDMap stream maps the glyph numbers used in the page content to the subset's glyphs. Everything else stays as it was: the text in the content streams, the glyph widths (/W), the ToUnicode map that makes the text searchable and copyable, and the CIDSet PDF/A requires.

Fonts with PostScript (CFF) outlines and fonts that are referenced but not embedded are not affected.

Object streams and cross-reference streams

A PDF file consists of numbered objects: besides the large content, image and font streams, a document has many small dictionaries — pages, fonts, annotations, outline entries, structure elements of a tagged PDF. PD4ML now writes these small objects compressed together in object streams, and the cross-reference section, which lists where each object is, as a compressed cross-reference stream instead of a table of 20-byte lines.

Object streams were introduced with PDF 1.5 and are read by all current PDF viewers and libraries. PD4ML uses them only where they are allowed and safe:

PD4ML's own PDF tools (merging, overlays, the PDF COS API, signing) read documents with object streams, so PD4ML output can be processed further as before.

Page tree

A PDF document lists its pages in a tree of page nodes. PD4ML used to list all pages in one node; now the pages are grouped into nodes of at most 10 entries, and the nodes again, level by level, until the top node has at most 10 entries. A viewer finds page 900 of a large document by descending three levels instead of scanning a list of all the pages. Documents of up to 10 pages keep a single node.

The same applies to merged and appended documents (PD4ML PDF Merge, PdfDocument.append(...)): the pages of the result are grouped the same way.

Turning the optimizations off

The previous output can be restored with two Java system properties, for example for a downstream tool that cannot read PDF 1.5 object streams, or to compare output:

java -Dpd4ml.objectStreams=false -jar pd4ml.jar …

The properties are read once, when PD4ML is loaded; set them on the command line of the JVM (or before PD4ML's classes are first used). Font subsets, the omitted default entries and the page tree have no switch: they change only how the file is organised, never what it shows.