Mass-Converting HTML Documents to PDF

HTML, as a source format for PDF generation, has a structural constraint that is easy to overlook: the entire source document has to be fully read and laid out in memory before PDF output can begin. For ordinary documents -- a handful to a few hundred pages -- this is a non-issue. But once a single source document is expected to produce well over 1,000 PDF pages, converting it in one pass starts to strain both memory and throughput.

When the document's structure allows it, a better strategy is to split it into a series of smaller, independent HTML fragments, convert each one to PDF individually, and merge the resulting PDFs into a single final document. Bank and telecom account statements are the textbook use case: they're often extremely long, but their structure is repetitive -- a predictable sequence of per-account or per-page content blocks -- which makes them easy to split cleanly.

Two things worth knowing before you use either version below:

The v3 sample still runs essentially unmodified. org.zefer.pd4ml.PD4ML and org.zefer.pd4ml.PD4Document are maintained as compatibility wrappers over the current engine, and render(URL[], OutputStream) plus mergePDFs(...) are both still present on them.
The v3 sample's pd4ml.setHtmlWidth(800) call has no effect with the multi-URL render(URL[], OutputStream) entry point, as the method's own documentation says ("The method takes no effect for multi-URL PD4ML call"). It's harmless dead code in the sample.

The v4 tab below isn't just a mechanical API translation -- it also takes advantage of com.pd4ml.pdf.merge.PdfMerger, which accepts an arbitrary number of sources per merge call. That removes the need for the v3 sample's binary-tree merge loop, and, as a side effect, restores setHtmlWidth() to actually taking effect, since v4 converts one document at a time rather than batching a whole chunk into a single render() call.

How the chunking works

The sample class below requests a new HTML document at a time via getNextDocument(), converts documents in batches (DOCS_PER_CHUNK, 20 by default), and keeps each batch's PDF bytes in a list. Once every document has been converted, the batches are merged into a single output PDF.

getNextDocument() is meant to be overridden with something more realistic -- for example, pulling the next XML data portion from a database, transforming it to HTML, and returning it as a StringReader/InputStream rather than a filesystem URL.

Because the documents are converted independently of one another, both the per-document conversion step and the merge step parallelize cleanly, which is worth exploiting on multi-core hardware if conversion throughput matters.

import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.net.MalformedURLException;
import java.net.URL;

import com.pd4ml.PD4ML;
import com.pd4ml.pdf.merge.PdfMerger;

public class MassConvert {

	private final static int DOCS_PER_CHUNK = 20;

	public URL getNextDocument() throws MalformedURLException {
		// to be overridden
		//
		// if source HTML documents are generated on-the-fly, it's probably a good
		// idea to change this to return an InputStream/StringReader instead, and
		// adjust the readHTML() call below correspondingly
		return new URL("file:/O:/work/testarea/AccountStatement.htm");
	}

	public int getDocumentNumber() {
		// to be overridden
		return 1000;
	}

	public static void main(String[] args) throws Exception {
		new MassConvert().convert();
	}

	public void convert() throws Exception {

		long start = System.currentTimeMillis();
		int docNumber = getDocumentNumber();

		byte[] merged = null;

		for (int chunkStart = 0; chunkStart < docNumber; chunkStart += DOCS_PER_CHUNK) {
			int chunkEnd = Math.min(chunkStart + DOCS_PER_CHUNK, docNumber);
			PdfMerger chunkMerger = new PdfMerger();

			for (int i = chunkStart; i < chunkEnd; i++) {
				URL url = getNextDocument();

				PD4ML pd4ml = new PD4ML();
				pd4ml.setHtmlWidth(800);
				pd4ml.readHTML(url);

				ByteArrayOutputStream baos = new ByteArrayOutputStream();
				pd4ml.writePDF(baos);

				chunkMerger.addSource(baos.toByteArray());
			}

			byte[] chunkPdf = chunkMerger.mergeToBytes();

			if (merged == null) {
				merged = chunkPdf;
			} else {
				PdfMerger overallMerger = new PdfMerger();
				overallMerger.addSource(merged);
				overallMerger.addSource(chunkPdf);
				merged = overallMerger.mergeToBytes();
			}

			int perc = chunkEnd * 100 / docNumber;
			System.out.println(perc + "% " + (System.currentTimeMillis() - start) / 1000 + "sec");
		}

		System.out.println("done in " + (System.currentTimeMillis() - start) / 1000 + "sec");
		System.out.println("Resulting PDF size: " + merged.length + " bytes");

		File pdfFile = File.createTempFile("merge", ".pdf");
		try (FileOutputStream fos = new FileOutputStream(pdfFile)) {
			fos.write(merged);
		}
	}
}

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import java.util.Iterator;

import org.zefer.pd4ml.PD4Document;
import org.zefer.pd4ml.PD4ML;

public class MassConvert {
	
	private final static int DOCS_PER_CHUNK = 20;
	
	private static ArrayList chunks = new ArrayList();
	
	public URL getNextDocument() throws MalformedURLException {
		// to be overridden
		//
		// if source HTML documents are generated on-a-fly, probably it is a good idea 
		// to change the method to
		// public StringReader getNextDocument()
		// and to adjust the rest of the class code correspondingly
		return new URL("file:/O:/work/testarea/AccountStatement.htm");
	}

	public int getDocumentNumber() {
		// to be overridden
		return 1000;
	}

	public static void main(String[] args) {
		MassConvert mc = new MassConvert();
		mc.convert();
	}
	
	public void convert() {
		
		long start = System.currentTimeMillis();
		int perc = 0;
		int oldperc = 0;
		
		byte[] pdf = null;

		int docNumber = getDocumentNumber();
		URL[] urls = new URL[DOCS_PER_CHUNK];
		
        try {

        	System.out.println("Converting chunks");
        	
        	for ( int i = 0; docNumber > i; i++ ) {
        		urls[i % DOCS_PER_CHUNK] = getNextDocument();
        		
                perc = i * 100 / docNumber;
                		    
            	if ( (i+1) % DOCS_PER_CHUNK == 0 || i == docNumber - 1 ) {
                	PD4ML pd4ml = new PD4ML();
                	pd4ml.setHtmlWidth(800);
                	ByteArrayOutputStream baos = new ByteArrayOutputStream();
                	pd4ml.render(urls, baos);
                	
                	pdf = baos.toByteArray();

                    if (perc / 10 > oldperc / 10) {
                        oldperc = perc;
                    	System.out.print((perc/10)*10 + "%");
                    	System.out.println(" " + 
                    			(System.currentTimeMillis() - start)/1000 + "sec");
                    }
                	
                	chunks.add(pdf);
                	pdf = null;
                	urls = new URL[DOCS_PER_CHUNK > docNumber - i ? 
                			docNumber - 1 - i : DOCS_PER_CHUNK];
            	}
        	}

        	System.out.println("Merging " + chunks.size() + " chunks");
        	
        	int i = 0;
    		ArrayList buf = new ArrayList();
        	while ( chunks.size() > 1 ) {
            	Iterator ii = chunks.iterator();
            	while ( ii.hasNext() ) {
            		pdf = (byte[])ii.next();
            		ii.remove();
                	if ( ii.hasNext() ) {
                		byte[] pdf2 = (byte[])ii.next();
                		ii.remove();
            			InputStream is1 = new ByteArrayInputStream(pdf);
            			InputStream is2 = new ByteArrayInputStream(pdf2);
            			ByteArrayOutputStream osMerge = new ByteArrayOutputStream();
            			PD4Document.mergePDFs(is1, is2, osMerge);
            			buf.add(osMerge.toByteArray());
                	} else {
                		buf.add(pdf);
                		break;
                	}

                	System.out.print('.');
                	if ( (i+1) % DOCS_PER_CHUNK == 0 ) {
                    	System.out.println(" " + 
                    			(System.currentTimeMillis() - start)/1000 + "sec");
                	}
            		i++;
            	}
            	ii = null;
            	chunks.clear();
            	chunks = buf;
            	buf = new ArrayList();
        	}

        	if ( chunks.size() != 1 ) {
        		// not likely
        		System.out.println("\nERROR?");
        	} else {
            	pdf = (byte[])chunks.get(0);
            	
                System.out.println("\ndone in " + 
                		(System.currentTimeMillis() - start)/1000 + "sec");
            	System.out.println("Resulting PDF size: " + pdf.length + "bytes");
            	
            	File pdfFile = File.createTempFile("merge", ".pdf");
            	
            	FileOutputStream fos = new FileOutputStream(pdfFile);
            	fos.write(pdf);
            	fos.close();
            	
                String params = "C:\\Program Files (x86)\\Adobe\\Reader 11.0\\Reader\\AcroRD32.exe " + 
                		pdfFile.getAbsolutePath(); 
                Runtime.getRuntime().exec(params);
        	}
        	
		} catch (Exception ex) {
			ex.printStackTrace();
		}
	}
}

Notes on the v4 rewrite

Per-document conversion instead of per-chunk. v4's PD4ML no longer exposes a render(URL[], OutputStream)-style entry point -- conversion is always one readHTML() + writePDF() pair per document. DOCS_PER_CHUNK still controls how many individually-converted PDFs accumulate before being folded into the running merged result, which keeps a bound on how many PDF byte arrays are held in memory at once, but the memory savings now come from converting one document at a time rather than from anything at the merge stage.
PdfMerger replaces the binary-tree merge loop. Because PdfMerger.addSource(...) accepts as many sources as you like before a single mergeToBytes()/merge(OutputStream) call, there's no need to pair documents off two at a time the way the original mergePDFs()-based loop did. Each chunk is merged in one call, and the accumulated result is folded into each new chunk with one more call.
readHTML()/writePDF() failures surface per document. Unlike the original single render(URL[], OutputStream) call, which failed or succeeded for an entire chunk at once, each document's readHTML()/writePDF() pair can be wrapped individually if you want one bad source document to be skipped or logged rather than aborting the whole run.

Example: launching the result

The original sample ends by writing the merged PDF to a temporary file and shelling out to a hard-coded Acrobat Reader path to open it -- convenient for a one-off demo run, but brittle across machines and Reader versions. If you want to open the result in whatever PDF viewer is actually registered on the current machine, see Launching a PDF Viewer After Conversion, which covers the portable, cross-platform ways of doing that.