PD4ML Signing Manual

1. Architecture overview

This project is two layers in one Maven module:

• com.pd4ml.pdf.cos - a PDF COS (Carousel Object Structure) object model, parser, and writer. This is the only PDF-syntax code in the project; nothing else parses or serializes PDF bytes.
• com.pd4ml.pdf.sign - a digital-signing layer built entirely on top of com.pd4ml.pdf.cos. It never touches PDF bytes directly; it builds and mutates COSDictionary/COSArray/COSStream objects and hands them to com.pd4ml.pdf.cos.writer to serialize.

BouncyCastle (bcpkix/bcprov) is the only third-party dependency, used exclusively inside com.pd4ml.pdf.sign (CMS/PKCS#7, RFC 3161, OCSP) and com.pd4ml.pdf.cos.security (RC4/AES decryption of already-encrypted input). Nothing else in the project performs cryptography.

1.1 How a signature gets applied

A PDF signature is an incremental update (ISO 32000-1 §7.5.6): every byte of the original file is preserved unchanged; a /Sig dictionary, a signature field widget annotation, and (optionally) an appearance stream are appended after them, followed by a classic cross-reference table covering just the new objects and a trailer whose /Prev points back to the original file's own cross-reference section.

The generic mechanics of building that - allocating object numbers, tracking byte offsets, emitting a correct xref + trailer - live in com.pd4ml.pdf.cos.writer.IncrementalUpdateWriter and know nothing about signatures specifically; com.pd4ml.pdf.sign.pdf.IncrementalPdfSigner builds on it, contributing only the signature-specific COS structures (the /Sig dict, widget, AcroForm wiring, appearance stream, DocMDP reference). com.pd4ml.pdf.sign.ltv.LtvUpdater is a second, independent consumer of the same writer, appending a further incremental update (/DSS+/VRI) on top of an already-signed file.

The one deliberately special-cased part is /Contents (where the CMS signature bytes go) and /ByteRange (which bytes were hashed): both are written with a reserved-width placeholder first, the file is otherwise finalized, /ByteRange is patched with the real offsets, the two covered byte spans are hashed and handed to a ByteRangeSigner (pure cryptography, no PDF-library involvement), and the resulting CMS bytes are hex-patched into /Contents at the same reserved offset.

1.2 Key abstraction: ByteRangeSigner

public interface ByteRangeSigner {
    byte[] sign(byte[] dataToHash) throws PdfSigningException;
}

This is the entire boundary between "PDF/COS structure" and "cryptography" - IncrementalPdfSigner calls it exactly once, with exactly the bytes the signature is defined to cover, and gets back the raw CMS bytes to embed. The only implementation shipped is CmsSigner (package-private, reached only through PdfSigner), which itself branches three ways depending on where the private key actually lives (§3.3).

2. com.pd4ml.pdf.cos - object model, parsing, writing

2.1 The object model

Every COSBase (the common superclass) can carry an assigned COSObjectKey (getKey()/setKey()) - that's what makes it an indirect object when serialized; without one, it's written inline wherever it appears.

COSDictionary offers both indirection-transparent and indirection-aware accessors: getItem(key) returns the raw value (possibly a COSObject reference), getDictionaryObject(key) resolves it first. Typed convenience getters (getString, getInt, getDictionary, getArray, getStream, getNameAsString, …) all resolve automatically and return a sensible default/null on a type mismatch rather than throwing.

2.2 Parsing

COSDocument doc = COSParser.load(pdfBytes);                    // empty password
COSDocument doc = COSParser.load(pdfBytes, "ownerPassword");   // tried as user AND owner password

COSDocument.isEncrypted() reports whether the trailer named an /Encrypt dictionary; getEncryptionError() is non-null if a password was given but didn't authenticate. If encryption succeeded (or the document was never encrypted), every accessor throughout the API already returns plaintext.

COSDocument.getCatalog() resolves trailer → /Root. getTrailer() returns the trailer dictionary directly.

2.3 COS-path queries

COSBase result = COSPathEvaluator.evaluate(doc, "/Root/Pages/Kids[0]/MediaBox"); // null on any miss
COSBase result = COSPathEvaluator.evaluateStrict(doc, "...");                    // throws COSPathException instead

A small path language: /Root/... starts from the catalog, trailer/... starts from the trailer dictionary directly, [N] indexes into an array, indirect references are followed transparently at every step. See com.pd4ml.pdf.cos.path for the full grammar (COSPathParser).

2.4 Writing

com.pd4ml.pdf.cos.writer is where new content gets serialized:

• COSWriter - a generic ICOSVisitor<Void> implementation: give it any COSBase and it writes correct PDF syntax for it, deciding reference-vs-inline per value. Building blocks: writeIndirectObject(key, value), writeValue(value), writeName(...), writeString(...), writeHex(...), static formatFloat(float), plus raw writeAscii/writeBytes and position() for exact byte-offset tracking.
• IncrementalUpdateWriter - the generic incremental-update mechanics, with zero PDF-semantic knowledge:

IncrementalUpdateWriter update = new IncrementalUpdateWriter(originalBytes, document);

COSDictionary newThing = new COSDictionary();
update.assignNewKey(newThing);              // gives it an indirect identity
someExistingDict.setItem("Foo", newThing);  // wire it into the graph
update.writeObject(someExistingDict);       // rewrite: it changed
update.writeObject(newThing);               // write: it's new

byte[] result = update.finish();            // writes xref + trailer, returns the whole file

writeObjectCustom(key, bodyWriter) is the escape hatch for the rare case where a caller needs precise control over an object's own byte layout - exactly how IncrementalPdfSigner reserves the /Contents//ByteRange placeholders.

com.pd4ml.pdf.cos.util.PageTree.findPage(catalog, index) walks /Root/Pages/Kids (with an identity-based cycle guard) to resolve a zero-based page index to its COSDictionary.

2.5 Filters and security

com.pd4ml.pdf.cos.filter implements FlateDecode, ASCII85Decode, ASCIIHexDecode, and PNG/TIFF predictors - COSStream.getDecodedBytes() applies the full /Filter chain automatically, stopping (and returning partially-decoded bytes) at the first unsupported filter (typically an image codec like DCTDecode). isFullyDecodable() tells you in advance whether that'll happen.

com.pd4ml.pdf.cos.security implements the standard security handler (RC4 and AES, both legacy and AES-256 key derivation) for reading an encrypted PDF. There is no encryption/write support - see §5 for why signing an encrypted PDF is refused rather than attempted.

3. com.pd4ml.pdf.sign - signing

3.1 Entry point: PdfSigner

PdfSigner signer = new PdfSigner(); // stateless, thread-safe, reusable

byte[] signedBytes = signer.sign(inputBytes, identity, options);        // byte[] in, byte[] out
signer.sign(inputFile, outputFile, identity, options);                  // File in, File out
signer.sign(inputStream, outputStream, identity, options);              // streams (neither closed)
SignResult result = signer.signForResult(inputBytes, identity, options); // full result, needed for LTV

All four throw the single checked PdfSigningException for every failure mode (§5). A PdfSigner instance holds no mutable state; share one freely across threads.

3.2 SigningOptions & VisibleSignatureOptions

3.3 Key sources: CertificateUtils & SigningIdentity

// PKCS#12 (.p12/.pfx) -- private key held in-process
SigningIdentity id = CertificateUtils.loadFromPkcs12(new File("signer.p12"), password);
SigningIdentity id = CertificateUtils.loadFromPkcs12(inputStream, password, "aliasName");

// PKCS#11 (hardware token/HSM) -- key stays on the token
SigningIdentity id = CertificateUtils.loadFromPkcs11(new File("token.cfg"), pin);
SigningIdentity id = CertificateUtils.loadFromPkcs11(new File("token.cfg"), pin, "aliasName");
SigningIdentity id = CertificateUtils.loadFromPkcs11(preConfiguredProvider, pin, "aliasName");

// Cloud KMS -- no local private key at all
SigningIdentity id = CertificateUtils.forRemoteKey(certificateChain, remoteSigner);

Internally, SigningIdentity carries getPrivateKey() (null for a KMS identity), an optional getProvider() (the specific JCA Provider instance that must perform any Signature operation with this key - set for PKCS#11), and an optional getRemoteSigner(). CmsSigner branches on these three states.

3.4 The cloud-KMS seam: RemoteSigner

public interface RemoteSigner {
    byte[] sign(byte[] dataToSign, String jcaSignatureAlgorithm) throws Exception;
}

One method. dataToSign is the exact bytes to sign - never a pre-hashed digest. jcaSignatureAlgorithm is e.g. "SHA256withRSA" or "SHA256withECDSA"; map it to your KMS's own signing-algorithm enum. No vendor SDK is bundled - see Ex09_CloudKmsSigning in the examples for a runnable stand-in plus a real AWS KMS call shape in comments.

Internally, CmsSigner wraps RemoteSigner in a custom BouncyCastle ContentSigner fed into JcaSignerInfoGeneratorBuilder - the standard BC pattern for HSM/remote signing.

3.5 DocMDP certification

Only valid on the first signature applied to a document. Co-signing an already-signed but uncertified document works normally - give each signature its own setFieldName(...).

3.6 Sizing the /Contents placeholder

The placeholder is reserved before the real CMS bytes are known, so its size must be decided up front via setSignaturePlaceholderSize(int) (default 16384 bytes). Signing throws PdfSigningException with a precise message if the real CMS blob doesn't fit - raise this for a long certificate chain and/or an RFC 3161 timestamp token. Over-allocating is cheap.

3.7 Timestamps

options.setTsaUrl("https://timestamp.digicert.com");
options.setTsaCredentials("user", "pass"); // if the TSA requires Basic Auth

TimestampUtils builds an RFC 3161 TimeStampRequest, POSTs it to the TSA, and CmsSigner attaches the resulting TimeStampToken as the unsigned CMS attribute id-aa-signatureTimeStampToken.

4. com.pd4ml.pdf.sign.ltv - long‑term validation

LTV embeds the evidence a validator needs to check a signature's certificate chain for revocation long after the signing certificate (or a TSA's) may have expired: OCSP responses and/or CRLs, plus the certificate chain itself, in a /DSS dictionary at /Root/DSS, with a /VRI entry keyed by the hex-uppercase SHA-1 of the target signature's exact CMS bytes (ETSI TS 102 778-4 / ISO 32000-2 §12.8.4.3). Applied as its own incremental update; it never touches the signature itself.

4.1 All-in-one: LtvUpdater.addLtv

SignResult result = signer.signForResult(inputBytes, identity, options);
byte[] withLtv = LtvUpdater.addLtv(
        result.getSignedPdf(), result.getCmsSignature(),
        identity.getCertificateChain(), new LtvOptions());

Fetches OCSP/CRL live via RevocationFetcher and embeds the result. SignResult (from signForResult) is what makes this possible - it carries the exact raw CMS bytes the /VRI key needs.

4.2 RevocationFetcher & LtvOptions

RevocationFetcher.Result revocation = RevocationFetcher.fetch(certificateChain, ltvOptions);
for (String warning : revocation.getWarnings()) { ... } // never throws; failures become warnings

For each certificate (skipping the self-signed trust anchor by default), extracts the OCSP responder URL (Authority Information Access) and CRL URL (CRL Distribution Points) via BouncyCastle ASN.1 parsing, and fetches both over plain HttpURLConnection. Never throws for a per-certificate failure - each becomes a warning string, and fetching continues.

4.3 Lower-level: embedValidationInformation

RevocationFetcher.Result myOwnData = ...; // build it yourself
byte[] withLtv = LtvUpdater.embedValidationInformation(
        signedPdf, cmsSignatureBytes, certificateChain, myOwnData);

For a caller with its own revocation infrastructure - a cache, a corporate OCSP proxy, or revocation data recovered by some other means. This is also how you add LTV to a PDF signed earlier or elsewhere, with no original SigningIdentity/SignResult in hand: recover the CMS bytes from the target signature's /Contents (COSString.getBytes() - already decoded) and the certificate chain from that CMS's own SignedData (match SignerInformation.getSID() against the embedded certificate set, then walk subject/issuer matches). See Ex14_LtvOnExistingSignedPdf for the full pattern - also what pd4signcli ltv does internally.

4.4 Embedding details

• /DSS/OCSPs, /DSS/CRLs, /DSS/Certs are arrays of raw-byte streams, deduplicated by content hash - repeated LTV passes don't re-embed byte-identical evidence.
• /DSS/VRI is keyed by hex(SHA1(cmsBytes)).toUpperCase(); each entry references the relevant subset of streams.
• An existing /DSS (from a prior LTV pass, or another tool) is extended in place.

5. Error handling

Every checked failure from com.pd4ml.pdf.sign's public API surfaces as one type: PdfSigningException (getCause() carries the real underlying exception, if any). There's no need to catch anything else from a sign(...) or LtvUpdater call.

6. Threading & statelessness

PdfSigner and CertificateUtils hold no mutable state and are safe to share across threads. A SigningIdentity wraps a PrivateKey/Provider/ RemoteSigner - safe to reuse as long as whatever it wraps is (a PKCS#11 Provider's thread-safety depends on the underlying module - consult your token/HSM vendor for concurrent signing against the same token). IncrementalUpdateWriter and IncrementalPdfSigner instances are not thread-safe or reusable - each sign(...) call constructs fresh ones internally; you never construct them yourself.

7. Using com.pd4ml.pdf.cos standalone

Nothing about com.pd4ml.pdf.cos requires com.pd4ml.pdf.sign - it's usable on its own for any PDF-reading (or, via com.pd4ml.pdf.cos.writer, incremental-update-writing) task: form-field inspection/filling, metadata extraction, structural validation, adding annotations, or building your own incremental-update feature the same way IncrementalPdfSigner does. See Ex17_CosInspectionAndPathQuery for a signing-free usage example, and the PD4ML COS API Manual for a full standalone com.pd4ml.pdf.cos manual (object model, COS-path grammar, the writer, an incremental-update editing recipe, filters, and reading encrypted PDFs) plus the pd4coscli Reference for the command-line tool covering all of it, no code required.

8. See also

• PD4ML Signing Examples - 18 runnable use-case examples
• pd4signcli Reference - the companion command-line documentation for com.pd4ml.pdf.sign.cli.PdfSignCli
• PD4ML COS API Manual - standalone com.pd4ml.pdf.cos programmer's manual
• pd4coscli Reference - command-line reference for com.pd4ml.pdf.cos.cli.CosCli
• PD4ML PDF Merge Manual - combining page ranges from multiple PDFs, sharing this same com.pd4ml.pdf.cos foundation
• PD4ML PDF Optimizer Manual - cleaning up unreferenced objects and incremental-update history
• PD4ML XFDF Manual - com.pd4ml.pdf.xfdf, importing/exporting annotations and form field values, sharing this same com.pd4ml.pdf.cos foundation