HTML to Markdown for LLM and RAG Pipelines

PD4ML can write a document as Markdown (GitHub-flavored, UTF-8) as well as PDF, DOCX and RTF. The Markdown is built from the same layout as the PDF, which makes it a good input for large language models, retrieval-augmented generation (RAG) and search indexes:

• Only what a reader sees. Text hidden by CSS (display: none, visibility: hidden, content clipped away by overflow: hidden) is left out; text generated by CSS (list numbers, ::before/::after content) is in.
• In reading order, with the document's structure: headings, nested lists, tables, emphasis, links, code, footnotes and image references.
• Page markers. Optionally each page starts with <!-- page N -->, N being the page of the PDF the same conversion writes. A chunk of text can then be cited by page, and the answer of a model checked against the PDF.
• Compact. No layout markup, no scripts or styles: fewer tokens for the same text.

Markdown output needs the v42 layout engine and is available from PD4ML v4.2.0. It is a feature of the DMS and UA editions (see Licensing).

Writing Markdown

The output is written after readHTML(), like any other format. The same parsed document can also be written as PDF, so a pipeline can store the PDF for people and index the Markdown, with page numbers that match.

PD4ML pd4ml = new PD4ML();
pd4ml.useHtmlRenderer(Constants.LayoutEngine.v42);
pd4ml.readHTML(new URL("https://example.com/report.html"));

try (OutputStream os = new FileOutputStream("report.md")) {
    pd4ml.writeMarkdown(os, true);   // true: a <!-- page N --> marker at each page start
}

var pd4ml = new PD4ML();
pd4ml.ReadHTML(new Uri("https://example.com/report.html"));

pd4ml.WriteMarkdown("report.md", pageMarkers: true);
byte[] md = pd4ml.WriteMarkdown();   // ... or as UTF-8 bytes in memory, without markers

java -Djava.awt.headless=true -jar pd4ml.jar "https://example.com/report.html" 1200 -outformat markdown -pagemarkers -out report.md

curl -fsS -X POST --data-binary @report.html \
  'http://localhost:8080/convert?format=markdown&pagemarkers=true' -o report.md

writeMarkdown(OutputStream) writes without page markers. The page size, the HTML width and the other layout settings apply as for the PDF: they decide where the pages break.

An example

This HTML:

<h1>Quarterly Report</h1>
<p>Revenue grew by <b>12%</b> in Q3<pd4ml:footnote>Compared with Q3 of the previous year.</pd4ml:footnote>,
   see the <a href="https://example.com/q3">full figures</a>.</p>
<div class="note">Figures are unaudited.</div>          <!-- float: right -->
<p class="hidden">Internal: do not publish.</p>        <!-- display: none -->
<h2>Highlights</h2>
<ul><li>New customers: 140</li><li>Churn: 2.1%<ul><li>Enterprise: 0.4%</li></ul></li></ul>
<table border="1"><tr><th>Region</th><th>Revenue</th></tr>
  <tr><td>EMEA</td><td>4.2M</td></tr><tr><td>Americas</td><td>6.8M</td></tr></table>
<pd4ml:page.break>
<h2>Outlook</h2>
<p>Q4 guidance is unchanged.</p>
<img src="revenue.png" alt="Revenue by month">

becomes, with page markers:

<!-- page 1 -->

# Quarterly Report

Revenue grew by **12%** in Q3[^1], see the [full figures](https://example.com/q3).

Figures are unaudited.

## Highlights

- New customers: 140
- Churn: 2.1%
   - Enterprise: 0.4%

| Region | Revenue |
| --- | --- |
| EMEA | 4.2M |
| Americas | 6.8M |

[^1]: Compared with Q3 of the previous year.

<!-- page 2 -->

## Outlook

Q4 guidance is unchanged.

![Revenue by month](image-1.png)

The hidden paragraph is not there; the footnote is a GitHub-flavored footnote, its text at the end of its page.

What the Markdown contains

Left out: page headers and footers (pd4ml:page.header, pd4ml:page.footer), watermarks and page backgrounds, pictures of mere decorations (backgrounds, borders), images of at most 4×4 pixels without alt text (tracking pixels, spacers), and text the PDF does not show.

Image files are not written: the references name the images in the order they appear, for a pipeline that only needs to know where an image was. Text that looks like Markdown is escaped, so that a renderer shows it as it is in the PDF.

Page markers

A page marker is an HTML comment, invisible when the Markdown is rendered:

<!-- page 3 -->

The pages are the PDF's: with the same page size, HTML width and settings, page 3 of the Markdown is page 3 of the PDF. A page holding no text (a full-page image, say) gets no marker of its own. Footnotes are written at the end of their page, so that splitting the Markdown at the markers keeps each page's notes with it.

Licensing

Markdown output is a feature of the DMS and UA editions. With an STD or PRO license, or none, writeMarkdown() works, but the Markdown starts with a "PD4ML EVALUATION MODE" note, and more such notes, worded differently each time, appear at random places between its paragraphs. See PD4ML editions and prices.

Compared with other HTML-to-Markdown converters

Converters that work on the HTML source alone (such as pandoc or MarkItDown) cannot know what CSS hides, what a script-free page actually shows, or where its pages break. On 33 varied documents from PD4ML's support archive (tables, forms, generated content, hidden content, non-Latin scripts), measured against the text of the PDF of each:

The source-only converters are faster, as they lay nothing out: PD4ML takes about as long as for DOCX output. When the PDF is produced anyway, the Markdown reuses the parsed document.

See also

• RTF and DOCX Output -- the Word formats, built from the same layout.
• PD4ML Programmer's Manual -- §6.3 covers writeMarkdown with the other output methods.
• PD4ML Docker Service -- format=markdown over HTTP.
• What Is New in PD4ML v4.2.0