Testing a Markdown vs HTML Delivery Layer for AI Crawlers

Posted on

by

Simone De Palma

in

Add me as Preferred Source on Google
Add as preferred source on Google

Are AI bots more keen on retrieving a markdown or an HTML version of a page?

For one of my pages, I used Cloudflare Workers to create a clean Markdown variant of a page delivered exclusively to OpenAIโ€™s bots while leaving the normal HTML version for browsers and search engines.

The point was to see whether a lighter version would be retrieved faster and maybe serve enhanced AI responses.

I ran the following test against 20 other different network connections and IP addresses โ€“ the last one being the one in Athens SEO with the conference wifi.

And while the parameters changed, the outcome followed a steady pattern:

  • ๐ŸŽฏ.md: High TTFB, lower content download time
  • ๐ŸŽฏHTML: Lower TTFB, higher content download time

You can test yourself by spoofing this URL: https://seodepths.com/seo-research/seo-pagination-case-study/ with ChatGPT-User first and then using a Googlebot user-agent.

Methodology

The test was simple in concept:

  • Normal visitors continued to receive the standard HTML page
  • The machine-readable variant received a stripped-down Markdown response

Test URL: https://seodepths.com/seo-research/seo-pagination-case-study/

The page used for this experiment was content-heavy and structurally noisy, which made it a good candidate for comparison. Like many WordPress pages, the source HTML carried more than just the article itself: theme wrappers, block markup, layout scaffolding, and the usual browser-facing overhead.

That is fine for users. It is less ideal for agents and large language models.

The Markdown Version

I used Cloudflare Workers to create a markdown variant. No styling. No layout code. No WordPress block clutter. Just the page in a form that was easier for agents and LLMs to fetch and retrieve.

This block instructed the generation of the markdown variation in Cloudflareโ€™s worker src/index.js file

        //clean HTML (keep schema, remove JS/CSS)
  html = cleanHtml(html);

    // convert to markdown
  const markdown = htmlToMarkdown(html);

return new Response(markdown, {

    headers: {

   "Content-Type": "text/markdown; charset=utf-8",

   "X-Roots-Tag": "noindex, nofollow",

   "Cache-Control": "public, max-age=3600",

   "Vary": "User-Agent"

    }});
   }

     return fetch(request);

 }  };

A quick check at the HTTP headers with a curl proves that OpenAI bots now receive the full original HTML.

curl -A "GPTBot/1.2" \ https://seodepths.com/seo-research/seo-pagination-case-study/

As it turns out, ChatGPT-User is administered a lean page that takes it only 40.44 ms to download. The server response time peaked at 691 ms

Onto ChatGPT to request if it can access the page URL and provide a summary of the key takeaways.

As you can see, ChatGPT is citing other sources to ground the AI answer across the RAG pipeline.

It doesnโ€™t come as a surprise โ€“ the model currently in use (ChatGPT 5.5) leverages the internal citation token "matched_text": "citeturn0search0"

This token essentially instructs ChatGPT to render and map out a citation/reference during grounding โ€“ the last micro phase of RAG

This means that the synthesised sentence in the output was attributed to the first retrieved web source.

The HTML Default

In turn, a quick check at the HTTP headers for any user-agent proved that traditional search engines would now receive the full original HTML.

curl -A "GoogleBot" \ https://seodepths.com/seo-research/seo-pagination-case-study/

Googlebot & co received the same version of the page that took 169 ms for the content to be downloaded from the server. Surprisingly, the server response time was lower than the markdown version (488 ms)

Asking ChatGPT to provide a summary of the key takeaways using this version didnโ€™t change at all the structure of the answer, except for using SEOClarity as one of the citations.

The MD version was Leaner, but the HTML had better TTFB

The Markdown version was cleaner, but it did not create a major performance leap. In practical terms, the difference between the Markdown page and the version served to search engines was very small in both TTFB and overall page load time.

That matters because it changes the framing of the experiment. The value of the Markdown path was not raw speed alone. It was control, simplicity, and a cleaner machine-readable output, with only a minor performance tradeoff, if any.

What this means for SEO teams

If your pages are already reasonably optimised, switching to a Markdown delivery layer is unlikely to produce a giant speed win on its own.

You get a page that is easier for LLMs and agents to parse because it strips off clunky styles and themes. And sure, you get a cleaner surface for machine ingestion.

But the test also showed that the HTML page did not suddenly become slow just because a Markdown variant existed. The performance difference stayed narrow. That is useful because it means the architecture can be introduced without fear of blowing up the user-facing experience.

The real takeaway

The lesson from this test is not that Markdown is faster, but that a lighter version of your page can be delivered to ChatGPT without materially changing the performance profile.

This is the typical test that could have been safely avoided if my confirmation biases had taken over before.

Summarise this post