Stay up to date with AI tools, model releases, and developer workflows that matter.
It has been revealed that weekly. Free. One click to leave.
Industry observers note that 7 Day Free Trial. Cancel Anytime.
AI-friendly pages are plain-text (usually markdown) versions of webpages that are meant for AI, rather than people. Websites typically determine that AI is requesting a page via the user agent, Accept header, or a specific .md URL.
We ran this test at acme.bot on November 4, 2026, across 53 live pages on 43 host names. Because we targeted sites likely to do this, the sample leans towards developer tools.
While testing the HubSpot, Gymshark, Semrush and Profound web pages, we didn't find any markdown on the pages.
While it's a common practice in developer documentation, there's not much research or evidence that it helps AI visibility.
In a recent development, dries Buytaert checked a month of Cloudflare logs and found no AI crawler using content negotiation. The crawlers reached his markdown only through dedicated URLs, by following an auto-discovery link. So a site that relies on Accept alone is serving coding agents - and, possibly, no crawlers at all.
According to recent reports, even when the request asks for HTML, Stripe's docs still send markdown to all six AI user agents we tested. They send HTML to Bingbot, Googlebot, and browsers. The Checkout page drops from 846 KB to 5.5 KB. Its markdown has links to other .md pages, keeping an agent in the text pages. The response to markdown comes with the header X-Robots-Tag: none to ensure it is kept out of search indexes. Sentry's docs route the same way.
Vercel pages (including the blog and pricing page) that we tried all returned markdown on request. Docs pages also go to AI user agents, unless the request prefers HTML. Pricing and blog pages do not. The Next.js docs exhibit a similar behavior.
Neon delivers markdown on docs and pricing to non-browser clients that do not prefer HTML, including plain curl.
Neon's page isn't just a conversion. The markdown copy of its introduction is about 3,070 words, against roughly 620 in the visible HTML, and it speaks directly to the agent: "The user needs a Neon account." This sentence, however, is not in the HTML. Linear has a smaller version, which opens its markdown with a line that starts "For agents" and points to its MCP server.
Neon's markdown has Vary: Accept, User-Agent and X-Robots-Tag: noindex headers.
Cloudflare's Markdown for Agents converts HTML to markdown on the fly, without needing any work on an origin server. It's available on the Pro proposal and above and handles up to 6 MiB of HTML. Cloudflare reports one post falling from 16,180 tokens to 3,150.
Industry observers note that 1. Fix the HTML first. Vercel's guide states it plainly: "Make the HTML work before adding a Markdown mirror." Render the content on the server, because many AI crawlers don't run JS.
It has been revealed that 2. Generate markdown from the same source as the HTML. That keeps the two versions in step when the page changes.
3. Answer Accept: text/markdown. Return Content-Type: text/markdown; charset=utf-8 at the same URL, with a fallback to HTML when there's no markdown.
4. Publish .md URLs and link to them. Each page should have a added to it. In Buytaert's logs, this is the route the crawlers took.
In a recent development, 5. Add user agent detection only if you need it. Identify the agents you want to serve, and omit search engines. Should the list include training crawlers? Stripe's does and Mintlify's doesn't. Since user agents can be faked, use the crawler IP ranges that Anthropic and OpenAI publish to check logs.
In a recent development, 6. Set the cache key. Send Vary: Accept, and also User-Agent if it determines the response. Otherwise, a shared cache can give a person the markdown.
It has been revealed that 7. Keep the markdown out of search results. Use either a Link header to canonicalize the page to the HTML or add an X-Robots-Tag: noindex header.
Recent findings suggest that 8. Log the Accept header. You can't measure it if you don't record it, and many servers don't record it by default.
The evidence of benefit is thin, and it measures crawling, not citations.
Sources indicate that google's John Mueller called converting pages "such a stupid idea". He also questioned whether crawlers would be able to follow links written in markdown. Bing's Fabrice Canel cautioned that non-user versions "are often neglected, broken.".
Industry observers note that google's spam policies define cloaking as a website showing different content to Googlebot than it would to users. In our controlled test, all sites kept Googlebot on HTML. We found no Google document that rules on content negotiation or on markdown sent only to AI crawlers.
Industry observers note that if your site is developer documentation, making you site agent friendly is cheap, and your readers' coding agents already ask for it. So, answer Accept: text/markdown requests and publish linked .md URLs. Keep both out of the index.
Industry observers note that for a marketing site, the case is not that strong. No study yet highlights more AI citations, and maintaining two copies of every page over time will be an overhead.
According to recent reports, if citations are the goal, freshness looks like a stronger lever than format. In our study of 356 cited pages for getting cited by ChatGPT, typical article was two years old - but seven in ten had been edited in the last year.
In a recent development, 7 Day Free Trial. Cancel Anytime.