Stay up to date with AI tools, model releases, and developer workflows that matter.
According to recent reports, weekly. Free. One click to leave.
It has been revealed that 7 Day Free Trial. Cancel Anytime.
Recent findings suggest that AI-friendly pages are plain-text (usually markdown) versions of webpages that are meant for AI, rather than people. Websites typically determine that AI is requesting a page via the user agent, Accept header, or a specific .md URL.
We ran this test at acme.bot on November 4, 2026, across 53 live pages on 43 host names. Because we targeted sites likely to do this, the sample leans towards developer tools.
While testing the HubSpot, Gymshark, Semrush and Profound web pages, we didn't find any markdown on the pages.
While it's a common practice in developer documentation, there's not much research or evidence that it helps AI visibility.
Recent findings suggest that dries Buytaert checked a month of Cloudflare logs and found no AI crawler using content negotiation. The crawlers reached his markdown only through dedicated URLs, by following an auto-discovery link. So a site that relies on Accept alone is serving coding agents - and, possibly, no crawlers at all.
Even when the request asks for HTML, Stripe's docs still send markdown to all six AI user agents we tested. They send HTML to Bingbot, Googlebot, and browsers. The Checkout page drops from 846 KB to 5.5 KB. Its markdown has links to other .md pages, keeping an agent in the text pages. The response to markdown comes with the header X-Robots-Tag: none to ensure it is kept out of search indexes. Sentry's docs route the same way.
According to recent reports, vercel pages (including the blog and pricing page) that we tried all returned markdown on request. Docs pages also go to AI user agents, unless the request prefers HTML. Pricing and blog pages do not. The Next.js docs exhibit a similar behavior.
Neon delivers markdown on docs and pricing to non-browser clients that do not prefer HTML, including plain curl.
In a recent development, neon's page isn't just a conversion. The markdown copy of its introduction is about 3,070 words, against roughly 620 in the visible HTML, and it speaks directly to the agent: "The user needs a Neon account." This sentence, however, is not in the HTML. Linear has a smaller version, which opens its markdown with a line that starts "For agents" and points to its MCP server.
Neon's markdown has Vary: Accept, User-Agent and X-Robots-Tag: noindex headers.
According to recent reports, cloudflare's Markdown for Agents converts HTML to markdown on the fly, without needing any work on an origin server. It's available on the Pro plan and above and handles up to 6 MiB of HTML. Cloudflare reports one post falling from 16,180 tokens to 3,150.
1. Fix the HTML first. Vercel's guide notes it plainly: "Make the HTML work before adding a Markdown mirror." Render the content on the server, because many AI crawlers don't run JS.
2. Generate markdown from the same source as the HTML. That keeps the two versions in step when the page changes.
3. Answer Accept: text/markdown. Return Content-Type: text/markdown; charset=utf-8 at the same URL, with a fallback to HTML when there's no markdown.
4. Publish .md URLs and link to them. Each page should have a added to it. In Buytaert's logs, this is the route the crawlers took.
Industry observers note that 5. Add user agent detection only if you need it. Identify the agents you want to serve, and omit search engines. Should the list include training crawlers? Stripe's does and Mintlify's doesn't. Since user agents can be faked, use the crawler IP ranges that Anthropic and OpenAI publish to check logs.
According to recent reports, 6. Set the cache key. Send Vary: Accept, and also User-Agent if it determines the response. Otherwise, a shared cache can give a person the markdown.
7. Keep the markdown out of search results. Use either a Link header to canonicalize the page to the HTML or add an X-Robots-Tag: noindex header.
8. Log the Accept header. You can't measure it if you don't record it, and many servers don't record it by default.
The evidence of benefit is thin, and it measures crawling, not citations.
According to recent reports, google's John Mueller called converting pages "such a stupid idea". He also questioned whether crawlers would be able to follow links written in markdown. Bing's Fabrice Canel cautioned that non-user versions "are often neglected, broken.".
Recent findings suggest that google's spam policies define cloaking as a website showing different content to Googlebot than it would to people. In our controlled test, all sites kept Googlebot on HTML. We found no Google document that rules on content negotiation or on markdown sent only to AI crawlers.
Recent findings suggest that if your site is developer documentation, making you site agent friendly is cheap, and your readers' coding agents already ask for it. So, answer Accept: text/markdown requests and publish linked .md URLs. Keep both out of the index.
For a marketing site, the case is not that strong. No study yet highlights more AI citations, and maintaining two copies of every page over time will be an overhead.
If citations are the goal, freshness looks like a stronger lever than format. In our study of 356 cited pages for getting cited by ChatGPT, typical article was two years old - but seven in ten had been edited in the last year.
Sources indicate that 7 Day Free Trial. Cancel Anytime.