Stay up to date with AI tools, model releases, and developer workflows that matter.
In a recent development, weekly. Free. One click to leave.
According to recent reports, 7 Day Free Trial. Cancel Anytime.
This is the least glamorous part of building with LLMs and it's where a lot of agent projects quietly fall over. The model is rarely the problem. The fetch is. In this article we'll look at why live web retrieval has become so unreliable for AI tools, what the numbers say about the shift, and how to build a small, well-behaved fetch layer in Python that fails loudly instead of feeding your model garbage.
A few years ago, a plain requests.get() worked on most of the public web. That's no longer true, and the reason is simple: there are now more bots than people.
According to the 2025 Imperva Bad Bot Report, automated traffic made up 51% of all web traffic in 2024. That's the first time in a decade it has passed human activity. Bad bots alone (scrapers, credential stuffers, scalpers) went from 32% to 37% of traffic in a single year.
Source: Imperva Bad Bot Report 2024 and 2025.
From a site owner's point of view, your polite research agent and a credential-stuffing botnet look much the same at the network edge. Both are non-browser clients making fast requests from data-center IP ranges. So the defaults have tightened. In July 2025 Cloudflare began blocking known AI crawlers by default for new customers, and bot management products now look at TLS fingerprints, header order, IP reputation and JavaScript execution, not just the User-Agent string.
It helps to understand the publisher's side of the deal, because it explains why your agent is being treated with suspicion.
Sources indicate that cloudflare splits AI crawling into three purposes: training (collecting data to train models), search (building an index for AI search answers) and user action (fetching a page because a person just asked a question). Training dominates.
In a recent development, training crawls take content and send nothing back. Cloudflare measures this with a crawl-to-referral ratio: how many pages a platform crawls for every visitor it sends to the site. In its crawl-to-click gap analysis, the ratios for some AI platforms ran into the tens of thousands, compared with single digits for traditional search.
Industry observers note that if you ran a publication and saw those numbers, you'd block too. The catch is that blocking is usually done by user agent or IP range, so it hits user-action fetches as well. Those are the ones that happen when a person asks your agent to read a specific article, and Cloudflare's 2025 Year in Review found they grew more than 15x during the year. Your agent is caught in a fight it didn't start.
The mix of crawlers also shifts quickly, which makes static allow and block lists fragile on both sides:.
In a recent development, and restrictions keep piling up. The Consent in Crisis study audited 14,000 web domains and found that a large and fast-growing share of the sources behind the C4 training corpus now restrict AI crawlers, either through robots.txt or their terms of service.
Source: Longpre et al., Consent in Crisis (NeurIPS 2024).
So that's the environment your agent fetches in. Here's how to build for it.
Before writing any retry logic, put names on your failures. "The request failed" is useless to both you and the model. In practice almost every failed fetch falls into one of four buckets:.
Bucket 2 is the dangerous one, because a block page often comes back with a 200 OK. If you don't check, the model will happily summarise "Checking your browser before accessing..." as if it were the article.
Recent findings suggest that here's a compact fetch layer in Python using httpx and the standard library's robots.txt parser. It's around 100 lines and covers the four buckets above.
According to recent reports, use a descriptive user agent with a contact URL. It costs you nothing and site operators are far more likely to allow a client they can identify and reach. Then check robots.txt (standardised as RFC 9309) before every new host.
According to recent reports, the lru_cache means you fetch robots.txt once per origin rather than once per page. In a long-running service you'll want a cache with a TTL instead, since robots.txt files do change.
Industry observers note that this is the core of the layer. Instead of returning raw HTML, return a result that says which bucket you landed in.
The marker list is deliberately simple. It won't catch every interstitial, but it catches the common ones, and a false "blocked" is much cheaper than a confident summary of a CAPTCHA page. Log every blocked result with its first few hundred characters so you can grow the list from real data.
Retries should depend on the bucket. Throttled and broken responses get exponential backoff with jitter. Blocked and not-allowed responses get no retries at all.
Note the cap on wait. A site sending Retry-After: 3600 is telling you to go away for an hour. An interactive agent should report that to the user rather than block a request thread for that long.
According to recent reports, the cheapest fetch is the one you don't make. Agents often re-read the same documentation page several times in one session. Store the ETag and Last-Modified headers and send them back as If-None-Match and If-Modified-Since. A 304 Not Modified response costs the server almost nothing and lowers the odds of being rate limited.
It has been revealed that wire these into fetch() by passing headers=conditional_headers(url) to client.get(). On a 304, return the cached text. On a 200, call remember().
Finally, don't hide failures from the LLM. Give the tool output a structured shape so the model can reason about it:.
(extract_main_text is whatever readability or boilerplate-removal step you already use. Libraries like trafilatura work well here.).
Recent findings suggest that with this in place, the model's answer changes from a made-up summary to "I couldn't read that page because the site blocks automated access. Here's what I found from two other sources." That's a better product, and it builds trust with owners.
Recent findings suggest that once you've done everything above, some legitimate user-action fetches will still fail. That's usually because the site blocks data-center IP ranges outright, not because of anything your agent did. Teams handle this in a few ways, roughly in order of preference:.
The principle is the same in every case: the network path is a reliability decision, not a way around a site's wishes. Your robots.txt check and your honest user agent should stay in place whichever route the request takes.
Once your fetch layer returns structured results, observability is easy. Emit one event per fetch with the host, status bucket and latency, and watch these:.
If you're prototyping agents with AI coding assistants, ask them to generate these metrics hooks from the start. They're easy to bolt on now and painful to add later. For a sense of how well current tools handle this kind of infrastructure code, Vibe Coder Journal publishes hands-on reviews and benchmarks of AI coding tools.
The web your agent reads in 2026 isn't the open, fetch-anything web many of us learned on. More than half the traffic is automated, publishers have good reasons to be wary of AI crawlers, and the defaults now lean toward blocking. You can't change that. You can make your agent a good citizen, though: one that identifies itself, respects the rules, backs off when asked, and tells the truth when it can't read something.
Industry observers note that most of the reliability gains come from the unglamorous parts: classifying failures, caching and passing honest errors to the model. Get those right and your agent will feel much more dependable, even on the days when half the web won't let it in.
It depends on the jurisdiction, the site's terms of service and what you do with the content. Respecting robots.txt and site terms is the baseline. For commercial products, get proper legal advice.
Industry observers note that many site owners expect it to, and RFC 9309 makes no exception for user-triggered fetches. The safest default is to obey it and tell the user when a page is disallowed.
In a recent development, many bot-protection systems serve a challenge page with a 200 status. Check the body for challenge markers before passing it to the model, as in the looks_blocked() function above.
According to recent reports, only for pages that render their main content with JavaScript. Try a plain HTTP fetch first. Headless browsers are slower, cost more to run and are easier to fingerprint.
According to recent reports, 7 Day Free Trial. Cancel Anytime.
