Agent Accessibility: Your Website Is Broken and You Don't Know It

Last week, my AI assistant — Karl — was trying to help me theory-craft a ship loadout for X4: Foundations. A new patch had rebalanced every weapon in the game. I needed current damage numbers, fire rates, tracking stats. The kind of thing wikis and fan sites exist to provide.

Karl couldn't get them.

The X4 wiki sat behind a Cloudflare challenge page that demanded JavaScript execution and browser fingerprinting. The community forum had Anubis anti-bot protection that returned a 403 to anything without a full browser stack. A fan-made damage calculator had a robots.txt that disallowed all crawlers — including the kind that fetch a single page on behalf of a human user. Three sites, three different walls, zero results.

I got the stats eventually. I opened a browser, clicked through the CAPTCHAs myself, and read the numbers to Karl. But that's the point. I had to do the browsing because the sites my assistant tried to reach had decided it wasn't a real visitor.

This isn't a story about AI entitlement. This is a story about accessibility.

Building planning-first AI systems, one pipeline at a time. Get weekly deep dives into local AI, Rust, and agent orchestration — no fluff, no hype.

The New Visitor You're Turning Away

Web accessibility usually means screen readers, keyboard navigation, color contrast, ARIA labels. It means making sure humans with different abilities can use your site. That framework is well-established and it's the right one.

But there's a new class of visitor that none of those frameworks cover: AI agents acting on behalf of human users.

Someone asks their assistant to compare prices. Someone with motor disabilities uses voice commands to browse. Someone who is visually impaired has their screen-reader agent fetch a page for summarization. In every case, the HTTP request comes from software, not a browser. And in 2026, a terrifying percentage of the web treats that as an attack.

If your site requires a full browser stack — JavaScript execution, canvas fingerprinting, CAPTCHA solving — to read a public page, your site is inaccessible to a growing class of users. You just can't see them.

The Wall: Four Layers of Lockout

Cloudflare's "I'm Under Attack" Mode

Cloudflare challenges are the most visible layer. When a request looks suspicious — no browser headers, unusual TLS fingerprint, data center IP range — Cloudflare serves an interstitial page that runs JavaScript to verify the client is a real browser.

For bots, this is a hard wall. API clients don't execute JavaScript. They fetch HTML. When the response is a challenge page instead of the article, the agent has no way forward. It can't click "I'm human." It can't wait for the redirect. It gets a page that says "Verifying you are human" and that's the end.

The irony: many of these challenges appear on pages that contain no sensitive data whatsoever. A public wiki page about weapon damage in a space game. A blog post about CSS animations. A product listing with a price tag. All public. All indexed by Google. All invisible to any client that doesn't run Chrome.

Anubis and the Anti-Bot Arms Race

Anubis is a newer player. It's an open-source anti-bot system that uses proof-of-work challenges — the client must compute a hash that meets a difficulty target before being served content. For a browser with JavaScript, this takes a fraction of a second. For an API client, it's an unsolvable equation.

Anubis is increasingly popular on self-hosted communities, wikis, and forums. The deployment pattern is almost always the same: someone notices scrapers hitting their server, installs Anubis as a nuclear option, and the bot problem disappears. Along with every legitimate agent client.

The motivation is understandable. Scrapers consume bandwidth. AI training crawlers ignore rate limits. Some bots are genuinely abusive. But the response is a sledgehammer that makes no distinction between a hostile scraper hammering your server at ten thousand requests per second and a single agent fetching one page for a human who asked.

Robots.txt: The Shotgun Approach

Robots.txt was designed to tell crawlers what they should and shouldn't index. It was a polite convention, not a security measure. In the AI era, it has become something else entirely: a blanket ban slapped on the entire site to prevent model training.

The problem is that robots.txt doesn't distinguish between "I don't want my content used to train a commercial AI model" and "I don't want an agent to read this page on behalf of a human." A Disallow: / directive blocks everyone — GPTBot, ClaudeBot, Karl, and the hypothetical future standard for agent-friendly crawling that doesn't exist yet.

Site owners who add blanket robots.txt blocks are making a reasonable choice about training data. But they're also making an unintentional choice about accessibility. They're saying: no agent may read this site, for any reason, on behalf of any user.

CAPTCHA Walls

The final layer is the one most users are familiar with. "Select all images with crosswalks." "Prove you're not a robot." These challenges are designed to be solvable by humans and unsolvable by bots, which means they are unsvable by agents too.

When a CAPTCHA appears mid-session — after an agent has already loaded several pages — the agent has no way to communicate the challenge back to the user. It can't render the image grid. It can't wait for the user to solve it. The session stalls, the request fails, and the user assumes the agent is broken.

The agent isn't broken. The site is.

The Real Cost

Here's what happens when an agent can't reach your content. The user doesn't visit your site manually. They don't see your ads. They don't discover your newsletter. They don't find your product.

They ask their assistant, the assistant tries to reach you, you turn it away, and the assistant finds the answer somewhere else — if it can. If it can't, the user goes without.

You are not losing traffic to scrapers. You are losing traffic to relevance. When an agent can't read your site, your site ceases to exist for that user.

This will sound apocalyptic. It isn't — yet. Most browsing is still done by humans in browsers. But the shift is accelerating. Voice assistants, AI-powered search, agent-based shopping — each one routes through an HTTP client that doesn't look like Chrome. Each one hits the same walls.

What to Do About It

This is the part where this stops being a complaint and starts being a roadmap. If you run a website and you want it to be reachable by agent clients — without opening the floodgates to scrapers — here are concrete steps.

Publish Structured Data

The single most effective thing you can do is expose your content as structured data. JSON-LD, Schema.org markup, an RSS feed with full content — these let agents consume your data without loading your rendered pages.

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Plasma Cannon Mk II",
  "damage": 4200,
  "fireRate": "2.1/s",
  "tracking": 0.85
}

Structured data is already how search engines understand your site. It's also how agents will. If your product specs, article content, or reference data are available as JSON, an agent doesn't need to render your JavaScript. It fetches the data, parses it, and moves on. No Cloudflare challenge triggered. No CAPTCHA needed.

Provide an API Endpoint

If you have the engineering resources, a simple read API is the gold standard. A REST endpoint that returns JSON for your public content. Version it, rate-limit it, require an API key if you want to track usage. But make it available.

The X4 wiki that blocked Karl? If it had an API endpoint like /api/weapons/plasma-cannon, the entire problem would have been solved in one request. Instead, three sites lost a reader because none of them offered a machine-readable path.

An API isn't a new concept. Most major platforms already have one — Reddit, Wikipedia, GitHub. The gap is in the long tail of community sites, wikis, and fan projects. These are the sites that get blocked most often because they run on shared infrastructure with aggressive defaults.

Allowlist Agent User Agents

If you use Cloudflare, Anubis, or similar protection, configure allowlists for known agent user agents. This is a one-line config change in most systems:

# Cloudflare WAF custom rule
(http.user_agent contains "GPTBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Applebot-Extended") or
(http.user_agent contains "PerplexityBot")
→ Action: Skip (Allow)

You can be selective. You don't have to allow every bot. But you should allow agents from providers that publish clear usage policies and respect rate limits. OpenAI, Anthropic, and others publish their user agent strings and crawling behavior. If you trust their policies, allow their agents.

Use Granular Robots.txt

Instead of blanket Disallow: /, use targeted rules:

# Block training crawlers you don't want
User-agent: GPTBot
Disallow: /private/
Allow: /

# Block specific paths, not the whole site
User-agent: *
Disallow: /admin/
Disallow: /api/internal/

This says: "You can read my public content. You can't scrape my admin panel. You can't use my private data for training." That's a policy. A blanket ban is not a policy — it's a surrender.

Serve Lite Content to Non-Browser Clients

If you're worried about bandwidth from agent requests, serve a lightweight version of your pages to clients that don't send browser headers. A stripped HTML version without JavaScript, images, or interactive elements. This is the same technique used for mobile-only or print stylesheets, just applied to agent clients.

The detection is simple: check for browser-specific headers like Sec-CH-UA or Accept: text/html. If they're absent, serve the lite version.

The Principle

Accessibility has always been about one principle: the medium shouldn't determine who can access the message. Screen readers exist because visual interfaces exclude visually impaired users. Keyboard navigation exists because mouse interfaces exclude motor-impaired users.

Agent accessibility is the same principle applied to a new axis: interactive, JavaScript-heavy, challenge-gated interfaces exclude agent users. Users who navigate by voice. Users who delegate browsing because they can't hold a mouse. Users who simply prefer to ask and receive rather than hunt and peck.

The web was built on the assumption that every visitor is a browser. That assumption was never fully true — screen readers proved that decades ago. It's becoming less true every day.

Your site doesn't need to choose between security and accessibility. It needs to recognize that the two are the same thing. A site that only works for Chrome with JavaScript enabled isn't secure — it's fragile. A site that serves its content through structured data, APIs, and clean HTML is secure and accessible. It works for browsers. It works for agents. It works for the person who will try to reach your site tomorrow through a method you haven't imagined yet.

The next time someone's assistant tries to read your site, make sure it can. That someone is your visitor. They're just arriving through a different door.