crawlwise.
AI Crawler Access Report · All sites

Which websites block Claude-User?

24 of 243 popular sites we measured block Claude-User (9.9%), checked 2026-10-01.

Claude-User is Anthropic’s crawler for user-triggered fetch (Claude). It fetches pages so an answer can cite them, so blocking it removes a site from those answers as a live source. More about Claude-User.

The robots.txt rule

To block it:

User-agent: Claude-User
Disallow: /

To allow it explicitly:

User-agent: Claude-User
Allow: /

A server or CDN can still refuse the crawler when robots.txt allows it. That is why the checker fetches as the crawler as well as reading the file.

Sites that block Claude-User

SiteHow
msn.comrobots.txt disallow
facebook.comrobots.txt disallow
fbcdn.netrobots.txt disallow
amazon.comrobots.txt disallow
instagram.comrobots.txt disallow
skybridge.clickrobots.txt disallow
linkedin.comrobots.txt disallow
creativecdn.comrobots.txt disallow
twitter.comrobots.txt disallow
pinterest.comrobots.txt disallow
reddit.comrobots.txt disallow
sharethrough.comrobots.txt disallow
amazon.co.zarobots.txt disallow
netcraze.iorobots.txt disallow
juicyscore.comrobots.txt disallow
pinimg.comrobots.txt disallow
t.corobots.txt disallow
x.comrobots.txt disallow
wp.comrefused at the server
amazonvideo.comrobots.txt disallow
meraki.comrobots.txt disallow
ad-score.comrobots.txt disallow
storygize.netrobots.txt disallow
starttest.comrobots.txt disallow

Check your own site

The free checker runs the same probe on any URL: robots.txt resolved for every AI crawler, then a live fetch as each one, so a CDN or firewall block shows up even when robots.txt looks fine.

Check AI crawler access, free

Need it across every page? Site crawls start at $4.99 a month, and Pro adds monitoring that re-checks your URLs on a schedule.

Other crawlers