Guides
AI search & crawlers
Guides to AI crawlers and AI search: which bots to allow, robots.txt rules, llms.txt, AI Overviews and how ChatGPT and Claude find pages.
What is an AI crawler? Training, search, and user-triggered fetchers
Training crawlers, search crawlers, and user-triggered fetchers do different jobs. Here is how to tell them apart.
Which AI crawlers should you allow? A vendor-by-vendor decision guide
Blocking training and blocking answer visibility are different decisions. Here is each vendor’s split.
Does robots.txt control AI crawlers? What it does and does not do
robots.txt is a request, not a lock. Here is where it works for AI crawlers and where it does not.
Does llms.txt affect Google Search? What Google says, and what it is for
Google says llms.txt neither helps nor harms Search. Here is the honest case for and against publishing one.
How to check whether AI crawlers can access your site (step by step)
Five layers decide whether an AI crawler can read your page. Check them in this order.
GPTBot vs OAI-SearchBot vs ChatGPT-User: what each OpenAI bot does
Three OpenAI agents, three different jobs, three independent robots.txt settings.
Page 1 of 2 · 6 articles on this page