Guides
AI search & crawlers (page 2)
Guides to AI crawlers and AI search: which bots to allow, robots.txt rules, llms.txt, AI Overviews and how ChatGPT and Claude find pages.
How ChatGPT, Claude, Gemini, Perplexity, Copilot and Grok find web pages
Each assistant searches an index before it answers. Here is whose index, per the vendors’ own docs.
robots.txt for AI crawlers: copy-paste examples with explanations
Three robots.txt files you can paste today, and why each line is there.
AI Overviews and AI Mode: what site owners can actually control
No special markup, no special file. The controls are the ones Search already had.
Structured data in the AI search era: what still matters for a SaaS or tool site
Schema helps machines understand a page. It is not an AI citation switch. Here is what to keep.
Page 2 of 2 · 4 articles on this page