Blog

Answer-engine crawlers

How answer-engine crawlers such as GPTBot, ClaudeBot and PerplexityBot reach your pages, and the blocking mistakes that keep them out.

Five crawler-blocking mistakes that hide your best pages

Blocking training bots is not the same as blocking retrieval. Here is how to tell what you blocked.