AI crawler · training · Meta
What is Meta-ExternalAgent?
Meta's crawler for AI products built on Llama and related systems.
How this helps you show up
Feeds Meta's AI stack. Link previews in Facebook or Instagram use other fetchers — this is about AI products, not social cards.
If it cannot reach your page
Meta's AI training and retrieval pipelines will not pick up new pages from your site.
The facts
- robots.txt token
- Meta-ExternalAgent
- Owner
- Meta
- Product it serves
- Meta AI
- Category
- training — building model training data
- Training opt-out
- Documented — only affects training-style use, not every AI product
- User agent as sent
Mozilla/5.0 (compatible; Meta-ExternalAgent/1.0; +https://developers.facebook.com/docs/sharing/webmasters/crawler)
robots.txt is a request, not a lock. Our checker fetches your page as Meta-ExternalAgent and reports what actually came back — status code, challenge page, or real HTML.
Check your site in a minute
The AI crawler checker tests Meta-ExternalAgent plus the other answer-engine bots on one URL. No account. Want robots.txt logic only? Use the robots.txt tester.
Other AI crawlers
- GPTBot — OpenAI
- OAI-SearchBot — OpenAI
- ChatGPT-User — OpenAI
- ClaudeBot — Anthropic
- Claude-SearchBot — Anthropic
- Claude-User — Anthropic
- Google-Extended — Google
- PerplexityBot — Perplexity
- Perplexity-User — Perplexity
- CCBot — Common Crawl
- Bytespider — ByteDance
- Applebot-Extended — Apple
- Amazonbot — Amazon
- cohere-ai — Cohere
- Diffbot — Diffbot
- Timpibot — Timpi