AI crawler · training · ByteDance
What is Bytespider?
ByteDance's training crawler (TikTok's parent company).
How this helps you show up
Relevant if you care how ByteDance-powered products might represent your category. Unrelated to Google; separate decision from your answer-engine retrieval bots.
If it cannot reach your page
Your pages are not added to ByteDance training collections going forward.
The facts
- robots.txt token
- Bytespider
- Owner
- ByteDance
- Product it serves
- ByteDance
- Category
- training — building model training data
- Training opt-out
- Documented — only affects training-style use, not every AI product
- User agent as sent
Mozilla/5.0 (compatible; Bytespider; +https://zhanzhang.toutiao.com/)
robots.txt is a request, not a lock. Our checker fetches your page as Bytespider and reports what actually came back — status code, challenge page, or real HTML.
Check your site in a minute
The AI crawler checker tests Bytespider plus the other answer-engine bots on one URL. No account. Want robots.txt logic only? Use the robots.txt tester.
Other AI crawlers
- GPTBot — OpenAI
- OAI-SearchBot — OpenAI
- ChatGPT-User — OpenAI
- ClaudeBot — Anthropic
- Claude-SearchBot — Anthropic
- Claude-User — Anthropic
- Google-Extended — Google
- PerplexityBot — Perplexity
- Perplexity-User — Perplexity
- CCBot — Common Crawl
- Applebot-Extended — Apple
- Meta-ExternalAgent — Meta
- Amazonbot — Amazon
- cohere-ai — Cohere
- Diffbot — Diffbot
- Timpibot — Timpi