Does yandex.ru block AI crawlers?
yandex.ru does not block any of the 17 AI crawlers we tested: each one is allowed by its robots.txt, though 6 live fetches did not complete (checked 2026-10-01).
Nothing in its robots.txt keeps AI products out: its pages can be used for model training and fetched by ChatGPT, Claude and Perplexity when they answer a question and cite sources.
Every crawler we tested
| Crawler | Used for | Verdict | What we saw |
|---|---|---|---|
| GPTBot OpenAI | ChatGPT (training) | Allowed | Allowed: robots.txt permits it and the page loaded. |
| OAI-SearchBot OpenAI | ChatGPT search (retrieval) | Not established | Allowed by robots.txt; the live fetch timed out, so access was not confirmed. |
| ChatGPT-User OpenAI | ChatGPT (retrieval) | Not established | Allowed by robots.txt; the live fetch timed out, so access was not confirmed. |
| ClaudeBot Anthropic | Claude (training) | Not established | Allowed by robots.txt; the live fetch timed out, so access was not confirmed. |
| Claude-SearchBot Anthropic | Claude search (retrieval) | Allowed | Allowed: robots.txt permits it and the page loaded. |
| Claude-User Anthropic | Claude (retrieval) | Not established | Allowed by robots.txt; the live fetch timed out, so access was not confirmed. |
| Google-Extended Google | Gemini (training) | Allowed | Allowed: robots.txt permits it and the page loaded. |
| PerplexityBot Perplexity | Perplexity (retrieval) | Allowed | Allowed: robots.txt permits it and the page loaded. |
| Perplexity-User Perplexity | Perplexity (retrieval) | Not established | Allowed by robots.txt; the live fetch timed out, so access was not confirmed. |
| CCBot Common Crawl | Common Crawl (research) | Allowed | Allowed: robots.txt permits it and the page loaded. |
| Bytespider ByteDance | ByteDance (training) | Allowed | Allowed: robots.txt permits it and the page loaded. |
| Applebot-Extended Apple | Apple Intelligence (training) | Allowed | Allowed: robots.txt permits it and the page loaded. |
| Meta-ExternalAgent Meta | Meta AI (training) | Allowed | Allowed: robots.txt permits it and the page loaded. |
| Amazonbot Amazon | Alexa (training) | Allowed | Allowed: robots.txt permits it and the page loaded. |
| cohere-ai Cohere | Cohere (retrieval) | Not established | Allowed by robots.txt; the live fetch timed out, so access was not confirmed. |
| Diffbot Diffbot | Diffbot (research) | Allowed | Allowed: robots.txt permits it and the page loaded. |
| Timpibot Timpi | Timpi (research) | Allowed | Allowed: robots.txt permits it and the page loaded. |
Homepage SEO snapshot
From the Crawlwise SEO Index run, checked 2026-10-05. One fetch of the homepage, so it describes that page and not the whole site.
| Homepage | https://sso.passport.yandex.ru/push?uuid=224a4998-aacb-4eff-88da-620beae4bcee&retpath=https%3A%2F%2Fdzen.ru%2F%3Fyredirect%3Dtrue%26is_autologin_ya%3Dtrue |
|---|---|
| HTTPS | yes |
| Title | missing |
| Meta description | missing |
| Canonical | none declared |
| Noindex | no |
| H1 elements | 0 |
| Language | not declared |
| JSON-LD types | none |
| hreflang alternates | 0 |
| /sitemap.xml | not measured |
| Response time | 756 ms |
How this was measured
We read yandex.ru’s robots.txt, resolved it with an RFC 9309 matcher for each crawler’s product token, then fetched the homepage once as each crawler. A 403 or a challenge page is a block even when robots.txt allows the crawler; a timeout or rate limit is “not established”, never a block. One probe, from one location, on 2026-10-01, so a rule changed since then is not reflected here. Full methodology.
Check your own site
The free checker runs the same probe on any URL: robots.txt resolved for every AI crawler, then a live fetch as each one, so a CDN or firewall block shows up even when robots.txt looks fine.
Need it across every page? Site crawls start at $4.99 a month, and Pro adds monitoring that re-checks your URLs on a schedule.
Other sites
- site.com — blocks none of 17
- medallia.com — blocks 1 of 17
- vimeo.com — blocks 3 of 17
- admaster.cc — blocks none of 17
- smadex.com — blocks none of 17
- nexx360.io — blocks none of 17