Crawlers
Every search engine, AI model and SEO tool that reads this site, across the website, the API and the image CDN. Updated every hour.
2026-08-31 to 2026-09-06
The period being compared against reaches back further than our records, which start 2026-08-28. Changes marked * read larger than they were.
Requests
6.0M
Served
200 GB
Providers
51
Hosts
14
By what they are for
- AI Crawler 4.0M 127 GB
- Search engines 1.5M 51.3 GB
- AI search 240k 5.8 GB
- SEO tooling 129k 5.8 GB
- Monitoring & Analytics 126k 7.7 GB
- Page Preview 32.1k 430 MB
- Archiver 6.4k 566 MB
- Security 6.0k 1.1 GB
- AI assistant 3.6k 251 MB
- Accessibility 1.1k 88.0 MB
AI Crawler are 66% of automated traffic.
Which host they hit
- mangabaka.org 3.7M 113 GB
- cdn.mangabaka.dev 1.3M 68.2 GB
- api.mangabaka.org 910k 4.3 GB
- images.mangabaka.dev 78.5k 14.0 GB
- ap-proxy.mangabaka.dev 19.8k 179 MB
- mangabaka.dev 10.0k 55.8 MB
- analytics.mangabaka.dev 607 3.8 MB
- www.mangabaka.org 163 158 KB
- www.mangabaka.dev 79 83.6 KB
- api.mangabaka.dev 74 90.5 KB
39% of bot requests never touch the website.
What they take away
- html 3.2M 110 GB
- txt 665k 643 MB
- json 623k 3.1 GB
- jpeg 389k 32.1 GB
- empty 375k 394 MB
- atom 324k 581 MB
- avif 192k 3.2 GB
- webp 120k 26.0 GB
- png 72.3k 20.6 GB
- js 44.9k 2.1 GB
Images are 13% of requests and 41% of bytes.
What we answered
- 200 OK 4.5M 193 GB
- 403 refused 911k 2.0 GB
- 301 moved 358k 344 MB
- 404 gone 193k 393 MB
- 206 partial 17.6k 3.9 GB
- 308 slash 13.2k 10.6 MB
- 302 found 4.8k 4.7 MB
- 502 bad gateway 4.7k 7.5 MB
- 304 unchanged 896 874 KB
- 429 slow down 889 782 KB
18% of bot requests are refused or errored.
Who crawls us
by requestsMeta
AI Crawler
+122%*
1.8M
58.1 GB 29%
Anthropic
AI Crawler
+92.5%*
1.3M
35.0 GB 22%
Yandex
Search engines
+106%*
852k
24.0 GB 14%
OpenAI
AI Crawler
+659%*
811k
25.7 GB 13%
Microsoft
Search engines
+151%*
630k
24.4 GB 10%
Apple
AI search
+1431%*
238k
5.7 GB 4.0%
DataDog
Monitoring & Analytics
+150%*
126k
7.7 GB 2.1%
Amazon
AI Crawler
+205%*
75.4k
8.8 GB 1.3%
Requests per day
all verified bots, every hostBytes per day
all verified bots, every hostCounts come from Cloudflare's edge across both zones, refreshed every hour. Only bots Cloudflare has verified are named, so anything claiming to be Googlebot without the network to prove it is left out. Our own monitoring is excluded.