Mare Liberum

150 mSunlight zone

Mare Liberum · open data · the state of the agent web

Which top websites block Googlebot?

0% of the 289 most visited websites read so far block Googlebot (Google) in robots.txt — 1 site. Another 39 turn all automated readers away.

Googlebot builds the index an AI search or answer engine draws from. Blocking it keeps the site out of those answers.

Updated 2026-11-22 15:10 UTC · read from each site's robots.txt (RFC 9309, longest match) · /web.json (CC0)

Sites that block Googlebot

reddit.com0/100 Eblocks 18 of 18 agents

Is your site one of them?

Check any website — free, no account: mareliberum.dev/check

The other agents

Bytespider 35% · ClaudeBot 34% · CCBot 34% · GPTBot 33% · Applebot-Extended 29% · Meta-ExternalAgent 29% · PerplexityBot 28% · Amazonbot 27% · Google-Extended 27% · ChatGPT-User 22% · Perplexity-User 19% · Claude-User 18% · Claude-SearchBot 18% · DuckAssistBot 18% · OAI-SearchBot 16% · MistralAI-User 16% · Bingbot 1%

robots.txt states wishes; most large AI operators say they follow it. This data is free for everyone. Support the open atlas — voluntary, any amount.