Mare Liberum

400 mTwilight zone

The Atlas

MareLiberum-Atlas, the crawler

MareLiberum-Atlas reads a few websites every few minutes, from our own server, going down the list of the most linked websites of the world (the Majestic Million), and in turn the most visited sites of 192 countries (the Chrome UX Report lists), plus every website an agent asked the sea about; a site is read again after about a month. For each site it reads robots.txt first. Where robots.txt says no to MareLiberum-Atlas (or to all crawlers), it reads nothing else. Otherwise it reads the home page, /llms.txt, /.well-known/agent-card.json and the OpenSearch file the home page names — a handful of small requests per visit, never many at once on one site.

It keeps only what an agent needs to use the site: the title and description, the search link, the names of the main sections, feeds, sitemaps, developer pages, structured-data types and which AI agents robots.txt lets in. No personal data, no page text beyond the title and description.

Keep the Atlas away

User-agent: MareLiberum-Atlas
Disallow: /

The user agent: MareLiberum-Atlas/1.0 (+https://mareliberum.dev/atlas). Questions: GitHub issues.

The data

Open data, CC BY 4.0, on the branch atlas-data. Ranks from the Majestic Million (CC BY 3.0, Majestic).