700 mTwilight zone
Web scraping MCP servers
The 30 closest MCP servers that let Claude, ChatGPT, Cursor and other assistants fetch and extract content from web pages; 24 of them run remotely, nothing to install. From the official MCP Registry, npm and other open sources, out of 84,179 in the directory.
| oassis — web scraping, crawling and browser control for AI agents remote Web Tools and Web Tasks for AI agents. 100-credit MCP trial; API-key credits or x402. listing · updated 2026-10-22 |
| Flarum Forum Scraper - Discussion Crawler Tool remote Search or monitor any Flarum-powered forum for discussions, replies, participants, dates, and… listing · updated 2026-09-28 |
| Web Scraper & Crawler for RAG (Markdown) remote Website to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields. source · listing · updated 2026-11-08 |
| ketch-cli local package Fast, stateless CLI for web search, OSS code search, library docs, scraping, and crawling — with an MCP server for agents. source · listing · updated 2026-11-13 |
| @mseep/mcp-smart-crawler local package A command-line tool acting as an MCP (ModelContextProtocol) server, using Playwright to crawl web content for AI models. listing · updated 2025-04-15 |
| Agent Harbor pay-per-call tools remote Pay-per-call lookups and audits from Agent Harbor. Listing the tools is free; each call is paid per successful result in USDC on Base (x402), and a call that fails or finds nothing is not charged. Returned page and docum listing · updated 2026-11-22 |
| Desvela AI Crawler Block Test — free What a domain's CDN actually serves each AI crawler, as opposed to what its robots.txt declares. crawler_block_test(domain) sends the real user-agent of GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and CCBot at the ho listing · updated 2026-11-22 |
| Atlas Web Compiler & Scraper remote Deterministic Zero-LLM Web Compiler, Scraper, Crawler, and Site Mapper for AI Agents. source · listing · updated 2026-10-25 |
| scrapeless-mcp-server remote Web search, browser automation, scraping, crawling and CAPTCHA solving for AI agents. source · listing · updated 2026-09-08 |
| AI Crawler Permission Check remote Check a site's robots.txt rules for AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more). listing · updated 2026-11-21 |
| Site SEO & AI Crawler Audit remote Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion. source · listing · updated 2026-10-21 |
| AI Visibility Audit — Can AI crawlers read and cite your site? remote One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus fixes. listing · updated 2026-09-23 |
| ActableSite AI Crawler Monitor remote Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes. source · listing · updated 2026-07-13 |
| Atalaya AI crawler check remote Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why. listing · updated 2026-11-03 |
| CrawlerData: Google Shopping & SERP remote Google Shopping products and seller prices, plus Google Search, Images, News and Local results. listing · updated 2026-10-12 |
| The Crawler Zoo Arena remote Games for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents. listing · updated 2026-10-24 |
| Crimson Crawler remote Enrich, search, assess, and manage threat intelligence through 80+ typed MCP tools. listing · updated 2026-08-08 |
| Web Crawler MCP by InsightfulPipe remote Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account. source · listing · updated 2026-10-21 |
| Enrichment Crawlers MCP by InsightfulPipe remote Hosted Enrichment Crawlers MCP server, with each action switched on or off per account. source · listing · updated 2026-10-21 |
| ai-crawler-readiness remote Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index listing · updated 2026-09-19 |
| mcp-site-crawler-links-tables local package Crawl a website for links, file download links, HTML tables as rows, and lists, with no AI source · listing · updated 2026-11-11 |
| Website Crawler (SEO) remote Crawl a website from Claude and get a scored technical SEO report with a prioritised fix list. listing · updated 2026-11-17 |
| AI Crawler Index remote {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key listing · updated 2026-09-07 |
| Crawler IP Verifier - real Googlebot or fake remote {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key listing · updated 2026-09-07 |
| Crawler Log Triage remote {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key listing · updated 2026-09-13 |
| Robots.txt & AI Crawler Checker remote robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check. listing · updated 2026-10-18 |
| GenPark Bloom Filter URL Dedup Crawler local package In-memory Bloom filter for URL and document hash deduplication preventing redundant web crawler inge source · listing · updated 2026-10-21 |
| Website to Markdown Crawler: HTML to Markdown for LLM and RAG remote Turn a website into Markdown with headings, links and word counts. listing · updated 2026-11-12 |
| Crawlerpeek local package Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can access a URL. Read-only. source · listing · updated 2026-10-29 |
| CrawlKit Scrapers remote 58 scrapers via Apify MCP: Booking, Airbnb, Amazon, Google, Zillow, Indeed, Yelp, Reddit, Vinted source · listing · updated 2026-11-05 |
Add an MCP server to your assistant
For a remote server, the URL is all it takes. This is how it looks with Mare Liberum, a free MCP server without account:
Claude Code
claude mcp add --transport http mare-liberum https://mareliberum.dev/mcp
Cursor · ~/.cursor/mcp.json
{ "mcpServers": { "mare-liberum": { "url": "https://mareliberum.dev/mcp" } } }
VS Code · .vscode/mcp.json
{ "servers": { "mare-liberum": { "type": "http", "url": "https://mareliberum.dev/mcp" } } }
Claude, ChatGPT and other apps: add https://mareliberum.dev/mcp as a custom connector.
Questions
What does a web scraping MCP server do?
It lets an AI assistant fetch and extract content from web pages, through the Model Context Protocol. You add it to Claude, ChatGPT, Cursor or VS Code once; then the assistant calls it when a task needs it.
How do I install a web scraping MCP server?
A remote server: add its URL as a connector (Claude, ChatGPT) or in mcp.json (Cursor, VS Code). A local package: add the command from its listing, usually npx or uvx, to the same file. Each entry below links its source and listing.
Are these servers checked?
They are listed as their makers published them in open registries. Mare Liberum shows where each entry comes from and when it changed; read the source before you give a server access to your data.
More topics
filesystem Figma finance and stocks calendar Notion Cloudflare YouTube crypto and blockchain n8n and Zapier automation vector database Kubernetes Vercel