Your crawl data, queryable with AI
Crawly ships with a built-in Model Context Protocol server. Connect it to Claude Code and ask questions about your SEO data in plain English - straight from your terminal.
What is MCP?
Model Context Protocol (MCP) is an open standard that lets AI assistants like Claude connect to external tools and data sources. Instead of copy-pasting data into a chat, the AI can call tools directly and read live data.
Crawly implements an MCP server that exposes your crawl database to Claude Code. When you ask Claude a question about your site, Claude calls Crawly's tools, reads the results, and answers in natural language.
No CSV exports. No SQL queries. No manual filtering. Just ask.
Without MCP
- 1.Run a crawl
- 2.Export pages to CSV
- 3.Open in Excel / Sheets
- 4.Filter manually
- 5.Copy results to a doc
- 6.Write your recommendations
With Crawly MCP
- 1.Run a crawl
- 2.Ask Claude: "What are the top issues?"
How to set it up
Three steps from download to your first AI query.
Need the full config walkthrough? Read the full setup guide or see the full feature breakdown.
Run a crawl in Crawly
Open Crawly, enter a URL, and start a crawl. All data is stored locally in a SQLite database on your Mac.
Add the MCP server to Claude Code
In your Claude Code MCP config, add Crawly as a server. Crawly exposes a set of tools Claude Code can call - get_pages, get_issues, get_summary, and more.
Ask Claude anything
Open Claude Code in your terminal and start asking questions. Claude calls Crawly's MCP tools automatically and returns answers in natural language.
What you can ask
Example queries
Ask anything about your crawl data in plain English.
›Which pages are missing a meta description?
Returns a list of all crawled URLs where no meta description tag was found, with their titles and crawl depths.
›Show me all 4xx errors and the pages that link to them
Lists every broken URL alongside the pages that contain inbound links to it - ready to fix at the source.
›Find pages with duplicate H1s
Groups pages by shared H1 text so you can identify and consolidate duplicate heading content.
›List all images missing alt text
Returns image URLs, their source page, and alt attribute status - filtered to only those missing alt text.
›What redirect chains exist on this site?
Surfaces URLs with multi-hop redirect chains so you can clean up unnecessary hops.
›Which pages are non-indexable and why?
Shows all non-indexable pages with the reason - noindex meta tag, canonical pointing elsewhere, or blocked by robots.txt.
›Show me pages with a word count under 300
Filters the crawl to thin content pages that might need expanding or consolidating.
›Which pages have no internal links pointing to them?
Identifies orphan pages that are crawlable but have no internal link equity flowing in.
Available MCP tools
Claude calls these tools automatically based on your question.
| Tool | What it returns |
|---|---|
crawl_site | Trigger a full crawl from Claude Code. Returns a crawl_id and summary immediately - no need to call get_summary separately. |
list_crawls | List all previous crawls with their URL, status, page count, and date. |
get_summary | Overview of a crawl: total pages, 2xx/3xx/4xx/5xx counts, indexable vs non-indexable, average and slow response times. |
get_issues | All SEO issues grouped by category with counts - missing/duplicate titles, H1s, meta descriptions, broken pages, redirect chains, and more. |
get_pages | Query crawled pages with filters: broken, missing_title, duplicate_title, missing_meta, missing_h1, slow, missing_alt, non_indexable, and more. Supports search and regex. |
get_page_content | Return the full Markdown body of any crawled URL from the local database - no network request needed. |
fetch_as_markdown | Fetch any URL live and return its content as clean Markdown. Useful for reading competitor pages or extracting content without running a full crawl. |
get_inlinks | Find all crawled pages linking to a given URL, with anchor text and status codes - essential for tracing internal link equity. |
get_broken_with_inlinks | All 4xx broken pages in one call, each with inlink count and a sample source page and anchor text. |
get_issue_diff | Compare SEO issues between two crawls - newly broken pages, fixed pages, title changes, and indexability regressions. |
generate_redirect_map | Auto-generate a redirect mapping between two crawls using title, H1, and URL similarity. Exports as CSV ready for .htaccess, Vercel, or Netlify. |
check_migration_redirects | Verify redirect coverage after a migration. Fires live requests to old URLs and checks they return the correct 3xx response to the new domain. |
get_robots | Fetch and parse robots.txt for a crawled domain - shows rules per user-agent, sitemap declarations, and which crawled pages were blocked. |
validate_schema | Validate Schema.org structured data across all pages. Checks JSON-LD against Google rich result requirements and reports missing required properties. |
get_linking_suggestions | Given a pillar page, returns its headings alongside all other indexable pages with their headings - so Claude can suggest internal linking opportunities and anchor text. |
Start crawling smarter
Download Crawly for free. Connect to Claude Code via MCP and start auditing your site in minutes.
Download free