Crawly
Learn

Guides and tutorials

In-depth articles on technical SEO, crawling, and getting the most from Crawly.

Guides

MCP

Claude Code MCP Setup

Connect Crawly's MCP server to Claude Code, Cursor, Codex or Windsurf.

Crawling

Crawl Configuration Guide

Understanding depth, concurrency, rate limits and user agents for accurate crawls.

Features

Using Crawl Comparison

How to diff two crawls and catch regressions before your clients do.

Migration

SEO Site Migration Checklist

Use Crawly to verify redirects, check indexability and protect rankings during a migration.

SEO

How to Run a Technical SEO Audit

A step-by-step walkthrough using Crawly to identify and fix technical SEO issues.

International

hreflang: The Complete Guide

Everything you need to know about implementing hreflang for international SEO.

Articles

SEO ToolsFree Tools

The Best Free SEO Tools in 2026

Paid SEO tools are good. Free tools, used correctly, are just as effective for most teams. The best free SEO tools in 2026, organised by what they actually do.

13 May 2026·7 min read
SEO AuditTechnical SEO

How to Do a Free SEO Website Check

Missing title tags, broken links, redirect chains - these problems accumulate silently. Here is how to do a thorough free SEO website check, step by step.

13 May 2026·6 min read
SEO CrawlerTechnical SEO

What is an SEO Crawler? (And How to Use One)

An SEO crawler visits every page on your site and records what a search engine sees. Here is how they work, what they find, and how to get the most from one.

13 May 2026·6 min read
SEO SpiderTechnical SEOCrawling

What is SEO Spider Software? (And How to Use It)

SEO spider software crawls your site like Googlebot, recording titles, status codes, and issues across every page. Here's how it works and what to look for.

13 May 2026·6 min read
Claude CodeMCPTutorial

How to Crawl Any Website with Claude Code

Crawly's built-in MCP server connects your crawl data directly to Claude Code. Here's how to set it up and what you can ask it.

12 May 2026·6 min read
SEO AuditTutorialTechnical SEO

How to Run a Technical SEO Audit with Crawly

A step-by-step walkthrough of Crawly's audit workflow: issues, response codes, security headers, hreflang, images, and crawl comparison.

10 May 2026·8 min read
Broken LinksTechnical SEO

How to Find (and Fix) Broken Links on Any Website

Broken links waste crawl budget, harm user experience, and lose link equity. Here's how to find them all — and track down which pages are causing the problem.

8 May 2026·5 min read
RedirectsTechnical SEO

How to Find and Fix Redirect Chains

Redirect chains waste crawl budget, slow page load, and dilute link equity. Here's how to find every chain on your site and fix them efficiently.

14 May 2026·5 min read
Crawl BudgetTechnical SEO

What Is Crawl Budget and Why Does It Matter?

Crawl budget determines how much of your site Google actually crawls. Here's when it matters, what wastes it, and how to audit it.

15 May 2026·6 min read