Site Crawler
Enter any URL to crawl up to 100 pages and surface common SEO issues instantly.
Pages to crawl
Need to crawl more than 100 pages? The Crawly desktop app has no page cap.
Download Crawly freeWhat this tool checks
Enter a URL and the crawler follows internal links, visiting up to 20 pages and auditing each one for common SEO issues.
Missing page titles
Checks whether each page has a non-empty title tag. Pages without titles are flagged so you can prioritise fixes.
Missing H1 tags
Detects pages with no H1 heading. A missing H1 is a common on-page SEO issue that can affect how search engines understand your page.
Missing meta descriptions
Flags pages with no meta description tag. While not a ranking factor directly, missing meta descriptions affect click-through rates in search results.
4xx errors and redirects
Highlights pages returning client errors (404, 410) or redirects (301, 302). Both can waste crawl budget and harm user experience.
Internal link discovery
Follows internal links automatically from the seed URL. No sitemap required. The crawler stays within the same domain throughout.
CSV export
Download the full crawl results as a CSV with URL, status, title, H1, meta description, canonical tag, issues found, and response time.
How to use the site crawler
A full crawl of 20 pages typically completes in under 30 seconds.
Enter your URL
Paste the homepage or any internal page URL. The crawler starts from that URL and follows internal links. It stays within the same domain and ignores external links.
Watch pages appear in real time
Results stream in as each page is fetched. The crawler runs up to 3 pages in parallel and stops once 20 pages have been crawled or there are no more links to follow.
Review issues and export
Use the filter tabs to focus on pages with issues. Export to CSV to download the full results for client reporting or further analysis in a spreadsheet.
When to run a site crawl
A quick crawl is one of the most efficient health checks you can run on a site. These are the most common situations where it saves time.
Before a client presentation
Run a crawl before presenting a site audit to quickly identify any obvious on-page issues - missing titles, broken pages, or missing H1s.
After a site update
Whenever new pages are added or content is restructured, a crawl confirms that titles, descriptions, and headings are all in place.
When rankings drop
A crawl can quickly rule out on-page causes of a ranking drop - missing tags, 404 errors, or pages that have been accidentally de-optimised.
Frequently asked questions
How many pages does the crawler visit?
The tool crawls up to 20 pages per run. It starts from the URL you enter, follows internal links, and stops when it reaches 20 pages or runs out of links to follow. For full-site crawls with no page limit, use the Crawly desktop app.
Does the crawler follow redirects?
Yes. The crawler follows redirects automatically and records the final URL. If an internal link points to a page that redirects, the redirect is noted in the Issues column.
Will it crawl external links?
No. The crawler only follows internal links that share the same domain as the starting URL. External links are ignored.
Why is a page showing as missing a title when it has one?
The crawler fetches the raw HTML and parses the title tag with regex. Titles set by JavaScript after page load will not be detected. If your site renders titles client-side, the desktop Crawly app - which parses rendered HTML - will give more accurate results.
Can I crawl a site that requires a login?
No. This tool can only crawl publicly accessible pages. For crawling behind authentication, you need the Crawly desktop app.
Is this tool free?
Yes. The site crawler is completely free to use with no login required.
Crawly
Audit your entire site, not just one page
Crawly crawls every URL on your site and surfaces issues automatically. Free to download for Mac.
Download free