How to Run a Technical SEO Audit with Crawly

A technical SEO audit finds the issues that stop search engines from crawling, rendering, and ranking your pages correctly. Unlike content audits, technical audits are about the infrastructure: status codes, directives, tags, headers, and site architecture.

This guide walks through running a complete technical audit using Crawly, section by section.

Before you start: run the crawl

Open Crawly and click New Crawl. Enter the root URL of the site you want to audit. For most audits, leave the defaults in place:

• Spider mode - follows links automatically from the seed URL
• Respect robots.txt - on for client sites, off if you want to see everything including blocked URLs
• User agent - use Googlebot to see how the site looks to Google, or Chrome 128 for a browser perspective
• Concurrency - 10 simultaneous requests is the default; reduce this for slower shared hosting

Hit start. Crawly will work through the site and you can watch pages come in live. Once complete, work through each tab in order.

1. Issues tab - your audit starting point

The Issues tab is the fastest way to understand the health of a site. Every problem Crawly detects is automatically grouped by type and sorted by count.

The 19 issues Crawly checks fall into three severity levels:

• Errors - broken pages (4xx), server errors (5xx), missing title tags, duplicate titles, mixed content
• Warnings - missing meta descriptions, missing H1, duplicate H1, redirect chains, non-indexable pages, images missing alt text, missing HSTS, near-duplicate content, hreflang missing x-default
• Opportunities - titles too short or too long, meta descriptions too long, missing CSP header

Start with errors. A page returning a 4xx or 5xx is failing completely - fix those before worrying about meta description length. Then work down through warnings, then opportunities.

Click any issue to see the full list of affected URLs. You can export the list directly for client reporting or developer handoff.

2. Response Codes tab - status code breakdown

The Response Codes tab shows the full distribution of status codes across the crawl: how many 2xx success pages, 3xx redirects, 4xx client errors, and 5xx server errors.

Things to look for:

• Unexpected 404s - pages that should exist but return Not Found. Cross-reference with your XML sitemap to find sitemap URLs that are broken.
• Soft 404s - pages returning 200 but showing "page not found" content. These appear as 200s in the response tab but will show as non-indexable if they contain a noindex tag.
• Redirect chains - any URL that takes more than one hop to reach its final destination wastes crawl budget. The Issues tab flags these automatically, but the Response Codes tab lets you see the full picture.
• 5xx errors - server errors during a crawl often indicate infrastructure problems, rate limiting, or pages that fail under load.

3. Pages tab - on-page data at scale

The Pages tab shows every crawled URL with its title, meta description, H1, status code, word count, response time, internal link count, and more. Use it to spot patterns that do not surface cleanly in the Issues tab.

Useful filters to run:

• Sort by word count ascending to find thin content pages that may need expanding or consolidating
• Sort by response time descending to find your slowest pages - useful context when cross-referencing Core Web Vitals data
• Filter by non-indexable to see every page Google cannot index, with the reason (noindex, canonical, etc.)
• Filter by depth to find important pages buried too deep in the site architecture

For a focused guide on auditing canonical directives, see how to check canonical tags.

4. Images tab - alt text audit

The Images tab lists every image across every crawled page. The key metric here is missing alt text - both for accessibility and for image search visibility.

Filter the tab to missing alt to see only the images that need fixing. You can bulk copy or export the list for developer handoff. The stats bar at the top shows the total image count, unique sources, and missing-alt count at a glance.

5. Headings tab - H1 and H2 audit

The Headings tab shows the H1 and all H2s for every page in a single scrollable table. It is the fastest way to do a site-wide heading audit without opening individual pages.

Look for:

• Pages with no H1 - flagged in Issues but visible here in context
• Multiple H1s on a single page - a common CMS template problem
• H1s that do not match the page title - often fine, but worth flagging if the mismatch is significant
• Very short H1s (under 10 characters) or very long ones (over 70 characters)

6. External Links tab - outbound link audit

The External Links tab lists every outbound link on the site with its destination URL, source page, and anchor text. Use this to:

• Spot links to outdated or low-quality external resources that should be updated or removed
• Check that important external links (partners, citations, references) are pointing to the right URLs
• Audit affiliate or sponsored link anchor text for consistency

If you need data beyond the built-in fields, Crawly's custom extraction feature lets you define CSS selectors to pull any element from every crawled page.

7. Site Structure tab - architecture review

The Site Structure tab renders your site as an expandable folder tree grouped by URL path. This gives you an immediate visual sense of how the site is organised and where pages sit in the hierarchy.

Check that your most important pages are near the top of the tree (shallow crawl depth). Pages buried five or six levels deep receive less crawl budget and internal link equity. If key conversion pages are deep, look for opportunities to add internal links that bring them closer to the surface.

8. Security headers - a quick win most auditors skip

Crawly checks five security headers on every page: HSTS, Content-Security-Policy, X-Frame-Options, Referrer-Policy, and X-Content-Type-Options. Missing headers are flagged in the Issues tab.

These are not direct ranking factors, but they are increasingly scrutinised in technical audits and affect how browsers handle the site. Missing HSTS on an HTTPS site is a common finding that is usually a quick server configuration fix.

9. hreflang - international site audit

For international sites, Crawly extracts and validates hreflang tags on every page. The most common issue it finds is missing x-default - the fallback tag that tells Google which version to show when no other language match exists.

Other hreflang problems Crawly surfaces include malformed language codes (e.g. en-UK instead of en-GB) and pages that have hreflang tags pointing to non-200 URLs.

10. Crawl Comparison - track changes over time

Once you have run a crawl, run another one after making changes and use the Crawl Comparison feature to diff them. Crawly shows you exactly which pages were added, removed, or changed - with field-level diffs for title, H1, status code, and indexability.

This is particularly useful for migrations. Crawl the old site, migrate, crawl the new site, and compare. Any URL that changed status code, lost its title, or became non-indexable will appear in the diff immediately.

Using Claude Code to speed up your audit

If you have Crawly's MCP server connected to Claude Code, you can skip the manual tab-by-tab process for the initial analysis:

Give me a summary of the top technical SEO issues on this crawl, prioritised by severity.

Claude calls the get_issues and get_summary tools and returns a structured breakdown - errors first, then warnings, with counts for each issue type. You still need to dig into the details, but the initial triage takes seconds instead of minutes.

---

A thorough technical audit using Crawly typically takes 30-60 minutes for a site of a few hundred pages - most of that time reading and thinking about what you find, not wrestling with the tool.

For a quick-reference version of this workflow, see the step-by-step audit guide. For an overview of the full free toolkit, see the best free SEO tools guide.

Download Crawly and run your first audit today.