Changelog

What's new in Crawly

Every release, documented.

v1.5.9Latest10 August 2026

v1.5.9

What's new

Bug fixes

Missing H1, Missing Title, and Missing Meta Description counts are no longer inflated by broken pages. The Issues tab was incorrectly counting 4xx and 5xx error pages toward these totals - only valid pages are now included.

MCP: fetch_batch

New `fetch_batch` tool fetches multiple URLs in a single call and returns each page as markdown, useful for bulk content analysis.

MCP: curl mode for crawl_site

`crawl_site` now accepts a `curl_mode` option that switches the crawler to curl-based fetching, helping bypass Cloudflare and similar bot detection.

MCP: Live progress notifications

`crawl_site` now sends live progress updates during a crawl so you can track pages discovered in real time without polling.

MCP: DB corruption recovery

The MCP server now automatically detects and recovers from database corruption instead of returning errors.

Engine improvements

Rate-limit handling is now more resilient - Retry-After headers are capped to avoid excessive waits, and per-domain backoff uses jitter to prevent thundering herd retries.

Download free

v1.5.319 May 2026

v1.5.3

What's new

MCP server - 10 new tools

The MCP server gains major new capabilities:

`check_external_links` - audit all outbound links from your crawl

`get_link_map` - full internal link graph showing which pages link to which

`get_images` - image audit with alt text, dimensions, and file size

`get_structure` - site structure grouped by URL path

`get_schema` - raw structured data (JSON-LD + microdata) per page

`get_inlinks` - bulk inlink lookup for any set of URLs

Orphan page detection - surface pages with no inlinks

Full meta description returned in long meta filter results

Auto-compare crawls - automatically diff consecutive crawls

`get_linking_suggestions` now ranks suggestions by opportunity

Claude Desktop support

The Settings screen now includes a one-click setup option for Claude Desktop alongside Claude Code, Cursor, Codex, and Windsurf.

Bug fixes

`crawl_site` now correctly saves external links to the database

Schema migrations updated to handle edge cases on first run

Fixed a column name bug in `get_linking_suggestions`

v1.5.013 May 2026

v1.5.0

What's new

Bulk selection across all tabs

Select rows on any tab and copy, export, or open in bulk. Extend selection to all matching results across pages.

Column resizing everywhere

Drag to resize columns on all tabs, not just All URLs.

Images tab improvements

New File Name column alongside Image Source

Next.js image URLs are automatically decoded to their original paths

Schema.org validation

Validates structured data against 15 rich result types. Issues surface in the Issues tab and All URLs table.

Open Graph auditing

Missing `og:image`, `og:title`, and `og:description` are detected and listed as issues.

MCP server enhancements

Two new tools: `get_linking_suggestions` for internal linking analysis, and `validate_schema` for structured data audits.

Bug fixes and polish

Response Codes tab filter now correctly filters by status group

Database corruption is auto-detected and recovered silently on launch

Header buttons now labelled: New Crawl, History, Settings

Fixed header layout jitter when crawl counts update

v1.4.012 May 2026

v1.4.0

What's new

Pause and resume crawls

Pause any running crawl and pick it back up later. Useful for large sites or when you need to free up resources temporarily.

Re-crawl a single page

Right-click any URL in the table to re-fetch just that page without re-running the full crawl.

Robots.txt viewer

A new Robots tab shows the raw robots.txt file alongside a list of pages that were blocked during the crawl.

AI Tools MCP setup

The Settings screen now includes a dedicated AI Tools section for configuring Crawly's MCP server. Supports Claude Code, Cursor, Codex, Windsurf, and a custom tool path option.

Usage analytics

Anonymous usage telemetry via PostHog helps improve the app. Opt out any time in Settings under Data and Storage.

MCP improvements

New tools: `check_migration_redirects`, `generate_redirect_map`, `get_robots`

`maxPages=0` now means unlimited - consistent with the in-app toggle

Fixed a bug where sites that redirect www to non-www (or vice versa) were not crawled correctly

v1.3.91 April 2026

v1.3.9

What's new

Settings screen

A new gear icon in the header opens the Settings panel, with sections for Crawl Defaults (max pages, user agent, crawl speed), Data and Storage (clear history, open DB folder), Claude Code integration, and Legal.

Response Codes tab

A dedicated tab showing a breakdown of all status codes (2xx, 3xx, 4xx, 5xx) with filterable cards and a full URL table including status text and indexability.

Headings tab

Browse all H1s and H2s across your crawl. Filter by Missing H1, Duplicate H1, or Missing H2. H1 length is highlighted when over 70 characters.

External Links tab

All outbound external links in one place, with source page and anchor text. Filterable and exportable.

URL bar in header

A permanent address bar lets you kick off a new crawl from anywhere in the app - just type a URL and press Enter or click Start.

Unlimited crawl pages

Toggle between a page limit and unlimited crawling in both the New Crawl modal and Settings. Unlimited crawls will spider the entire site.

MCP server improvements

User agent picker, URL prefix scoping, and depth filter in `get_pages`

Inline crawl summary and status text in results

Bot detection guidance - tips when a site appears to be blocking crawlers

`meta_too_long` filter, grouped duplicate detection, page cap warning

Inlinks surfaced directly in issue results

`fetch_as_markdown` crash fix when a single URL string is passed instead of an array

Bug fixes

Fixed a crash on macOS when clicking the dock icon after the app was already open

Canonicalised pages are no longer flagged as duplicate titles or duplicate H1s

DB syncs automatically when the app regains focus, so crawls started from the CLI or MCP server appear without needing a restart