Free Backlink CheckerFree Backlink Checker
Crawly
Learn

Guides and tutorials

In-depth articles on technical SEO, crawling, and getting the most from Crawly.

Articles

SEOBeginners

SEO for Beginners: A Complete Guide

SEO gets your pages to rank in Google without paying for ads. This guide covers how search engines work, the three pillars of SEO, and where to start.

19 May 2026·9 min read
SEORankings

How to Improve Your Google Rankings

Improve Google rankings by diagnosing what is holding your pages back. This guide covers technical fixes, on-page SEO, internal links, and backlinks.

19 May 2026·8 min read
SEOKeyword ResearchFree Tools

How to Do Keyword Research for Free

Free keyword research tools cover most of what beginners and small teams need. Here is how to use Google Search Console, autocomplete, and free tools.

19 May 2026·7 min read
Technical SEORedirects

What is a 301 Redirect and How Does It Work?

A 301 redirect permanently moves a URL to a new address. Here is how it works, when to use it, and how redirect chains affect SEO and crawl budget.

16 May 2026·7 min read
Technical SEOIndexing

What is Page Indexability?

Indexability determines whether Google can discover, crawl, and add a page to its index. Here is every factor that can prevent a page from being indexed.

16 May 2026·7 min read
Technical SEOInternal Links

How to Audit Internal Links on Your Site

Internal links distribute ranking signals and help Google discover pages. Here is how to find orphan pages, fix broken links, and improve anchor text.

16 May 2026·7 min read
Technical SEOIndexing

What is a Noindex Tag and When to Use It?

A noindex tag tells search engines not to include a page in their index. Here is how it works, when to use it, and the mistakes that cause problems.

16 May 2026·6 min read
SEOLocal SEO

What is Local SEO and How Does It Work?

Local SEO determines which businesses appear in the Google Local Pack. Here is how it works, what Google's three ranking factors are, and how to improve.

16 May 2026·8 min read
SEOEcommerce SEO

What is Ecommerce SEO? A Complete Guide

Ecommerce SEO covers category pages, product descriptions, faceted navigation, and crawl budget. Here is how to optimise an online shop that scales.

16 May 2026·8 min read
Technical SEOInternational SEO

What is International SEO?

International SEO ensures search engines serve the right language version to users in each country. Here is how hreflang works and the common mistakes to avoid.

16 May 2026·7 min read
SEOKeyword ResearchContent

What is Keyword Research in SEO?

Keyword research identifies the phrases your audience searches for. Here is how to do it step by step, the metrics that matter, and the free tools available.

16 May 2026·7 min read
Technical SEOMobile

What is Mobile-First Indexing?

Mobile-first indexing means Google ranks your mobile page, not desktop. Here is what that means in practice, the common problems, and how to check compliance.

16 May 2026·7 min read
ContentSEO Audit

How to Do an SEO Content Audit

A content audit reviews every page on your site to decide whether to keep, improve, consolidate, or remove it. Here is the full process, step by step.

16 May 2026·8 min read
SEOContent Strategy

What is Topical Authority in SEO?

Topical authority is your site's demonstrated depth of expertise on a subject. Here is how it affects rankings, how to build it, and how content clusters work.

16 May 2026·7 min read
Technical SEOSite Architecture

What is Site Architecture in SEO?

Site architecture is how pages on a website are organised and connected through internal links. Here is why it matters for crawling, ranking, and link equity.

16 May 2026·7 min read
Technical SEOInternal Links

What Are Orphan Pages and How Do You Fix Them?

Orphan pages have no inbound internal links. Search engines rarely crawl them and they receive no link equity. Here is how to find and fix them.

16 May 2026·6 min read
SEOContentE-E-A-T

What is E-E-A-T and How Does It Affect SEO?

E-E-A-T stands for Experience, Expertise, Authoritativeness, and Trust. Here is what each component means, how it affects rankings, and how to demonstrate it.

16 May 2026·7 min read
Technical SEOJavaScript

What is JavaScript SEO and Why Does It Matter?

JavaScript SEO is about whether search engines can access JS-rendered content. Here is how rendering approaches differ and the most common problems to fix.

16 May 2026·7 min read
On-Page SEOTechnical SEO

What is On-Page SEO? The Complete Guide

On-page SEO covers the elements you control on each page: title tags, headings, content, images, and schema. Here is what each one does and how to audit them.

16 May 2026·7 min read
Technical SEOCrawlingCrawl Budget

What is a Crawl Trap and How Do You Fix It?

A crawl trap causes search engine crawlers to follow an infinite number of URLs, wasting crawl budget. Here is every trap type and how to detect and fix them.

16 May 2026·7 min read
Technical SEOURL Parameters

What is a URL Parameter and How Does It Work?

URL parameters are a major source of duplicate content and crawl waste. Here is how different parameter types affect SEO and how to handle each one correctly.

16 May 2026·7 min read
Technical SEOLink Building

What is Link Equity (Link Juice) in SEO?

Link equity is the ranking value passed between pages through hyperlinks. Here is how it flows through a site, how redirects affect it, and how to audit it.

16 May 2026·6 min read
Technical SEOInternal LinksBacklinks

What is Anchor Text and Why Does It Matter?

Anchor text is the clickable text of a hyperlink. Here is how Google uses it, the types that matter, and how to audit your internal anchor text distribution.

16 May 2026·6 min read
Technical SEOContent

What is Thin Content and Why Does It Matter?

Thin content fails to satisfy search intent regardless of word count. Here is what Google considers thin, what to do about it, and how to find it.

16 May 2026·6 min read
Technical SEODuplicate Content

What is Duplicate Content in SEO?

Duplicate content is a consolidation problem, not a penalty. Here is how Google handles it, what causes it, and how to fix it with canonicals and redirects.

16 May 2026·6 min read
Technical SEO

What is Technical SEO? The Complete Guide

Technical SEO is the foundation every other SEO effort builds on. Here is what it covers, why it matters, and how to audit it properly.

16 May 2026·8 min read
BacklinksLink BuildingSEO Tools

How to Check Backlinks in Bulk

Checking backlinks one at a time is slow. Here is how to check referring domains, backlink counts and authority scores for up to 100 domains simultaneously.

16 May 2026·5 min read
APIDomain AuthorityDevelopers

Free Domain Authority API for Developers

Crawly's free SEO API returns domain authority, referring domains, backlink counts and spam scores for any domain. 100 requests per day, no credit card.

16 May 2026·6 min read
APIBacklinksDevelopers

Free Backlink API: Query Any Domain's Link Profile

Crawly's free backlink API returns referring domains, individual backlinks with quality ratings, and outbound links for any domain. No credit card required.

16 May 2026·6 min read
APISEODevelopers

How to Automate SEO Reporting with an API

Manual SEO reporting is slow. Here is how to automate DA tracking, link prospecting, and backlink audits using Crawly's free SEO API with Python and JavaScript.

16 May 2026·8 min read
GEOAI SearchSEO

What is Generative Engine Optimisation (GEO)?

GEO is the practice of getting cited by AI search tools: Google AI Overviews, ChatGPT, Perplexity, and Claude. Here is how it works and how to do it.

16 May 2026·9 min read
SEOSERP

What is a SERP and How Does It Work?

A SERP is the page Google shows after a search. Here is what it contains, how different result types affect click-through rates, and how to preview yours.

16 May 2026·6 min read
Technical SEOURLs

How to Write SEO-Friendly URLs

Clean URLs are short, readable, and keyword-focused. Here is how to structure them correctly, what to avoid, and how to change URLs without losing rankings.

16 May 2026·6 min read
Technical SEOSocial Media

What is Open Graph and Why Does It Matter?

Open Graph tags control how your pages look when shared on social media. Here is what the tags are, how to write them, and how to preview your social cards.

16 May 2026·5 min read
AnalyticsSEO

What Are UTM Parameters and How Do You Use Them?

UTM parameters track where your traffic comes from. Here is how the five parameters work, how to name them consistently, and how to build them without errors.

16 May 2026·5 min read
On-Page SEOTechnical SEO

How to Write a Meta Description That Gets Clicks

Meta descriptions do not affect rankings but directly affect click-through rate. Here is how to write them, what length to aim for, and what Google rewrites.

16 May 2026·6 min read
On-Page SEOTechnical SEO

What is a Title Tag and How Do You Optimise It?

The title tag is the clickable headline in search results. It is one of the most important on-page SEO signals. Here is how to write one that ranks.

16 May 2026·5 min read
Technical SEOCanonical Tags

What is a Canonical Tag and How Does It Work?

Canonical tags tell search engines which version of a page to index. Here is how rel=canonical works, when to use it, and the mistakes that cause problems.

16 May 2026·7 min read
Technical SEOCore Web VitalsPage Speed

What Are Core Web Vitals and Why Do They Matter?

Core Web Vitals are Google's three page experience metrics: LCP, INP, and CLS. Here is what each one measures, the thresholds, and how to fix common problems.

16 May 2026·8 min read
Technical SEOSchemaStructured Data

What is Schema Markup and How Does It Help SEO?

Schema markup tells search engines what your content means. Here is how it works, which types matter most, and how to implement and validate it correctly.

16 May 2026·7 min read
Technical SEOXML Sitemap

What is an XML Sitemap and Do You Need One?

An XML sitemap lists every URL you want search engines to index. Here is what to include, what to leave out, how to submit it, and the mistakes to avoid.

16 May 2026·6 min read
Domain AuthorityBacklinksSEO

What is Spam Score and How Do You Use It?

Spam score is a Moz metric that estimates how likely a domain is to be penalised by Google. What it measures, the thresholds, and what to do about it.

16 May 2026·6 min read
Domain AuthorityBacklinksSEO

Domain Authority vs Domain Rating

DA is from Moz, DR is from Ahrefs, and Authority Score is from Semrush. They measure the same thing using different data. Here is how to use each one correctly.

16 May 2026·5 min read
Domain AuthorityLink BuildingSEO

How to Increase Your Domain Authority

Domain authority rises when your backlink profile improves. Here is the order of operations, what moves the score fastest, and how to track your progress.

16 May 2026·7 min read
Domain AuthorityBacklinksSEO

What is Domain Authority and How is it Calculated?

Domain authority is a score that predicts how well a site ranks in search engines. Here is how it works, what counts as good, and how to check yours.

16 May 2026·6 min read
GEOAI SearchTechnical SEO

What is llms.txt and Does Your Website Need One?

llms.txt is a plain-text file that tells AI crawlers what your site is about. Here is what it is, whether it actually helps, and how to generate one.

16 May 2026·5 min read
Technical SEOCrawling

What is a robots.txt File and How Does it Work?

A robots.txt file tells search engine crawlers which pages to visit and which to skip. Here is how it works, common mistakes, and how to check yours.

16 May 2026·6 min read
BacklinksLink Building

How to Check Your Backlinks (Step by Step)

Checking your backlinks takes minutes with the right tool. Here is a step-by-step guide to seeing every domain that links to your site, for free.

14 May 2026·5 min read
BacklinksSEO

What Are Referring Domains and Why They Matter

Referring domains is the most important metric in backlink analysis. Here is what they are, how they differ from backlinks, and why the count matters.

14 May 2026·5 min read
BacklinksSEO Audit

How to Do a Backlink Audit

A backlink audit tells you which links are helping your site, which are hurting it, and what to do about each. Here is the full process, step by step.

14 May 2026·6 min read
BacklinksLink BuildingCompetitor Analysis

How to Find and Analyse Competitor Backlinks

Competitor backlink analysis is one of the fastest ways to find link building opportunities. Here is how to do it properly.

14 May 2026·5 min read
BacklinksTechnical SEO

How to Build a Google Disavow File

A disavow file tells Google to ignore specific backlinks when assessing your site. Here is when to use one and how to build it correctly.

14 May 2026·6 min read
Link BuildingSEO

What Is Link Building? A Beginner's Guide

Link building is the process of earning backlinks from other websites. Here is why it matters, how it works, and the most effective approaches in 2026.

14 May 2026·7 min read
BacklinksLink Building

How to Read a Backlink Profile

A backlink profile tells you who links to a site, how good those links are, and where risks exist. Here is how to interpret what you see.

14 May 2026·5 min read
BacklinksFree ToolsSEO Tools

The Best Free Backlink Checker Tools in 2026

Paid backlink tools are good. Free tools are good enough for most audits. Here are the best free backlink checkers available in 2026.

14 May 2026·6 min read
SEO ToolsFree Tools

The Best Free SEO Tools in 2026

Paid SEO tools are good. Free tools, used correctly, are just as effective for most teams. The best free SEO tools in 2026, organised by what they actually do.

13 May 2026·7 min read
SEO AuditTechnical SEO

How to Do a Free SEO Website Check

Missing title tags, broken links, redirect chains - these problems accumulate silently. Here is how to do a thorough free SEO website check, step by step.

13 May 2026·6 min read
SEO CrawlerTechnical SEO

What is an SEO Crawler? (And How to Use One)

An SEO crawler visits every page on your site and records what a search engine sees. Here is how they work, what they find, and how to get the most from one.

13 May 2026·6 min read
SEO SpiderTechnical SEOCrawling

What is SEO Spider Software? (And How to Use It)

SEO spider software crawls your site like Googlebot, recording titles, status codes, and issues across every page. Here's how it works and what to look for.

13 May 2026·6 min read
Claude CodeMCPTutorial

How to Crawl Any Website with Claude Code

Crawly's built-in MCP server connects your crawl data directly to Claude Code. Here's how to set it up and what you can ask it.

12 May 2026·6 min read
SEO AuditTutorialTechnical SEO

How to Run a Technical SEO Audit with Crawly

A step-by-step walkthrough of Crawly's audit workflow: issues, response codes, security headers, hreflang, images, and crawl comparison.

10 May 2026·8 min read
Broken LinksTechnical SEO

How to Find (and Fix) Broken Links on Any Website

Broken links waste crawl budget, harm user experience, and lose link equity. Here's how to find them all — and track down which pages are causing the problem.

8 May 2026·5 min read
RedirectsTechnical SEO

How to Find and Fix Redirect Chains

Redirect chains waste crawl budget, slow page load, and dilute link equity. Here's how to find every chain on your site and fix them efficiently.

14 May 2026·5 min read
Crawl BudgetTechnical SEO

What Is Crawl Budget and Why Does It Matter?

Crawl budget determines how much of your site Google actually crawls. Here's when it matters, what wastes it, and how to audit it.

15 May 2026·6 min read