llms.txt Generator
Crawl your website and generate a structured llms.txt file for AI tools to understand your content.
Pages to crawl
Link types to extract
What is llms.txt?
llms.txt is a plain-text file you place at the root of your website, at /llms.txt. It gives AI language models and agents a structured, human-readable summary of your site - your brand, what each section covers, and links to key pages.
The format was proposed in 2024 as a practical way for websites to communicate with AI systems. Unlike a sitemap (which just lists URLs for crawlers), llms.txt is written for AI to read and understand. It helps AI assistants, coding agents, and AI-powered search tools give more accurate answers about your content without needing to process every page.
AI-readable format
A structured, Markdown-based file AI tools can parse reliably - no HTML, navigation, or boilerplate to wade through.
Grouped by section
The generator crawls your site and automatically groups pages by their first URL path segment, making the structure easy for any AI tool to follow.
Generated in seconds
Paste your homepage URL, click Generate, and the tool crawls up to 20 pages and outputs a ready-to-publish llms.txt file instantly.
llms.txt vs robots.txt vs sitemap.xml
These three files serve different audiences. You need all three - they are not interchangeable.
| File | Audience | Purpose | Content |
|---|---|---|---|
| robots.txt | Search engine crawlers | Control what bots can access | Allow/disallow rules per user-agent |
| sitemap.xml | Search engine crawlers | List URLs for indexing | URLs, last modified dates, priority |
| llms.txt | AI language models | Explain your site to AI | Clean summaries, titles, descriptions, structure |
Why your site needs an llms.txt file
Traditional SEO targets the ten blue links. AI SEO is about getting cited in AI-generated answers from ChatGPT, Perplexity, Google AI Overviews, and similar tools. llms.txt is one of the simplest signals you can give those systems.
Control how AI describes you
Without llms.txt, AI models guess your site's purpose from raw HTML. With it, you tell them directly - in clean, structured Markdown they can parse accurately.
Improve AI search citations
ChatGPT, Perplexity, and Google AI Overviews increasingly pull citations from structured sources. llms.txt makes your key pages easier to reference correctly.
Complement your existing SEO
llms.txt sits alongside robots.txt and sitemap.xml. It does not replace them. Think of it as adding an AI-optimised layer on top of your existing technical SEO.
Future-proof your content
The format is still evolving, but early adoption matters. Sites that add llms.txt now will be better positioned as AI search matures and citation patterns solidify.
How to add llms.txt to your website
Once generated, the file needs to live at the root of your domain - the same convention as robots.txt. It should be accessible at https://yourdomain.com/llms.txt.
Generate and copy your file
Enter your URL above, click Generate, then copy or download the output. You get a plain-text Markdown file ready to publish.
Upload to your server root
Place the file in the public root of your site - the same folder that serves your homepage. Do not put it in a subfolder.
Verify it is live
Open https://yourdomain.com/llms.txt in a browser and confirm the Markdown content loads. No server configuration needed.
| Platform | Upload path |
|---|---|
| WordPress | /public_html/llms.txt (root, not /wp-content/) |
| Next.js / React / Nuxt | /public/llms.txt |
| Shopify | Online Store > Files, then add a URL redirect from /llms.txt |
| Netlify / Vercel / GitHub Pages | /public/llms.txt (commit to repo) |
| cPanel / shared hosting | /public_html/llms.txt |
Who should add an llms.txt file
Any site that wants to be understood correctly by AI tools will benefit. These are the most common use cases.
SaaS and documentation sites
Give AI tools a structured knowledge base so they can answer support questions, explain features, and reference your docs accurately.
Agencies and service businesses
Make it clear what services you offer, which markets you serve, and what makes you different - in language AI can cite when someone asks for recommendations.
Blogs and content publishers
Help AI assistants reference your articles correctly. A well-structured llms.txt points AI tools to your best content rather than letting them guess.
E-commerce
Summarise your product categories and policies so AI-powered search tools can describe your offerings accurately when users ask relevant questions.
Frequently asked questions
Is llms.txt an official web standard?
Not yet. llms.txt is a proposed convention introduced in 2024 as a practical way for websites to communicate structured content to AI language models. Adoption is growing but the format has not been ratified by W3C or IETF. Think of it like structured data in the early days - not a standard, but increasingly useful to have.
Do search engines like Google use the llms.txt file?
No. Google, Bing, and other traditional search engines do not use llms.txt. It is intended for AI language models and tools such as ChatGPT, Claude, Perplexity, and AI-powered search products that may use it to better understand your site.
How often should I update my llms.txt file?
Whenever you add significant new pages or restructure your site. For most sites, regenerating once a month or after major content updates is more than enough.
What is the difference between llms.txt and llms-full.txt?
The spec allows for an optional llms-full.txt at the same location, containing the full text of each page rather than just summaries. The full version is larger but gives AI tools more context. Most sites only need the standard llms.txt file.
Will this improve my rankings in ChatGPT or Perplexity?
Having a well-structured llms.txt increases your chances of being cited accurately in AI-generated answers. No single factor guarantees inclusion, but giving AI tools a clear, structured summary of your site is one of the most direct signals you can provide.
Is this tool free?
Yes. The llms.txt generator is completely free to use with no login required. It crawls up to 20 pages and generates a ready-to-publish file instantly.
Crawly
Audit your entire site, not just one page
Crawly crawls every URL on your site and surfaces issues automatically. Free to download for Mac.
Download free