llms.txt: The New File Every Site Needs for AI Search Visibility (Complete Guide)

Key Takeaways

• llms.txt is a plain-text file at yourdomain.com/llms.txt that gives AI assistants a structured, authoritative description of your site — what it is, what pages matter, and what audiences it serves.
• It is different from robots.txt. robots.txt controls crawl access (allow/disallow). llms.txt controls content description — how you want AI systems to represent your site when answering user queries.
• Without it, AI assistants guess. They synthesise a description of your product from crawled content — which is often incomplete, outdated, or simply wrong.
• The format is simple. A heading, a one-paragraph description, and a few Markdown lists. Any developer can ship it in under an hour.
• llms-full.txt is an extended variant for larger sites, supplying complete documentation in a single context file so LLMs can answer detailed questions accurately.

---

llms.txt is a plain-text Markdown file placed at the root of your domain — yourdomain.com/llms.txt — that tells AI language models what your website is about, which pages are most important, and how you want your product or service described when an AI assistant answers a question about it.

The standard was proposed by Jeremy Howard, co-founder of fast.ai, in September 2024. The premise is straightforward: AI assistants are increasingly the first place users go for product comparisons, tool recommendations, and how-to answers. Those assistants need to understand your site to represent it accurately. llms.txt gives them a structured, curated summary — the same way sitemap.xml gives search crawlers a structured index of your pages.

---

What llms.txt Is — and What It Is Not

What It Is

llms.txt is a content description file. It tells an AI assistant:

• The name of your site and what it does (in one concise paragraph)
• Which pages are the canonical, highest-value destinations
• How your documentation or content is organised
• Basic facts about your company — founding date, target audience, pricing tiers

When an AI assistant like Claude, Perplexity, or a future AI search agent reads your llms.txt, it has an authoritative summary it can draw on when forming answers about your site. It no longer has to infer who you are from a mix of home page copy, blog posts, and third-party mentions.

What It Is Not

llms.txt is not robots.txt. These two files serve completely different purposes, and conflating them is the most common misconception in the wild.

File	Controls	Mechanism
robots.txt	Which pages crawlers can access	Crawl allow/disallow directives
llms.txt	How AI systems describe your site	Structured content summary
sitemap.xml	Which URLs exist and when they changed	URL index for crawlers
llms.txt	What your site means and what matters most	Semantic description for LLMs

Blocking GPTBot in robots.txt prevents OpenAI from crawling your content at all. llms.txt does the opposite — it actively helps AI systems understand and represent your content accurately. You want both: open crawl access (via robots.txt) and an accurate description (via llms.txt).

For a full guide on configuring AI crawler access, see our robots.txt guide for AI crawlers.

---

Why llms.txt Matters for AI Search

Here is the problem it solves.

When a user asks Perplexity "what's the best free SEO audit tool," the AI assistant draws on everything it has indexed about the tools in that category. For each tool, it synthesises a description from whatever it found during crawling — the home page, a few blog posts, some reviews on third-party sites. That synthesis is often inaccurate. Pricing is wrong. Features are mixed up. The target audience is mischaracterised.

The tool that has a well-written llms.txt gives the AI a clean, authoritative source for that synthesis. Instead of guessing, the AI can refer to the file's one-paragraph description, pull the correct pricing tiers, and surface the right pages for each use case.

This matters more as AI search grows. Perplexity processes over 100 million queries per month. ChatGPT has 200 million weekly active users. Google's AI Overviews appear on an expanding share of search result pages. If AI assistants are describing your product to potential customers, you want to control that description. llms.txt is how you do it.

---

The llms.txt Format

The spec is intentionally minimal. It uses standard Markdown with a few conventions:

• First line: # Site Name (H1) — the canonical name of your site
• Second block: > Description (blockquote) — one paragraph describing what the site does and who it serves
• H2 sections: Named groupings (## Key Pages, ## Documentation, ## About, etc.)
• List items under each section: - [Page Title](URL): Brief description

Here is the full spec pattern:

# Your Site Name
> One-paragraph description of the site and what it does, who it serves, and what makes it distinct.

## Key Pages
- [Page Title](URL): Brief description of what this page covers or does

## Documentation
- [Docs Section](URL): What this section covers

## About
- Founded: [year]
- Audience: [target audience description]
- Pricing: [pricing summary or link]

That is it. No special syntax, no proprietary schema, no tooling required. A plain .txt file with Markdown formatting.

---

A Complete llms.txt Example

Here is a production-ready example for a real site:

# seo.yatna.ai
> Automated SEO auditing platform that scores websites across 7 categories:
> E-E-A-T, Technical SEO, On-Page SEO, Schema Markup, Performance, AI Readiness,
> and Images. Uniquely checks whether AI assistants (ChatGPT, Perplexity, Claude)
> can find and cite your content — not just whether Google can rank it.

## Key Pages
- [Free SEO Audit](https://seo.yatna.ai/lp/free-seo-audit-tool/): Run a free
  automated SEO audit covering all 7 categories — no account required
- [SEO Academy](https://seo.yatna.ai/seo-academy/): Educational guides on
  technical SEO, schema markup, and AI search readiness
- [Robots.txt Checker](https://seo.yatna.ai/tools/robots-checker/): Free tool
  to verify AI crawler access (GPTBot, ClaudeBot, PerplexityBot)

## Documentation
- [AI Search Readiness](https://seo.yatna.ai/seo-academy/ai-search-readiness-audit-guide/):
  How the AI readiness score is calculated and what signals are checked
- [Schema Validation](https://seo.yatna.ai/seo-academy/softwareapplication-schema-rich-results/):
  Schema types checked during an audit and how to fix errors

## Pricing
- Free tier: 1 audit, 5 pages
- Starter: 5 audits/month, 25 pages
- Pro: 20 audits/month, 100 pages
- Business: Unlimited audits, 500 pages

## About
- Audience: Founders, marketers, and SEO professionals who want their content
  found and cited by AI assistants as well as traditional search engines
- Focus: AI search readiness alongside traditional technical SEO

Notice the description in the blockquote does two things: it names the product category ("automated SEO auditing platform"), lists the specific scoring categories, and calls out the differentiating feature (AI assistant visibility). A model reading this can answer "what is seo.yatna.ai" accurately without having to synthesise from scattered pages.

---

llms-full.txt: The Extended Variant

For sites with rich documentation — developer tools, SaaS platforms, technical libraries — the llms.txt spec also defines llms-full.txt.

Where llms.txt is a curated index of key pages, llms-full.txt is a single large Markdown file containing the complete content of your documentation in one context window. The purpose is to let users drop your llms-full.txt directly into an AI assistant's context and ask detailed questions about how your product works — without the AI needing to crawl individual pages.

When to create llms-full.txt:

• Your product has a public API or developer docs
• Users frequently ask "how do I do X with your product" in AI assistants
• You want AI coding assistants (GitHub Copilot, Cursor) to understand your API accurately

When llms.txt alone is sufficient:

• Marketing or content site (blog, landing pages, academy)
• Informational or lead-generation site where the goal is accurate description, not deep Q&A

Both files can coexist. Start with llms.txt; add llms-full.txt when your documentation is comprehensive enough to be worth packaging.

---

Where to Place the File

Next.js

Place the file at /public/llms.txt in your project root. Next.js serves everything in /public as static files at the root of your domain, so yourdomain.com/llms.txt will resolve automatically with no additional routing configuration required.

your-nextjs-project/
├── public/
│   ├── llms.txt        ← place it here
│   ├── robots.txt
│   └── sitemap.xml
├── app/
└── ...

WordPress

Place the file directly in the WordPress root directory — the same directory that contains wp-config.php, wp-login.php, and your existing robots.txt. It will be served as a static file at yourdomain.com/llms.txt without any plugin or permalink configuration.

Other Platforms

For any web platform, the rule is: the file must be served as a static plain-text file at /llms.txt from your domain root. If your platform uses a CDN or object storage for static assets, upload it there and ensure the path resolves at root level. Verify by visiting https://yourdomain.com/llms.txt in a browser — you should see the raw Markdown text with no HTML wrapper.

---

How to Verify an AI Reads Your llms.txt

Once the file is live, you can test whether it is influencing AI assistant responses within a few days of deployment. AI crawlers re-index relatively quickly for new signals.

Test 1 — Perplexity direct query:

Go to perplexity.ai and ask:

"What is [yoursite.com] and what does it do?"

Note the response. If it includes accurate details from your llms.txt description — correct pricing, accurate feature list, right target audience — the file is being used. If the response contains errors you fixed in llms.txt, the crawler may not have revisited yet; check back in 3–5 days.

Test 2 — Claude direct prompt:

Open claude.ai and paste:

"I'm going to paste the contents of a file. Based on it, tell me what this site does and who it is for."

Then paste your llms.txt content. This tests whether the file is semantically clear and would produce an accurate AI summary — independent of whether Claude has crawled it yet.

Test 3 — Baseline comparison:

Before publishing llms.txt, screenshot the AI assistant responses to questions about your site. Repeat the same queries 2 weeks after publishing. Measurable improvement in accuracy and completeness indicates the file is working.

---

FAQ

What is llms.txt?

llms.txt is a plain-text Markdown file placed at the root of your website (yourdomain.com/llms.txt) that tells AI language models — ChatGPT, Claude, Perplexity, and others — what your site is about, which pages are most important, and how your product or service should be described. It was proposed by Jeremy Howard of fast.ai in 2024.

How is llms.txt different from robots.txt?

robots.txt is a crawl-control file: it tells bots which pages they are allowed or not allowed to access. llms.txt is a content description file: it tells AI assistants what your site does and how to represent it accurately. They do opposite jobs. robots.txt controls access; llms.txt controls meaning. Both files should be present and correctly configured.

Is llms.txt an official standard?

llms.txt is a community-proposed standard, not a formal W3C or IETF specification. It was introduced by Jeremy Howard and has been adopted by a growing number of sites — including major developer tool providers. AI crawlers are adding support for it incrementally. Adoption is early, which means sites that implement it now have a discovery advantage over those that wait for it to become mandatory.

Do I need llms-full.txt as well?

llms-full.txt is optional and most useful for sites with rich technical documentation. If you run a SaaS with a developer API, a platform with detailed how-to docs, or a product where users ask AI assistants detailed "how do I" questions, llms-full.txt supplies that detail in one file. If you run a marketing site, content blog, or simple lead-generation property, llms.txt alone is sufficient.

Will llms.txt improve my Google rankings?

Not directly. llms.txt is not a ranking signal in Google's traditional PageRank-based algorithm. Its value is in AI search channels — ChatGPT, Perplexity, Claude, and Google's AI Overviews — where AI assistants use it to describe your site accurately. As AI search grows as a traffic and brand-awareness channel, accurate AI representation becomes increasingly valuable, even when the path to your site runs through an AI answer rather than a traditional search result page.

---

Check whether your site has llms.txt, correct AI crawler access, and all 7 SEO categories scored — run a free audit at seo.yatna.ai.