llms.txt: Should You Include All Pages or Only Key Content?

by Lynn Dreyer | Nov 16, 2025 | Insights and Trends, Technical SEO

llms.txt: Should You Include All Pages or Only Key Content?

As AI search and overviews grow, more SEOs are talking about adding an llms.txt file to their websites. Tools and blog posts call it a “new standard” for AI crawling, but there’s still confusion about what it actually does – and whether you should list every page or only your most important content.

This question and answer style article explains how llms.txt works, what the current adoption looks like, and the practical recommendation we give SEO Partners clients.

1. What is llms.txt?

llms.txt (sometimes written as LLMs.txt) is a plain text, Markdown-formatted file placed at the root of your domain, for example: https://yourdomain.com/llms.txt. It’s designed as a curated map for large language models (LLMs). Instead of listing every single URL like an XML sitemap, it highlights:

• Your most important pages
• Short descriptions of what each page covers
• Optional links to “AI-friendly” versions of content (often Markdown files) llms-txt

In other words, llms.txt says to AI tools:

“If you’re going to reference my site, start with these pages – they’re the clearest, most authoritative ones.”

2. Is llms.txt an official standard and who uses it?

As of late 2025: llms.txt is an emerging, unofficial convention, not a formal web standard.

Several CMS / SEO tools now support it (Yoast, Drupal modules, TYPO3, Webflow integrations, etc.), which makes it easier to generate.  However, multiple analyses report that major AI providers (Google, OpenAI, Anthropic, etc.) are not systematically using llms.txt for training or AI Overviews yet. So it sits in an interesting space:

Pro: Low-effort way to prepare for future AI crawling patterns and give AI agents a cleaner index of your content.

Con: There is no hard evidence that it directly drives rankings or traffic right now.

At SEO Partners, we treat llms.txt as a “nice-to-have experiment”, not as a core SEO ranking factor.

3. The core question: all pages or only key content?

Short answer:
For almost every site, especially small–to–medium businesses, we recommend that llms.txt includes only curated, high-value pages, not every URL.

Well, why not list everything?

Based on community testing and current guidance:

• llms.txt is intended as a curated overview, not a duplicate sitemap.
• Large, noisy files risk:
  • Hitting context limits when an AI reads it, so important sections may be ignored.
  • Diluting the signal about which content really matters.
  • Creating more maintenance work with little benefit.

Several practitioners and guides explicitly warn that dumping your full URL set into llms.txt offers no measurable advantage and can be overkill.

Which pages should go into llms.txt?

We recommend focusing on URLs that:

• Are evergreen and regularly kept up to date
• Demonstrate your expertise, authority and trust
• Accurately represent your core services or products
• You’d be happy to see quoted by an AI assistant

Typical candidates:

• Homepage and core service / package pages
• High-quality pillar blog posts and guides
• Documentation / help centre content
• Main FAQs and case studies

You can safely skip:

• Thin tag pages and internal search results
• Privacy policy / terms pages
• Fragmented archive or filter URLs
• Short-lived promo landing pages

Rule of thumb:

If you wouldn’t proudly send the page to a new client as “a great explanation”, it probably doesn’t belong in llms.txt.

4. How does llms.txt differ from robots.txt and XML sitemaps?

It helps to see these three as different tools in your technical SEO stack:

• robots.txt – controls where crawlers are allowed to go; you can also block specific AI agent user-agents here.
• XML sitemaps – list most or all indexable URLs to help search engines discover and crawl your site efficiently.
• llms.txt – acts as a curated content guide for AI systems, pointing them at the pages that best represent your site.

They’re complementary, not replacements for each other.

5. Will llms.txt improve my Google rankings or AI Overviews?

Right now, there is no evidence that llms.txt directly improves organic rankings or guarantees inclusion in AI Overviews:

• Several independent reviews and Google-facing discussions emphasise that standard SEO (content quality, structure, links, technical health) remains the real driver of visibility.
• Analyses of AI crawler behaviour show that most LLM crawlers still ignore llms.txt, fetching regular HTML pages and respecting robots.txt instead.

However, there is a strategic upside:

A well-structured llms.txt file may make it easier for future AI agents to find your best content quickly, potentially improving how your brand is cited in AI answers once (and if) adoption grows.

SEO Partners’ stance:

If it’s easy to generate and maintain, implement llms.txt as part of your AI-readiness checklist – but don’t expect instant traffic or rankings from it.

6. How should I structure an llms.txt file?

A simple, best-practice structure (adapted from the proposed spec) looks like this:

# SEO Partners – Website Knowledge Base
> Curated resources for understanding our SEO services and processes.

## Core Pages
- https://seopartners.co.za/
- https://seopartners.co.za/seo-packages/
- https://seopartners.co.za/seo-maintenance/

## Knowledge Base – AI & Technical SEO
- https://seopartners.co.za/knowledge-base/llms-txt-include-all-pages-or-key-content/
- https://seopartners.co.za/knowledge-base/how-to-index-your-website-in-google-and-bing/
- https://seopartners.co.za/knowledge-base/core-seo-ranking-factors/

## Client Guides
- https://seopartners.co.za/resources/seo-quick-start-guide/
- https://seopartners.co.za/resources/technical-seo-checklist/

Key principles:

• Start with a clear H1 title and short summary blockquote.
• Group URLs under logical H2 sections (Core Pages, Guides, Docs, etc.).
• Keep descriptions brief but meaningful – think “index page for AI”.
• Keep the file reasonably small and easy to read.

7. How often should you update llms.txt?

You don’t need to touch it every week.

Update llms.txt when:

• You launch a new flagship service or major evergreen guide.
• You significantly rewrite a core page.
• You retire or redirect URLs that are currently listed.
• Many plugins (like Yoast’s llms.txt feature) will update the file automatically on a schedule, but it’s still worth checking periodically that your hand-picked “hero content” list is up to date.

8. Should you pay an agency just to implement llms.txt?

Because llms.txt is:

• Unofficial,
• Lightly adopted, and
• Simple to implement, we do not recommend buying a standalone “llms.txt optimisation package”.

It’s reasonable for an agency to:

• Include llms.txt as a small line-item in a broader AI/technical SEO project.
• Use it to document and surface your most important content for both humans and AI.

But if someone is pitching it as a game-changing AI SEO service on its own, ask for real-world before/after data – in most cases, there won’t be any.

At SEO Partners, we treat llms.txt as a supporting file – useful, low-risk, but always secondary to solid SEO fundamentals.

9. Quick implementation checklist (SEO Partners recommendation)

If you want llms.txt on your site, here’s a pragmatic approach:

Decide if it’s worth it. 

• If your site is small and updates rarely, you can skip it for now.
• If you produce in-depth content or documentation, adding llms.txt is a sensible, low-cost experiment.

Create a curated URL list

• 10–50 of your best pages is plenty for most SMEs.
• Focus on services, evergreen guides, FAQs and key resources.

Write a clean markdown file

• Use headings and short descriptions as shown in the example above.
• Save it as llms.txt in your website root.

Check your robots.txt

• Make sure AI user-agents you want to allow are not blocked.
• If you want to block some AI bots entirely, do that here – not in llms.txt. 

Monitor, but don’t obsess

• Keep focusing on content quality, technical health and links.
• Treat llms.txt as bonus scaffolding for the AI-shaped future of search.

In conclusion – it’s better to be ready and prepared that not and the trick is to stay up to date with the latest developments. If you have any questions – please don’t hesitate to book a consultation with us