The way Large Language Models (LLMs) interact with websites is still pretty fluid. There’s a new standard being talked about, llms.txt, which is supposed to tell LLMs how to understand your site. Think of it like robots.txt, but specifically for LLMs. While robots.txt guides web crawlers on what to index, llms.txt aims to directly inform LLMs about your content’s structure and purpose.
llms.txt: What it is and Where it’s Going
Right now, here’s what’s happening:
- Most sites just convert their
sitemap.xmlinto Markdown and use that as theirllms.txt. WICH IS NOT LLM FRIENDLY - Major LLMs aren’t really using
llms.txtyet because not many people have adopted it.
But don’t dismiss it. If you can clearly explain your product, API, or information to an AI, it could be a big deal for getting your stuff in front of users by influencing how an LLM summarizes or recommends your content.
Many SEO folks say llms.txt isn’t worth the effort right now. But it doesn’t have to be a big job. You can even set it up to update automatically with your site, meaning zero maintenance. If llms.txt does take off, getting in early could definitely pay off.
An Easy Way to Generate llms.txt
You can implement this right now with a super simple method. llms.page lets you grab an llms.txt file for your site, even if you don’t have one.
It’s literally as easy as going to llms.page or just hitting this URL:
https://get.llms.page/{yourdomain}/llms.txt
It’ll give you the file super fast.
How it Works
Send a GET request to https://get.llms.page/example.com/llms.txt, and it will build and serve an llms.txt file automatically.
This file is generated based on your homepage’s structure and internal links, for free.
Example
Hitting https://get.llms.page/github.com/llms.txt returns something like this:
# GitHub
> Join the world's most widely adopted, AI-powered developer platform where millions of developers, businesses, and the largest open source community build software that advances humanity.
## Customer Stories
- [Explore customer stories](https://github.com/customer-stories)
- [Customer stories](https://github.com/customer-stories?type=enterprise)
- [Read customer story](https://github.com/customer-stories/duolingo)
- [Figma](https://github.com/customer-stories/figma): TechnologyFigma streamlines development and strengthens securityRead customer story
- [Mercado Libre](https://github.com/customer-stories/mercado-libre): Financial servicesMercado Libre cuts coding time by 50%Read customer story
- [Mercedes Benz](https://github.com/customer-stories/mercedes-benz): AutomotiveMercedes-Benz standardizes source code and automates onboardingRead customer story
## Enterprise
- [Home | github.com](https://github.com/enterprise): Enterprise platform AI-powered developer platform
- [Premium Support](https://github.com/enterprise/premium-support)
- [Startups](https://github.com/enterprise/startups)
## Features
- [All features](https://github.com/features)
- [Actions Automate any workflow](https://github.com/features/actions)
- [AI](https://github.com/features/ai)
- [Code Review Manage code changes](https://github.com/features/code-review)
- [Code Search Find more, search less](https://github.com/features/code-search)
- [Codespaces Instant dev environments](https://github.com/features/codespaces)
- [Copilot](https://github.com/features/copilot): GitHub Copilot Write better code with AI
- [Copilot for business Enterprise-grade AI features](https://github.com/features/copilot/copilot-business)
- [Discussions Collaborate outside of code](https://github.com/features/discussions)
- [Issues Plan and track work](https://github.com/features/issues)
- [Models](https://github.com/features/models): GitHub Models New Manage and compare prompts
- [Spark](https://github.com/features/spark): GitHub Spark New Build and deploy intelligent apps
## Github
- [Home | github.com](https://github.com/github): GitHub’s organization on GitHub
- [Roadmap](https://github.com/github/roadmap)
- [(Updated 02/2024)02/2024](https://github.com/github/site-policy/pull/582)
....
Integrate and Benefit
You have a couple of options:
- Redirect: Configure your server to redirect LLM requests for
/llms.txtto theget.llms.pageURL. - Fetch and Serve: Fetch the content from
get.llms.pageyourself and serve it directly from your own/llms.txtendpoint.
Here’s why this is useful:
- Simple: Super easy to set up.
- No Maintenance: Update your site, and the
llms.txtupdates itself. - Future-Proof: New LLM standards? You’re covered, no manual tweaks needed.
