What is llms.txt, and should you care about it?

In essence, llms.txt is a way to optimize your website for AI consumption, making it easier for LLMs to understand and utilize your content.

llms.txt is a proposed standard for websites to provide information to large language models (LLMs) in a structured, human-readable format.

It’s a simple text file, like robots.txt or sitemap.xml, that helps LLMs understand a website’s content more effectively. It essentially acts as a “treasure map” for AI, highlighting important information and filtering out noise like navigation menus or ads.

(Fig: Example llms.txt page for Brandkit’s demo account https://demo.brandkit.com/llms.txt)

Here’s a more detailed explanation:

Purpose:

llms.txt aims to improve how LLMs understand and process website content, making it easier for them to access key information and provide more accurate responses.

Format:

It’s a text file, typically in Markdown format, located at the root of a website (e.g., yourdomain.com/llms.txt).

Content:

It contains a summary of the website’s most important content, often including sections like a project summary, key information, and lists of relevant files.

Benefits:

By providing a simplified view of the website, llms.txt helps LLMs avoid the complexities of parsing HTML, JavaScript, and other elements. This can lead to more relevant and accurate information retrieval from AI chatbots and other LLM-powered systems.

Relationship to other files:

llms.txt complements robots.txt (which controls crawling) and sitemap.xml (which lists URLs) by focusing on providing content information for LLMs.

Current Status:

While not yet officially supported by all major LLMs, llms.txt is an emerging standard gaining traction as AI models become more prevalent.

More reading

• https://searchengineland.com/llms-txt-proposed-standard-453676
• https://ahrefs.com/blog/what-is-llms-txt
• Youtube video https://youtu.be/l-7GU1fhh-E?si=WCWrwt-NpLZbBOQ2