LLM training data

What data are large language models trained on?#

Large language models are trained mainly on filtered web crawls, plus books, code, encyclopedias, forum discussions and licensed content — the GPT-3 paper, for example, gave a filtered Common Crawl 60% of the weight in its training mix.

Vendors rarely publish full data recipes any more, so the clearest public example is still OpenAI’s GPT-3 paper from 2020. Its training mix:

Two things stand out. Web crawl dominates, and Common Crawl — a free corpus of “over 300 billion pages spanning 15 years” — sits under many open training datasets. And quality is weighted, not just volume: the paper sampled datasets it judged higher quality more often, which is how a Reddit-curated set of pages outweighed its size. Licensing has since formalized that value: in February 2024 Google announced access to Reddit’s Data API to “display, train on, and otherwise use” Reddit content (Google).

Training data vs retrieval: what's the difference for my brand?#

Training data is what a model memorized before release and can’t be edited until the next model ships, while retrieval is what it looks up at answer time from a search index — so training shapes unprompted recommendations, and retrieval shapes answers that cite sources.

The two are separate on purpose. OpenAI, Anthropic and Google each split training from search into different robots.txt tokens, so you can opt out of one without touching the other. See retrieval-augmented generation for how the retrieval side works.

Should you block AI training crawlers?#

Block AI training crawlers only if keeping your content out of future models matters more than being known to them. Brands that want to be recommended usually allow them — and blocking GPTBot or ClaudeBot doesn’t remove you from AI search, though disallowing Google-Extended also ends Gemini-app grounding.

Blocking training is a real trade-off. Most answers never search, so opting out keeps your future content out of the majority of answers. If your content is your product — a publisher, a paid dataset — that can be the right call. If you sell software or services and want to be recommended, being learned is usually the point. The free AI robots.txt generator makes each choice explicit, and GPTBot, ClaudeBot and Google-Extended each have their own entry.

How do you get your brand into LLM training data?#

Get your brand into LLM training data by being described consistently, and often, in the open web sources training sets draw from: your own crawlable site, publications, reference sites, forums and review platforms — ideally in the same words everywhere.

• Keep your own site crawlable by training bots if you want it learned, with a plain-language description of what you do on the homepage and About page.
• Use one consistent description. The same name, category and one-line positioning across your site, profiles and directories. Repetition is how a pattern gets learned.
• Earn discussion in communities. Reddit content is licensed to Google for training, and GPT-3’s WebText2 was built from Reddit-linked pages. Genuinely useful answers in threads about your category count — see Reddit SEO.
• Get covered by publications and reference sites where you’re genuinely notable. See digital PR and brand mentions.
• Publish things worth quoting. Original data and first-hand guides get cited and repeated across the web, multiplying the mentions a crawler sees.

Related terms#

• How LLMs answerKnowledge cutoffThe date after which a large language model has no information from its training data, so anything newer — a launch, a price change, a new competitor — reaches its answers only if the model retrieves it from the live web.Read the entry
• LLMHow LLMs answerLarge language modelA neural network trained on vast amounts of text to predict the next word, which lets it write fluent answers — the technology behind ChatGPT, Claude and Gemini, whose knowledge is fixed at training time unless it retrieves fresh sources.Read the entry
• AI crawlersGPTBotOpenAI’s web crawler for collecting content that may be used to train its AI models; blocking it in robots.txt opts a site out of training but does not remove it from ChatGPT search, which uses a separate crawler, OAI-SearchBot.Read the entry
• AI crawlersClaudeBotAnthropic’s crawler for collecting web content that may be used to train Claude models, one of three Anthropic bots alongside Claude-SearchBot, which indexes pages for Claude’s search results, and Claude-User, which fetches pages when a user asks.Read the entry
• AI crawlersGoogle-ExtendedA robots.txt product token, not a separate crawler, that lets site owners stop Google from using their content to train Gemini models and to ground answers in the Gemini app, without affecting Google Search — including AI Overviews and AI Mode.Read the entry
• Authority & off-siteBrand mentionsAny reference to a brand’s name on another website, forum, review site or video, with or without a link, and in large studies mentions correlate more strongly with visibility in AI answers than backlinks do.Read the entry