Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 13: Data (Sources, Datasets)

Data secrecy as competitive advantage

Companies like Meta disclose full architecture details in Llama 3 papers but deliberately omit data sources to protect competitive advantages and avoid copyright liability.

The human bottleneck in curation

Unlike systems engineering, data curation scales with human effort and remains a long-tail bottleneck requiring surprisingly large teams for cleaning and quality control.

Three-stage quality progression

The pipeline progresses from massive low-quality pre-training corpora, through mid-training with synthetic and high-quality web data, to specialized post-training on chat transcripts and task-specific environments.

The 'entire internet' fallacy

Pre-training uses static crawls rather than live web access, missing dynamic app-based content, authenticated platforms like Facebook and X, and pages behind paywalls.

Technical barriers to crawling

Robots.txt files increasingly block specific AI crawlers (OAI, Perplexity, Claude), while Cloudflare challenges, rate limits, and IP bans prevent automated data collection.

Surge in access restrictions

Research by Shane Lampray shows robots.txt restrictions jumped significantly after mid-2023, with terms of service increasingly prohibiting AI training usage across nearly half the web.

Automatic copyright protection

Under the 1976 Copyright Act, all web content is automatically protected for 75 years upon fixation, requiring no registration and covering virtually everything on the internet.

Licensing and public domain

Legal training data comes from public domain works (75+ years old), Creative Commons sources like Wikipedia, or expensive licensing deals with content platforms.

Shadow libraries and piracy

Repositories like LibGen and Anna's Archive provide massive copyrighted book and paper corpora but constitute clear copyright infringement and legal liability for model developers.