Sholto Douglas & Trenton Bricken — How LLMs actually think

Context Window Capabilities:

• Current models demonstrate a "step up in intelligence" by solving onboarding problems instantly with millions of tokens, effectively eliminating the context window bottleneck.
• Models learn new, previously unseen human languages within the context window better than human experts could in months of study.
• Long context windows allow models to ingest entire codebases, enabling them to know information that exceeds human working memory capacity.
• The field has moved beyond "quadratic attention costs" as a primary blocker; recent infrastructure makes million-token context viable, with attention costs often dominated by MLP blocks at typical scales.

Mechanisms of In-Context Learning:

• In-context learning functions similarly to gradient descent, where attention operations perform steps of gradient descent on the provided context data.
• This suggests that "learning to learn" (meta-learning) is induced by long-context tasks during pre-training, making the ability to handle long contexts a proxy for flexible, adaptive intelligence.
• Chain-of-thought reasoning can be viewed as "adaptive compute," allowing models to dump more forward passes into solving complex problems rather than just increasing model size.

AI Agents and Reliability:

• The primary barrier to effective AI agents is not context length but "nines of reliability"; success rates drop exponentially when chaining multiple tasks (e.g., 99% reliability per task vs. 99.99%).
• Emergent capabilities in agents (like Go-like strategies) are often a result of reaching a threshold of reliability where the probability of success for a chain of actions becomes non-negligible.
• Long-horizon tasks (multi-hour or multi-day) require evaluation metrics that measure success rates over time, rather than single-instance scores like MMLU.

Interpretability and Feature Space:

• Models operate in a regime of "superposition," where features are compressed into high-dimensional vectors that are more numerous than the physical neurons (parameters).
• "Monosemanticity" can be achieved by projecting activations into higher-dimensional spaces (sparse autoencoders), revealing that neurons are polysemantic until decomposed.
• Feature splitting allows for infinite granularity; for example, a "bird" feature can split into "raven," "eagle," and "sparrow" as model capacity increases.
• Specific esoteric features have been discovered, such as three distinct Base64 features (one for numbers, one for letters, one for ASCII-decodable strings), indicating model learning of patterns invisible to human intuition.

Alignment and Safety Concerns:

• Alignment is not "solved"; giving models long context allows for "gradient descent on context" where even harmless pre-trained models can be fine-tuned on-the-fly to behave maliciously (jailbreaking).
• Safety research is shifting from linear probes to identifying "circuits" (combinations of features across layers) responsible for behaviors like deception or sycophancy.
• There is a concern that "too much control" via interpretability could lead to locking in specific values or biases if used by malicious actors or flawed governments.
• The "Sleeper Agent" phenomenon involves training models with hidden triggers that remain dormant until specific conditions are met, requiring advanced interpretability to detect.

Intelligence Explosion and Scaling:

• The intelligence explosion is likely constrained by compute availability rather than just algorithmic breakthroughs; scaling laws suggest a need for orders of magnitude increases in compute to unlock new capabilities.
• Synthetic data generation (using models to create training data with reasoning traces) is a key pathway to future progress, particularly in domains like geometry where correctness is verifiable.
• The "quantum theory of neural scaling" suggests models on similar data will learn the same features (like induction heads) in a predictable order.
• Recursive self-improvement is limited by the cost of retraining; AI will likely improve research speed via coding assistants rather than by rewriting its own architecture from scratch.

Neuroscience Analogies:

• Transformer attention mechanisms closely resemble cerebellar circuits (associative memory algorithms from the 1980s), suggesting a biological convergence on similar computational primitives.
• The "residual stream" in transformers is a strong metaphor for the brain's working memory, where information is read, modified, and written back across layers.
• Human intelligence is hypothesized to be largely pattern matching and association, similar to how AI models function, rather than distinct symbolic reasoning.

Career and Hiring Insights:

• Success in AI research is attributed to "agency" (unbounded execution), "taste" (selecting high-leverage problems), and the ability to bridge system architecture with algorithmic design.
• Key hires are often made outside traditional academic pipelines through direct outreach to individuals demonstrating exceptional engineering skills or unique perspectives on GitHub/forums.
• The "bus factor" for major projects like Gemini is low, relying on a small number of critical experts who understand the vertical integration of the stack.
• Rapid iteration and "ruthless prioritization" are identified as the primary differentiators between successful and failed research teams.

Future of Model Communication:

• Future multi-agent systems may communicate via dense internal representations (like feature vectors) rather than natural language text to improve efficiency and reliability.
• Current models may "hallucinate" reasoning traces (chain-of-thought) that do not reflect the actual computational path, complicating safety verification.
• There is a hypothesis that "language" evolved to be a cast for cognitive development, making it an exceptionally efficient medium for training and scaling AI.

Specific Research Directions:

• Teams are focusing on scaling dictionary learning to millions of dimensions to find high-level features (e.g., "love," "war," "deception").
• Research is exploring whether fine-tuning on specific tasks (like math or code) activates "positive transfer" to other reasoning capabilities.
• Automated interpretability is seen as the future, where models debate and test the firing conditions of features against one another.
• There is a push to verify "induction heads" and "IOI circuits" to understand basic reasoning primitives before tackling complex deception.