Why Scale Will Not Solve AGI | Vishal Misra - The a16z Show

Current LLM Capabilities and Limitations

• LLMs function as "grains of silicon doing matrix multiplication" and lack consciousness, an inner monologue, or survival instincts.
• They are driven exclusively by the objective function to predict the next token accurately based on training data, not by biological imperatives like reproduction or self-preservation.
• Current models perform "Shannon entropy" (correlation) but have not yet achieved "Kolmogorov complexity" (finding the shortest program to reproduce a string) or true causation.
• LLMs operate within a fixed, frozen manifold learned during training; they cannot generate new "manifolds" or reject existing axioms to create fundamentally new scientific representations.

Mathematical Modeling of LLMs (Bayesian Wind Tunnel)

• LLMs can be abstracted as a gigantic, sparse matrix where rows represent every possible prompt and columns represent probability distributions over the vocabulary.
• In-context learning functions as real-time Bayesian updating: as the model sees evidence (examples), it updates its posterior probability distribution of the next token.
• Researchers created a "Bayesian wind tunnel" using small models on tasks impossible to memorize but analytically solvable to prove the mechanism.
• Results showed Transformers match the precise Bayesian posterior down to $10^{-3}$ bits of accuracy, while Mamba models perform reasonably well, LSTMs partially, and MLPs fail completely.
• This Bayesian signature persists in production-scale models (hundreds of millions of parameters), though the signal is noisier due to messy training data.
• The ability to perform this updating is an architectural feature of the transformer, not merely a result of the training data.

Historical Context and Early Implementation (2020)

• In October 2020, the speaker implemented the first production Retrieval Augmented Generation (RAG) system for the cricket database "Stats Guru" at ESPN using GPT-3.
• This system converted natural language queries into a custom Domain Specific Language (DSL) via few-shot learning and semantic search.
• The speaker designed the DSL and a dataset of 1,500 query pairs, which the model had never seen during training, proving the model could learn the task in real-time.
• This initial work empirically demonstrated the model updating its token probability distribution toward the correct DSL syntax as more examples were provided.

Divergence Between Human and Machine Intelligence

• Humans exhibit lifelong plasticity and "don't die/reproduce" optimization, whereas LLMs have frozen weights after training and reset context with every new session.
• Human cognition involves simulation and causal modeling (intuition based on physical laws), while LLMs rely on statistical correlation and pattern matching.
• Unlike humans, who can build internal causal simulators to predict interventions and counterfactuals (Judea Pearl's hierarchy levels 2 and 3), LLMs are currently limited to association (level 1).
• The "Einstein Test" proposes that AGI requires training a model on pre-1916 physics and verifying if it can derive the theory of relativity; current LLMs cannot do this as they cannot discard old axioms to form new causal manifolds.

Pathways to AGI and Future Research

• Achieving AGI requires solving two specific problems: implementing true continual learning (plasticity) and moving from correlation to causation.
• Scaling parameters alone will not solve the issue; a new architecture capable of updating weights without catastrophic forgetting is required.
• Current attempts at "generalization" (e.g., Donald Knuth using an LLM for Hamiltonian cycles) are "hacked" plasticity where the model improves context but does not alter its internal weights.
• Future research directions focus on creating algorithms that reduce Kolmogorov complexity by building causal simulators rather than expanding the training manifold.
• Judea Pearl's "do-calculus" and causal hierarchy are identified as the necessary theoretical frameworks for bridging the gap between correlation and simulation.

Community Reception and Tools

• The "Bayesian wind tunnel" papers have received positive feedback, with other researchers successfully reproducing the experiments using the archive versions.
• The speaker released "TokenProbe" (tokenprobe.cs.columbia.edu), an open interface allowing users to visualize next-token probabilities and entropy during inference.
• Google Research is acknowledged as moving in the same direction with recent work on teaching LLMs vision learning via RLHF, indicating a shift toward causal understanding.
• Despite the scientific validation, the speaker notes a "political backlash" in the field regarding the application of the term "Bayesian" to machine learning, stemming from the historical frequentist-Bayesian divide.