Fireside Chat, Interview
Will LLMs Get Us To AGI?
- Definition of AGI: Martin and Vishal define Artificial General Intelligence as the capability to generate entirely new scientific paradigms, mathematics, and results that extend beyond the bounds of existing training data.
- An AGI system must create "new manifolds" rather than merely navigating or interpolating within the manifolds of its training set.
- Current Large Language Models (LLMs) are characterized as sophisticated Bayesian reasoners that refine existing knowledge but cannot fundamentally rewrite the rules of science.
- The Matrix Abstraction Model: Vishal introduces a formal mathematical model where an LLM is represented as a massive, sparse matrix containing next-token probability distributions for every possible prompt.
- The matrix dimensions exceed the number of atoms in the observable universe, necessitating compression where the model only represents a sparse subset of known rows.
- New prompts are handled via Bayesian interpolation, where the model uses the prompt as "new evidence" to compute a posterior distribution based on learned variants.
- In-Context Learning and Bayesian Reasoning: The model equates in-context learning directly to Bayesian inference, where examples provided in the prompt serve as evidence to update the posterior distribution.
- This mechanism explains "few-shot" learning without requiring weight updates; the architecture processes examples and new queries using the same underlying inference logic.
- The model demonstrates that LLMs can learn new, unseen structures (like a custom DSL) immediately if the prompt provides sufficient contextual evidence.
- Entropy and the "Manifold" of Confidence: LLM reasoning occurs within a reduced-dimensional geometric manifold where the model maintains high confidence as long as it stays within the learned distribution.
- Low Prediction Entropy: Occurs when prompts are information-rich or follow learned algorithms (e.g., step-by-step math), leading to a collapse of possible tokens and high confidence.
- High Prediction Entropy: Occurs with vague prompts or novel concepts, resulting in a diffuse distribution and increased likelihood of hallucination.
- Hallucination Mechanism: Occurs when the model veers away from the learned manifold, producing "confident nonsense" by forcing a high-probability token that does not align with reality.
- Limits of Recursive Self-Improvement: Vishal argues that LLMs cannot achieve recursive self-improvement or AGI through data iteration alone because their output is the inductive closure of their training data.
- Models can only "fill in the blanks" of existing knowledge (e.g., solving new math problems using known axioms) but cannot invent new axioms or fundamental scientific laws (e.g., Relativity, Quantum Mechanics).
- Even multiple models interacting without external information cannot generate new information; they merely recombine existing manifold paths.
- Historical Context and RAG Invention: Vishal details his accidental invention of Retrieval-Augmented Generation (RAG) in 2020 while attempting to query the "Stats Guru" cricket database via GPT-3.
- Unable to fit the complex database schema into GPT-3's 2,048-token context window, he built a Natural Language to DSL translator that retrieved relevant examples to act as a prompt prefix.
- This system achieved high accuracy in production by September 2021, demonstrating the efficacy of context-based learning before RAG became a formalized industry standard.
- Architectural Critique and Future Direction: The consensus is that current transformer architectures are reaching a plateau where scaling data and compute will only smooth existing manifolds rather than creating new ones.
- Vishal compares current progress to the iPhone lifecycle: incremental improvements in capability (camera, memory) without fundamental functional shifts.
- He advocates for a new architectural leap (e.g., energy-based models, multi-modal simulation) that mimics human few-shot learning and physical simulation rather than language-based probability.
- Industry Reception and Methodology: Vishal notes a cultural divide between the AI community's heavy reliance on empiricism ("prompt engineering" or "prompt twiddling") and his approach of formal modeling and theory.
- He cites instances where formal models explaining why LLMs work were rejected by conferences in favor of large-scale empirical experiments, despite the models' lack of theoretical understanding.
- He critiques "prompt engineering" as distinct from true engineering, arguing it relies on trial-and-error tuning rather than system reliability.
- Verification Tools: Vishal promotes "Token Probe," a tool developed with a16z that visualizes entropy levels in real-time.
- The tool allows users to observe the confidence of an LLM rise as the prompt provides more information, visually validating the entropy reduction and manifold navigation theory.
- Benchmarks for AGI: Vishal suggests that an LLM creating a large-scale software project without human supervision would be a significant milestone, though likely still within the bounds of existing code patterns.
- True AGI validation requires the model to produce outputs that fall completely outside the distribution of its training data, effectively generating a new scientific or mathematical paradigm.