Lecture
What rebuilding AlphaGo teaches us about self-play, RL, and future of LLMs - Eric Jang
- Project Scope & Motivation: Eric Zhang rebuilt AlphaGo from scratch during a sabbatical to understand how a 10-layer neural network can approximate an intractable search problem, specifically the game of Go, which was previously deemed computationally impossible to solve via exhaustive search.
- Compute Efficiency Trends: An open-source project, KataGo (2020), achieved a 40x reduction in compute needed to train a strong Go bot compared to AlphaGo Zero; modern LLM coding capabilities now allow building similar systems for ~$3,000 in compute, down from the millions required by DeepMind.
- Go Game Mechanics: Go involves placing black and white stones to occupy territory; capturing occurs when stones are completely surrounded on four neighbors (or three on an edge), with rules variations including Chinese, Japanese, and Trump-Staylor (unambiguous algorithmic scoring).
- Game Complexity: A 19x19 Go board has a branching factor of ~361 decreasing over 250–300 moves, resulting in a game tree far larger than the number of atoms in the universe, requiring pruning strategies rather than exhaustive enumeration.
- Monte Carlo Tree Search (MCTS): The core algorithmic breakthrough of AlphaGo is using MCTS to prune the search tree, selecting actions based on a balance of exploitation (maximizing known value, Q) and exploration (visiting untried actions), utilizing the PUCT (Predicted Upper Confidence with Trees) formula.
- Neural Network Architecture: AlphaGo utilizes a dual-headed neural network (typically ResNet or Transformer) where one head predicts the policy (probability distribution over moves) and the other predicts the value (probability of winning from a given state); the value head acts as a heuristic to replace deep simulation with a single forward pass.
- Training Pipeline (AlphaGo Lee): Initial training uses supervised learning on human expert game data to initialize the policy and value networks; subsequent training involves self-play where the network plays against itself to generate data.
- Training Pipeline (AlphaZero/CataGo): Subsequent iterations (AlphaZero, KataGo) remove human data, using "tabula rasa" self-play where the value function learns to predict outcomes from random play, initially struggling on mid-game states but improving via MCTS bootstrapping.
- MCTS as Data Generator: MCTS serves as an "improvement operator" that relabels actions taken by a policy with better alternatives found via search; the policy network is then retrained via supervised learning to imitate these MCTS-derived distributions (soft labels), creating a stable feedback loop.
- Value Function Learning: The most difficult aspect of training is learning the value function for mid-game states; end-game states are easily resolved via rule-based scoring, while early states are roughly 0.5 win probability, leaving the mid-game as the primary challenge for the value head.
- Off-Policy vs. On-Policy: While standard RL often requires on-policy data, Go training tolerates off-policy updates (states visited by previous policies) because MCTS can relabel old states with current optimal actions, acting as a form of "hindsight planning" or DAgger-style correction.
- Scaling Laws & Test-Time Compute: Research (e.g., Andy Jones, 2021) indicates a trade-off where increased inference-time compute (more MCTS simulations) can replicate the performance gains of increased training-time compute; modern hardware allows reducing training complexity by increasing simulation depth at test time.
- LLM Comparison: Unlike LLMs that rely on high-variance policy gradient RL (learning from sparse final rewards), Go's MCTS provides a low-variance, dense supervision signal for every move, making it a more sample-efficient learning paradigm for sequential decision-making.
- Chaos & NP-Hardness: Deep neural networks effectively "amortize" simulation of NP-hard problems by learning macroscopic structures (like the Lorenz attractor) rather than simulating every micro-step, suggesting that worst-case complexity may not reflect practical solvability for structured problems like Go or protein folding.
- Automated Research Observations: Using LLM agents (e.g., Claude 4.6) for coding and hyperparameter tuning successfully optimized model performance and executed experiments, but struggled with high-level "lateral thinking" to pivot research directions or identify fundamental bottlenecks without human intervention.
- Infrastructure Simplification: Modern hardware (NVIDIA Blackwell GPUs) and improved initialization strategies (e.g., pre-training on 9x9 boards) allow for simplified, synchronous training setups, removing the need for the complex distributed asynchronous systems required by earlier iterations like KataGo.
- Soft Labels Importance: Training the policy to match the full MCTS probability distribution (soft targets) is superior to training on just the single best move (one-hot), as it preserves more information ("dark knowledge") and stabilizes learning, a key factor in AlphaGo's success.
- Future Research Directions: Zhang suggests Go remains a viable sandbox for testing automated AI research, specifically for developing agents that can perform "search-like" reasoning, with the potential for transferable skills to robotics and drug discovery via verifiable outer loops (win rates).