Lecture
What rebuilding AlphaGo teaches us about self-play, RL, and future of LLMs - Eric Jang
- Future projects like AlphaGo Zero are expected to become reproducible and trainable with significantly reduced compute costs (potentially a few thousand dollars of rented resources) compared to previous requirements of millions of dollars and large teams, potentially driven by LLM coding capabilities.
- Deep neural networks, approximated by 10-layer architectures, are predicted to amortize the simulation of extremely deep game trees previously considered intractable for search algorithms.
- While ResNets currently offer superior performance and cost-efficiency for small data regimes in Go, Transformers are expected to eventually outperform them when global context is required.
- Future AI research may revisit tree structures and forward search mechanisms for LLM reasoning, though uncertainty remains regarding the success of PUCKED heuristics in continuous action spaces.
- Test-time scaling and model size scaling are anticipated to interact profoundly, potentially enabling complex reasoning within a single forward pass rather than through explicit sequential search.
- As hardware improves (e.g., NVIDIA Blackwell GPUs), specific architectural innovations and auxiliary training tricks previously necessary for strong bots may become less critical for achieving high performance.
- Automated scientific research tools are expected to become highly effective at hyperparameter optimization and executing defined experiments but currently lack the lateral thinking required to select the next most valuable experiment.
- Deep learning principles derived from games like Go are projected to transfer positively to complex domains such as drug discovery or general AI research.
- Scaling laws may fail to appear or become predictive if the studied system is not yet "bug-free" or producing high-quality data.
- Future AI iterations will likely require "good initialization," such as pre-training against existing strong bots, as starting from scratch remains computationally expensive compared to leveraging existing knowledge.
- The "Bitter Lesson" of relying on scale is expected to eventually dominate, rendering specific algorithmic details less critical than the sheer amount of compute available, although this effect is currently transitory and hardware-dependent.
- Automated agents are expected to discover fundamental deep learning principles by optimizing for verifiable outer loops like win rates, provided they can navigate the "long horizon" problem of resisting early negative feedback.
- A separation between the "outer loop" (verifiable goals) and "inner loop" (complex engineering) is expected to persist, with the outer loop serving as a crucial stabilizer for automated research.
- The compute efficiency of training strong Go bots is expected to continue improving, potentially allowing strong agents to be trained on consumer hardware rather than requiring massive TPU pods.
- Models trained on smaller 9x9 boards are expected to offer transferability benefits, accelerating warm-start times for models operating on larger 19x19 boards.
- Variance in reinforcement learning training signals remains a critical bottleneck for LLMs until methods are found to provide granular, step-by-step supervision similar to MCTS relabeling.
- The transition from model-free RL to MCTS-based training is expected to enable more sample-efficient learning by providing a supervision target for every action rather than just the final outcome.
- The "chaotic" nature of complex systems allows for macroscopic predictions, such as win probability, even when microscopic details remain impossible to compute deterministically.
- Future research will likely involve "automated AI research" where agents design and train future systems, potentially leading to rapid, non-linear improvements.
- The "off-policy" nature of MCTS training is expected to remain stable and effective in Go for correcting trajectories and handling opponent-drifted states, unlike in some continuous control settings where such training can be unstable.