Lecture, Presentation
Exponential Progress of AI: Moore's Law, Bitter Lesson, and the Future of Computation
- AI progress is projected to follow trajectories that may or may not sustain exponential advancement, with the long-term driver likely being a combination of human ingenuity and raw computational power.
- General automated methods leveraging big compute are expected to remain superior to specialized, human-expertise-injected approaches relying on small compute, with the latter anticipated to fail as resources scale over the next 5, 10, or 20 years.
- Brute-force learning and search techniques are predicted to outperform other approaches as computation grows, provided algorithms are perfectly parallelizable across thousands to billions of processors to achieve exponential improvement.
- Researchers should prepare for evaluation metrics that test linear scalability with compute increases of 10X or 100X over the next 5 to 10 years, noting that current algorithmic improvements may receive disproportionate respect compared to raw computational power.
- Total global compute capacity is expected to potentially explode exponentially in the near or long term, driven by the expansion of computing surfaces via gaming consoles, smartphones, and the Internet of Things.
- Virtual reality and augmented reality devices are anticipated to gain ground, making substantial computational resources available for virtual and augmented worlds.
- Hardware efficiency is expected to improve through the use of application-specific integrated circuits (ASICs) and general-purpose GPUs, which offer better energy use and algorithm performance than classical hardware.
- Quantum and neuromorphic computing are currently in very early stages, facing significant difficulties in building large-scale systems or designing algorithms that outperform classical AI methods.
- Efficiency gains in deep learning domains are expected to continue increasing exponentially, far outpacing the improvement rates of Moore's Law, which is defined as the doubling of transistor count every two years.
- While industry experts like Jim Keller expect Moore's Law to continue due to unmet theoretical physics limits, a majority of the industry believes the trend of doubling transistor counts every two years is dead.
- Human ingenuity is considered essential for continued exponential improvement if algorithmic gains continue to outpace hardware efficiency improvements.
- Innovation in active learning is expected to increase exponentially, with frameworks like Tesla's autopilot continuously discovering and learning from edge cases to improve performance over time.
- Scalability of learning methods remains unclear without a natural coupling to data annotation scalability, though deep learning methods require much smaller and more directed amounts of human expertise compared to classical machine learning.
- High-bandwidth, two-way brain-computer interfaces are expected to enable society to leverage human brain computation to add to global capacity, potentially achieving exponential AI growth without significant algorithmic innovation if successful.
- In the short term, brain-computer interfaces are expected to assist in understanding and treating neurological diseases, while in the long term, they could completely change the nature of computation and artificial intelligence.
- The scientific community is expected to have previously underestimated the impact of general brute-force methods relative to their actual contribution, despite the evolutionary process leveraging computation effectively despite appearing wasteful to humans.
- Future exponential improvement of AI is expected to be determined by the interplay of human ingenuity and raw computational power, with humanity expected to continue adapting well to the exponential movement.