Deep Learning State of the Art (2020)

Historical Context and Origins

• Artificial intelligence is framed as the modern engineering realization of the ancient desire to "forge the gods," contrasting the 300,000-year history of modern humans with the mere 300 years of the industrial revolution and the 60–70 years of AI specifically.
• Alan Turing predicted by the year 2000 machines would pass the Turing Test for natural language and predicted that machines would eventually "take control" through self-play and sharpening their own wits.
• Early foundational work includes Frank Rosenblatt's 1957 single-layer perceptron and 1962 multi-layer perceptron, alongside Soviet researchers Alexei Ivakhnenko and V.G. Lapa (1965) who introduced multi-layer learning algorithms.
• Key architectural evolutions spanned from backpropagation (1970s) and recurrent neural networks (1970s–80s) to Convolutional Neural Networks (CNNs, 1980s–90s) and LSTMs, culminating in the "deep learning" revolution starting in 2006 with ImageNet and AlexNet.

Recent Milestones and Awards (2017–2019)

• The 2019 Turing Award was granted to Yann LeCun, Geoffrey Hinton, and Yoshua Bengio for making deep neural networks a critical component of computing.
• Seminal "win" events include IBM's Deep Blue defeating Gary Kasparov in chess (1997) and AlphaGo defeating Lee Sedol in Go (2016).
• The natural language processing (NLP) field was dominated by the Transformer architecture (2017), followed by BERT (2018) and a proliferation of variants like GPT-2, XLNet, and Megatron LM (2019).
• 2019 marked a shift toward highlighting the limitations of deep learning, with public discourse (e.g., Rodney Brooks) predicting the end of the deep learning era, while conference submissions (CVPR, NeurIPS, ICER) continued to grow year-over-year.

Framework Convergence and Tooling (2020 Outlook)

• TensorFlow 2.0 and PyTorch 1.3 converged features, with TensorFlow adopting eager execution (imperative programming) as default and PyTorch introducing TorchScript for graph representation.
• Support for Python 2 ended on January 1, 2020, for both major frameworks.
• Mobile and edge computing capabilities expanded via TensorFlow Lite (Android/iOS) and experimental PyTorch Mobile/TPU support.
• The speaker advocates for framework-agnostic research and easier model migration between PyTorch and TensorFlow, noting current transfer processes remain time-consuming (3–5 hours).

Natural Language Processing Trends

• Transformer-based models (BERT, GPT-2, XLNet, Albert, Megatron) dominate benchmarks, with Megatron LM utilizing 8.3 billion parameters compared to GPT-2's 1.5 billion.
• The "Write with Transformer" tool demonstrates that current models memorize statistical patterns rather than possessing true conceptual understanding, often failing at basic reasoning tasks (e.g., "2+2=3, 5=6").
• The Alexa Prize highlights that current dialogue systems struggle with multi-domain context switching and maintaining long-term conversation flow compared to human conversation.
• Future NLP goals include integrating common sense reasoning, extending context windows to tens of thousands of words, and achieving true open-domain conversation.

Reinforcement Learning (RL) and Self-Play

• OpenAI 5 achieved a 99.9% win rate against the 2018 version of themselves after 45,000 years of simulated self-play training on Dota 2.
• DeepMind's AlphaStar reached Grandmaster level in StarCraft II in late 2019 by learning through camera-based observation (mimicking human input) rather than direct game API access.
• Carnegie Mellon University's Pluribus defeated world-class human players in six-player No-Limit Texas Hold'em using self-play and action abstraction.
• Robotic manipulation achieved a Rubik's Cube solution via Deep RL using automatic domain randomization (ADR) to handle perturbations and noise.

Autonomous Vehicles: Competing Paradigms

• Tesla (Level 2): Uses a camera-based, deep-learning-centric approach ("deep learning is the cake") requiring heavy data and human supervision; the speaker notes the "data engine" (active learning loop) is the core competitive advantage.
• Waymo (Level 4): Uses a LiDAR, map, and sensor-fusion approach where deep learning aids perception ("deep learning is the icing on the cake"); relies on high-definition maps and safety drivers or teleoperation fallbacks.
• Key Debate: The "hard problem" of autonomy remains whether perception (Tesla view) or action/planning (Waymo view) is the bottleneck.
• Open Questions: The speaker questions if Level 2 systems can maintain driver vigilance as performance improves and if simulation-to-reality transfer can solve the full spectrum of edge cases.

Scientific Foundations and Theory

• The "Lottery Ticket Hypothesis" suggests dense neural networks contain smaller, trainable subnetworks that achieve similar accuracy, pointing toward more efficient architectures.
• Theoretical work at ICML 2019 proved that disentangled representations are impossible without explicit inductive biases.
• "Double Descent" phenomena observed by OpenAI show test error can decrease a second time after a model fully overfits the training set, applying to model size, training time, and data scale.
• Graph Neural Networks (GNNs) and Bayesian Deep Learning are highlighted as critical areas for solving combinatorial problems and providing uncertainty estimates in small-data regimes.

Ethics, Politics, and Societal Impact

• Recommendation systems (Twitter, Facebook, YouTube, Google Play) are identified as the most powerful AI force on society, controlling information flow and potentially human thought, yet remain the least published upon.
• The American AI Initiative launched to guide government regulation, funding, and education, though early discourse is described as "awkward" and lacking technical depth.
• The speaker argues for democratizing AI to prevent concentration of power in the hands of tech company executives, likening the need for ethical stewardship to George Washington's relinquishment of power.
• Skepticism and criticism are deemed healthy for the field, but the speaker urges the community to focus on "solid research" and "doing" rather than "anti-hype" or "hype."

Future Outlook for 2020 and Beyond

• Research Priorities: Integration of common sense reasoning, active/lifelong learning, open-domain dialogue, medical applications, and algorithmic ethics (fairness, bias, privacy).
• Robotics: Continued development in deep RL for manipulation and legged robots, with a hope for "RL psychology" departments to study emergent social behaviors in multi-agent simulations.
• Philosophical Stance: The speaker argues that "thinking" and "feeling" are human constructs; if machines can convincingly simulate them, they effectively "do" them, though the ethics of AI suffering remain observer-dependent.
• Ultimate Risk: The speaker posits that AGI itself is not the primary existential threat, but rather humans using AI as tools for control, surveillance, and warfare.