Dec 12, 2020, 11:30 PM ETInterview

Michael Littman: Reinforcement Learning and the Future of AI | Lex Fridman Podcast #144

Michael Littman cites the movie Robot and Frank as a profound influence, appreciating its near-future depiction of domestic robots where users can mold the technology to their own quirky needs.
Littman admits he has "no musical taste" in the traditional sense, noting that his preference for songs is based on familiarity, often leading to liking a track only after repeated exposure via weekly Billboard top 10 lists.
He participated in a TurboTax commercial after a family friend, an advertiser, recruited him as a "B-level scientist" to demonstrate that the software does not require expert knowledge.
Littman enjoys producing parody songs for his own education or class endings; the most challenging was a Piano Man parody about the Halting Problem due to its internal rhyme scheme.
His most significant philosophical takeaway on music was learning to listen to layers and individual instruments, a skill developed during his time in a jazz fusion band in college.

Littman rejects the existential threat of superintelligence, arguing that the "intelligence explosion" scenario lacks a fundamental understanding of the complexity required to build robust systems that operate in the world.
He posits that creating AGI requires a progressive, evolutionary process where we learn the mechanisms of intelligence, making it unlikely to "spring into existence" as a fully formed, uncontrollable entity.
He identifies social media algorithms as a current, tangible form of "collective intelligence" that is already manipulating human behavior, potentially leading to societal self-destruction through toxicity and chaos.
Littman believes the "TikTok generation" is uniquely responsible for navigating the balance between utilizing social media's benefits and preventing it from overwhelming them.
He argues that the primary risk is not AI destroying itself, but rather humans failing to control or understand AI systems fast enough as they develop, similar to the rapid deployment of nuclear weapons.

Littman's career began in the 1980s with a TRS-80 computer, initially attempting to teach it to play tic-tac-toe and later studying neural networks in a psychology context.
He was introduced to Temporal Difference (TD) learning by Rich Sutton after meeting mentor Dave Ackley, which resolved earlier limitations in making predictions over time.
He credits Gerald Tesauro's TD-Gammon as a remarkable breakthrough, noting it could learn backgammon strategies through self-play without human expert data, though he acknowledges Tesauro's unique ability to coax results from neural networks.
Littman views the AlphaGo and AlphaZero achievements as engineering marvels that successfully integrated diverse techniques, though he suggests the strategic depth of Go may eventually hit an asymptotic ceiling similar to chess.
He expresses skepticism about the "Bitter Lesson" argument (that simple algorithms + massive compute always win), noting that hardware development costs are rising and S-curves suggest diminishing returns on pure exponential growth.

Littman emphasizes that large language models like GPT-3 are limited by their lack of direct human interaction and the inability to be "pushed back" or corrected in real-time, which hinders true conversational intelligence.
He highlights the immense difficulty of self-driving cars, specifically the need for "theory of mind" to predict the erratic, social signaling of pedestrians and other drivers.
He suggests that driving is a social interaction that requires co-evolution between human behavior and vehicle capabilities, a complexity often underestimated in academic modeling.
Littman advocates for a world where programming is a universal literacy, citing Douglas Rushkoff's Program or Be Programmed as a foundational text on the democratization of technical power.
He is currently reading Brian Christian's The Alignment Problem, which he finds effective at contextualizing the history of AI fairness, reinforcement learning, and the broader philosophical implications of superintelligence.

Littman considers "balance" the meaning of life, illustrated by a 42nd birthday party where guests used physical objects (unicycles, pogo sticks) to represent their personal equilibriums.
He contrasts the rigorous, often slow-paced academic approach to AI research with the risk-tolerant, "move fast and break things" ethos of capitalist startup culture, noting that both are necessary for progress.
He finds the process of teaching his children to drive to be a profound encounter with mortality, as he must constantly manage their safety while they learn to interpret complex social signals on the road.
Littman recommends Ted Chiang's short story collection Exhalations for its ability to extrapolate scientific concepts into mind-warping, insightful narratives.
He predicts that the next phase of AI development will rely on systems that can efficiently learn from low-bandwidth human interaction rather than purely self-play or massive data scraping.