Interview
Jeremy Howard: fast.ai Deep Learning Courses and Research | Lex Fridman Podcast #35
- Jeremy Howard's early programming experience began with a Commodore 64 BASIC program designed to test musical scales for harmonic accuracy against the standard 12-tone system.
- Howard's deep connection to music was interrupted by repetitive strain injury (RSI), forcing a reduction in hands-on playing despite his lifelong background with saxophone, clarinet, piano, guitar, and drums.
- Howard identifies Microsoft Access (VBA) as his favorite historical programming environment due to its superior graphical interface for combining databases, forms, and event handlers, a level of integration he feels no modern platform has matched.
- He expresses regret that modern relational database programming is cumbersome, often requiring Object-Relational Mappings (ORMs) and complex server setups compared to the streamlined experience of Access.
- Howard praises the F# language for its ability to integrate database schemas directly into the type system, offering auto-completion for tables and fields without external boilerplate.
- Delphi (Object Pascal) is highlighted as a former favorite for combining the speed of compiled code with the ease of use found in Visual Basic, a balance he hopes Swift will eventually replicate.
- J is identified by Howard as his current favorite programming language, described as an array-oriented, APL-derived language where entire programs can be expressed in a single screen of math-like notation.
- The proprietary K language, a sibling to J, is noted for its extreme performance, utilizing only L3 CPU cache and costing $100,000 per CPU for use in hedge funds, though it is inaccessible to most developers.
- Perl was Howard's primary tool for his 1990s email startup "Fast Mail," but he attributes its decline to the cessation of leadership by Larry Wall, whereas Python succeeded due to its robust ecosystem of data science libraries despite being less elegant.
- Howard argues that Python's performance limitations and "unhackable" nature at the C level stifles innovation in deep learning, particularly for optimizing Recurrent Neural Networks (RNNs) and implementing sparse convolutional networks.
- The future of programming in AI, according to Howard, relies on MLIR (Multi-Level Intermediate Representation) and domain-specific languages (DSLs) built on frameworks like Halide to allow high-level expressive code (resembling J/APL) to compile efficiently for GPU hardware.
- Howard criticizes Google TPUs for being intentionally unprogrammable to protect intellectual property, forcing users into inefficient, high-level abstractions that prevent direct hardware optimization.
- Fast AI was founded to solve the bottleneck of deep learning accessibility, specifically to enable domain experts (like doctors) with data to implement solutions without requiring years of computer science training.
- The greatest opportunity for AI in medicine lies in developing on-device diagnostic and triage tools for developing nations (e.g., Africa, India) where specialist shortages are critical, such as the five pediatric radiologists available for the entire African continent outside of South Africa.
- Howard asserts that automating medical diagnosis entirely is counterproductive; instead, AI should act as a triage system to magnify the productivity of the limited number of human experts available.
- Regulatory and legal hurdles, specifically the misinterpretation of HIPAA by hospital lawyers who fear liability over data sharing, are identified as the primary barriers to progress in medical AI, rather than the legislation itself.
- Howard critiques the industry's reliance on massive datasets, arguing that transfer learning allows state-of-the-art results with significantly less data, reducing privacy risks and infrastructure costs compared to the "bigger is better" approach of Google and IBM.
- The DawnBench competition saw Fast AI win by training ImageNet models in 10 hours (later 18 minutes in a distributed version) and CIFAR-10 in minutes, utilizing techniques like smaller input resolutions (64x64) and multi-GPU setups previously thought unnecessary for accessibility.
- Howard contends that multi-machine training is largely a waste of time that slows iteration speed, asserting that major deep learning breakthroughs like Batch Norm and Dropout were achieved on single GPUs.
- Fast AI has released smaller subsets of ImageNet, including "ImageNet the French" and "ImageWolf" (dog breeds), which allow researchers to achieve transferable results and iterate rapidly on a single GPU.
- "Superconvergence," discovered by Leslie Smith, allows neural networks to be trained 10 times faster using significantly higher learning rates and cyclical schedules, a discovery Howard notes was initially rejected by academic reviewers for lacking a theoretical explanation.
- The future of hyperparameter tuning involves moving away from manual learning rate adjustments toward automated systems that interpret gradient changes to set parameters dynamically.
- Howard recommends PyTorch with the Fast AI library for beginners and practitioners due to its interactive debugging capabilities, dynamic graphs, and high-level API that reduces boilerplate.
- TensorFlow is criticized for its complex, disorganized codebase and poor performance when attempting interactive development, with Howard predicting Swift for TensorFlow will eventually supersede it by leveraging MLIR for better compiler optimizations.
- Students can complete the two Fast AI courses in two months (approx. 70 hours of work) or a year, with the primary bottleneck being coding proficiency rather than statistical knowledge.
- Howard's core advice for becoming an expert is to train and fine-tune models on domain-specific data, emphasizing that the key differentiator for success is tenacity rather than innate mathematical brilliance.
- Successful startups, according to Howard, should be self-funded to avoid the pressure of rapid growth demands from Venture Capitalists, allowing the founder to focus on solving actual problems rather than creating artificial milestones.
- Howard uses spaced repetition (Anki) specifically for learning Chinese, relying on memory cues like stories to retain the language, whereas for concepts and code, he prefers deep conceptual understanding over rote memorization.
- He advocates for a daily commitment of at least half the day to learning or practicing new skills, a habit that compounds over decades to accelerate problem-solving capabilities.
- Howard expresses deep concern regarding AI-driven labor force displacement and the resulting economic hollowing out of the middle class, viewing it as a significant societal risk.
- He urges data scientists to take ethical responsibility for their models, specifically regarding human-in-the-loop constraints, feedback loops, and ensuring explainability for those impacted by algorithmic decisions.