Conference Presentation, Tutorial
Nuts and Bolts of Applying Deep Learning (Andrew Ng)
Lex FridmanAndrew Ng, Hugo Larochelle, Andrej Karpathy, Richard Socher, Sherry Moore, Ruslan Salakhutdinov, John Schulman, Pascal Lamblin, Adam Coates, Alex Wiltschko, Quoc Le, Yoshua Bengio, Shubho Sengupta, lexfridman, Peter, Andre, Shubo, Sammy
- Project patterns observed at Baidu are expected to be most useful for work undertaken in the coming weeks or months, amidst a shifting workflow for organizing machine learning projects in the deep learning era.
- Societal advancement past the data curve over the last decade or two is attributed to the rise of the Internet, mobile, and IoT, driving a trend where large neural nets absorbing large amounts of data yield the best performance, whereas small to medium nets yield slightly to moderately better performance respectively.
- In small data regimes, algorithm performance ordering remains undefined and contingent on feature engineering quality, while the "more data, bigger model" formula is considered sufficient for many problems, though end-to-end deep learning requires immense input-output label data and may fail in data-scarce scenarios like predicting a child's age from X-rays or implementing fully end-to-end self-driving systems today.
- Optimal performance increasingly necessitates co-locating AI and computer systems (HPC) teams, as it is difficult for a single individual to master both fields, and bringing together these distinct communities allows for superior outcomes.
- Current industry value is driven primarily by General DL, Sequence models, and Image models, which have the highest probability of application success on Mondays, while the "other" bucket encompassing unsupervised and reinforcement learning is anticipated to represent the future of AI.
- Deep reinforcement learning algorithms are expected to have an even greater hunger for data than supervised learning, and while automatic data synthesis via engineered parameters (such as blurring for OCR or adding background sounds for speech) can dramatically improve systems, synthetic datasets may appear visually authentic while being impoverished for learning algorithms.
- Strategic team practices include establishing a unified data warehouse, ensuring development and test sets share the same distribution, and avoiding changes to a specified test set once defined; when distributions mismatch, a specific "train-dev" set from the training distribution and a separate dev/test set from the target distribution are recommended best practices.
- Estimating the base error rate becomes significantly harder once models surpass human-level performance, making a team of expert doctors with a 0.5% error rate the preferred definition for mathematical analysis and progress tracking, while tasks a person can complete in under one second are considered highly automatable via deep learning.
- Future researchers are expected to develop expertise by reading 20 to 50 papers and replicating results, combined with engaging in data cleaning, debugging, and tuning, with a year of weekend study without short-term rewards cited as a path to deep learning mastery.
- The speaker anticipates that AI will transform a wide range of industries similar to electricity 100 years ago, with the path to societal impact in transportation, healthcare, and logistics becoming increasingly clear, while the coupling between bias and variance is expected to weaken in the deep learning era.
- Predicting the outcome of the next event in a sequence, such as ad clicks or delivery times, is identified as a major application bucket creating significant value, and the trend of end-to-end deep learning enabling complex outputs like sentences and images is noted to work well in some contexts but is not a universal solution.