Nuts and Bolts of Applying Deep Learning (Andrew Ng)

Organizational Strategy

• Baidu leads a 1,000-person AI team covering vision, speech, NLP, autonomous driving, and medical diagnosis.
• Teams are increasingly organized with adjacent Machine Learning and High-Performance Computing (HPC) groups to manage specialized knowledge barriers.
• A unified company-wide data warehouse is mandated to avoid siloed data accumulation and streamline access.

Trend 1: Scale as the Primary Driver

• Performance of deep learning models continues to improve linearly with scale, whereas traditional algorithms (SVM, logistic regression) plateau as data volume increases.
• Large neural networks are required to absorb massive datasets; small or medium nets often fail to outperform traditional methods in big data regimes.
• Small data regimes still allow hand-engineered features (e.g., SVMs) to compete or win if data is scarce.

Trend 2: End-to-End Learning for Rich Outputs

• The field is shifting from predicting single numbers (regression/classification) to generating complex sequences (text captions, audio waveforms, translations).
• End-to-end architectures are effective for tasks like speech recognition and image captioning but require massive labeled datasets to bypass the need for intermediate feature engineering (e.g., phonemes).
• Constraints on End-to-End:
  • Fails in data-scarce scenarios (e.g., predicting child age from X-rays) where domain knowledge (bone length) provides a necessary intermediate step.
  • In self-driving cars, hybrid approaches (perception + explicit planning) are currently preferred over full end-to-end steering due to safety and data volume requirements.

Diagnostic Workflow: Bias, Variance, and Distribution Mismatch

• Standard Metrics: Track five key error rates: Human-level error, Training error, Train-Dev error, Dev-set error, and Test-set error.
• High Bias (Underfitting): Indicated by large gaps between human-level error and training error; solvable by increasing model size, training longer, or trying new architectures.
• High Variance (Overfitting): Indicated by low training error but high dev/test error; solvable by regularization, early stopping, or collecting more data.
• Train-Test Mismatch: Occurs when training and test data come from different distributions; solvable by synthesizing data to match the test distribution or collecting more real test-distribution data.
• Dev-Set Overfitting: If dev error is low but test error is high, the team has over-optimized for the specific dev set; requires more dev data.

Human-Level Performance as a Benchmark

• Human-level performance serves as a proxy for the theoretical "Bayes rate" (optimal error rate).
• Progress is easiest to drive when algorithms are below human-level performance, as human error provides a clear baseline for bias-variance analysis.
• Defining the Baseline: For driving technical decisions, the error rate of a "team of expert humans" (e.g., multiple expert doctors collaborating) is the most useful definition, as it best approximates the optimal error rate, despite being harder to collect data for.

Data Synthesis and Engineering

• Automatic Data Synthesis: Generating synthetic labels (e.g., pasting text on random backgrounds for OCR, mixing clean audio with noise for speech) allows for near-unlimited training data.
• Synthesis Risks: Synthetic data can be "impoverished" compared to real-world data (e.g., video game cars may look realistic to humans but fail to capture real-world distribution for AI).
• Requires significant engineering effort to fine-tune synthesis parameters (blurring, contrast) to match real distributions.

Product Design Heuristics for AI

• Rule of Thumb 1: Tasks a typical human can complete in less than one second (perception tasks) are high-probability candidates for deep learning automation.
• Rule of Thumb 2: Tasks involving predicting the outcome of a sequence of repeated events (e.g., ad clicks, delivery times) are highly lucrative due to the availability of vast training data.

Career Development Advice

• The "PhD Process": The most reliable path to becoming a researcher is reading 20–50 papers and replicating their results to internalize patterns.
• Dirty Work: Success requires engaging in unglamorous tasks (data cleaning, debugging, parameter tuning) while simultaneously reading papers; avoiding this "dirty work" stalls progress.
• Long-Term Consistency: Career advancement is driven by sustained, unpaid effort (e.g., studying on weekends for a year) rather than short-term rewards.