Lecture, Conference Presentation
MIT 6.S094: Deep Learning for Human-Centered Semi-Autonomous Vehicles
- Data Collection Gap: Current semi-autonomous and autonomous vehicles lack driver-facing video data, which is critical for training deep learning models to understand human-machine interaction.
- Scope of Data: Researchers are collecting billions of video frames of humans driving 60+ mph in semi-autonomous Teslas to analyze driver behavior.
- Computer Vision Challenge Map:
- Green (Feasible): Face detection and body pose are well-studied, robust even in poor lighting, variable pose, and noisy environments.
- Red (Hard): Micro-saccades (eye tremors occurring ~1,000 times/second) and subtle occlusions remain extremely difficult for computer vision systems.
- Trust Mechanism: Vehicles currently possess minimal awareness of the occupant (relying only on seatbelt or seat pressure sensors), hindering the ability to build the trust necessary for full autonomy.
- Safety Recommendation: The speaker advocates for mandatory driver-facing cameras in all vehicles despite privacy concerns, citing massive safety and trust benefits.
- Body Pose Application:
- Current crash test dummies assume static, optimal body positions, ignoring real-world variability like reaching for bags or phones.
- Detecting body pose allows the car to adjust passive safety systems for the actual occupant position at the moment of a crash.
- Pose Estimation Methodology:
- Utilizes Convolutional Neural Networks (CNNs) as regressors to output XY coordinates for skeletal landmarks (e.g., shoulders, head).
- Employs a cascade of regressors and temporal constraints (physics of movement) to minimize error across frames.
- Alternative approach uses 3D CNNs that process sequences of frames simultaneously rather than individually.
- Gaze Classification:
- A classification problem using CNNs to predict six driver attention areas: forward roadway, left, right, center stack, instrument cluster, and rearview mirror.
- Requires video stabilization and noise removal to counter vehicle vibration and lighting fluctuations.
- Data Annotation Strategy:
- Transitioning from fully supervised learning (human-labeled data) to semi-supervised/unsupervised methods.
- Automation: Algorithms handle annotation for the 90%+ of driving that is repetitive (staring forward).
- Human Intervention: Humans are only required to label "corner cases" (e.g., partial occlusions, extreme lighting), achieving high accuracy with an 84-fold reduction in human annotation effort.
- Emotion Detection Findings:
- Training data links specific facial cues (e.g., eyebrow movement, smiling) to self-reported frustration levels.
- Counter-intuitively, "frustrated" drivers often exhibit smiling, whereas "satisfied" drivers appear stoic and cold due to the boredom of driving.
- Solo vs. Passenger Data: Driver behavior and emotional expression differ significantly when alone versus with passengers; solo data lacks social performance cues.
- Cognitive Load Analysis:
- Metrics: Uses eye movement patterns, specifically saccades (jumping movements) and smooth pursuit, to gauge mental workload.
- Limitations: Pupil dilation is unreliable in real-world driving due to interference from ambient light changes.
- Blinking: Higher cognitive load correlates with decreased blink rates and shorter blink durations.
- Implementation: 3D CNNs analyze sequences of eye images (e.g., 90 frames over 6 seconds) to classify cognitive load into low, medium, or high categories.
- Emergent Learning Complexity:
- Neural network performance improvements with depth remain theoretically unexplained.
- The speaker draws an analogy to Conway's Game of Life, where simple local rules in individual cells generate complex, life-like global patterns, suggesting similar emergent behavior in deep networks.
- Competition Results:
- Three winning neural networks were identified for a self-driving car game competition.
- One deterministic algorithm achieved a score of 74, beating previous bests of 73.
- Winners received a free term at the Udacity Self-Driving Car Engineering course.
- Future Steps: The speaker encourages reliance on community-driven resources (deep learning books, GitHub repositories, blogs) and open-sourcing winning competition networks.