MIT 6.S094: Deep Learning for Human Sensing

Core Philosophy & Requirements for Real-World Deployment

• Data is the critical success factor: Real-world data collection and annotation are prioritized over algorithm design as the most difficult and essential components of successful deep learning systems.
• Algorithmic limitations: While deep learning algorithms (CNNs, RNNs, LSTMs) are exciting, system robustness and generalizability depend fundamentally on the volume and quality of the training data.
• Annotation tooling as a design challenge: Creating custom annotation tools for specific tasks (e.g., glance classification vs. body pose estimation) is a human-computation problem critical to system performance, not a standard computer vision step.
• Infrastructure requirements: Processing real-world datasets requires large-scale distributed compute and storage; the MIT group currently manages over 5 billion driving data images.
• Temporal dynamics: Effective algorithms must capture both spatial visual characteristics and temporal dynamics (sequences of images) to model the physics of the real world.
• Human-centered autonomy: The speaker argues that fully autonomous vehicles (Level 4/5) are at least two decades away; the immediate path involves L2 human-machine collaboration and overcoming human imperfections.

Human Imperfections & Safety Data

• Driver performance statistics: In the U.S., there were over 40,000 fatalities in 2016 (1 per 80 million miles traveled), with Massachusetts having the lowest risk and Montana the highest.
• Distraction risks:
  • 400,000 injuries and 3,000 deaths occurred in 2014 due to distraction.
  • 170 billion text messages were sent monthly in the U.S. in 2014.
  • Texting for five seconds at 55 mph covers the length of a football field.
• Specific behavioral hazards:
  • 31% of traffic fatalities involve a drunk driver.
  • 23% of nighttime drivers test positive for legal or over-the-counter medications.
  • Nearly 3% of fatalities involve drowsy driving.
• Systemic over-trust risks: Users of semi-autonomous systems (e.g., Tesla Autopilot) often bypass safety mechanisms, such as hanging objects (e.g., oranges) on steering wheels to trick touch sensors, indicating a need for perception systems that actively monitor human engagement.

MIT Naturalistic Driving Data Collection

• Dataset scale: The group utilizes 25 instrumented vehicles (21 Tesla Model S/X/3) collecting over 1,000 miles daily, amounting to thousands of miles under automated control and over 5 billion video frames.
• Sensor suite per vehicle:
  • Two cameras on the driver's face for high-definition video.
  • A fish-eye camera for full-body pose estimation.
  • A forward-facing camera for scene segmentation.
  • Synchronized GPS, audio, and CAN bus data.
• Usage metrics: Tesla Autopilot is currently active for 33% of total miles driven, indicating significant user value and enjoyment while maintaining physical engagement (eyes on road).
• Glance behavior findings: Aggregate data shows no significant difference in glance distribution between manual and autopilot driving, suggesting users remain attentive enough to take over when necessary.

Technical Approaches to Human Sensing

• Task difficulty spectrum:
  • Easier tasks (Green zone): Pedestrian detection, full body pose, face classification, head pose.
  • Harder tasks (Red zone): Micro-saccades, pupil diameter, cognitive load estimation, blink dynamics, and fine-grained eye pose.
• Pedestrian detection evolution:
  • Moved from inefficient sliding window approaches (HOG, Histogram of Oriented Gradients) to candidate generation networks (R-CNN, Fast R-CNN, Faster R-CNN).
  • State-of-the-art includes Mask R-CNN for segmentation and VoxelNet for 3D LiDAR point clouds.
• Body pose estimation:
  • Utilizes "DePose" (holistic view) to detect individual body parts (elbows, hands, hips) before connecting them via Part Affinity Fields to resolve multi-person occlusions.
  • Applied to monitor driver alignment (e.g., slouching) relative to standard airbag testing positions.
• Glance classification vs. Gaze estimation:
  • Converted the geometric "gaze estimation" problem into a supervised "region-based classification" problem (On-road vs. Off-road) to enable learning from data without precise geometric ground truth.
  • Regions include: On-road, off-road, left, right, center stack, rear-view mirror, and instrument cluster.
• Annotation efficiency:
  • Uses a human-in-the-loop pipeline where the neural network annotates low-confidence frames for human review.
  • High-confidence predictions can still be incorrect; continuous manual annotation of specific individuals (to account for glasses, lighting, hair) is required to correct false positives.

Emotion & Cognitive Load Detection

• Emotion recognition pipeline:
  • Distinguishes between general emotion (primary categories like joy, anger) and application-specific emotions (e.g., frustration with navigation).
  • Key Finding: In frustration experiments, the strongest indicator was a smile (often a "forced" or nervous smile), highlighting the complexity of mapping facial expressions to internal states.
  • Uses 42 facial muscle Action Units for feature extraction.
• Cognitive load detection:
  • Relies on blink dynamics and eye movement patterns (saccades vs. smooth pursuit) rather than pupil diameter due to uncontrolled real-world lighting variations.
  • Methodology: 3D Convolutional Neural Networks process sequences of 6-second eye video clips (15 fps) to classify load levels (e.g., "n-back" tasks).
  • Accuracy: Achieved 86% classification accuracy on real-world highway data.
  • Aggregate finding: Higher cognitive load correlates with reduced eye movement, as drivers fixate more intensely on the roadway.

Future Outlook & Competitive Events

• Mass market timeline: L2 automation is expected to penetrate the mass market in the early 2020s; full removal of steering wheels and humans is estimated 20+ years away.
• Personal Robot Paradigm: Self-driving cars should be viewed as personal robots requiring trust and communication, not just perception and control systems.
• Upcoming initiatives:
  • CHI 2018 Course: Will cover deep learning for human understanding including NLP, voice, and GANs.
  • Deep Traffic Competition: Participants must achieve 65 mph (with a 70 mph goal for high performers); submissions are due by the deadline, with results targeted for NIPS.
  • Data for AGI: The AGI class will feature speakers like Josh Tenenbaum, Ray Kurzweil, and Ilya Sutskever to explore natural and artificial intelligence architectures.
• Collaboration call: The team is seeking researchers to work on automotive deep learning problems, noting thousands of competition submissions and strong community support.