newsfilter.io
Conference Presentation, Keynote

MIT-AVT: Data Collection Device (for Large-Scale Semi-Autonomous Driving)

Lex Fridman
Watch on YouTube →

MIT Autonomous Vehicle Technology Study: Ryder System Overview

  • Mission Scope

    • Collects large-scale naturalistic driving data to analyze human supervision of semi-autonomous and autonomous systems.
    • Current fleet comprises 30+ vehicles, accumulating over 320,000 miles (averaging 500–1,000 miles daily).
    • Upcoming vehicle additions include the Tesla Model 3 and Cadillac CT6 with Super Cruise.

  • Hardware Architecture ("Ryder" Box)

    • Core Processing: Single-board computer running a custom Linux distribution designed for reliability across multiple vehicles and weather conditions.
    • Sensor Integration:
      • Three Logitech C920 webcams providing 1080p video at 30 fps (on-board H.264 compression).
      • Integrated Inertial Measurement Unit (IMU) and GPS receiver.
      • Vehicle Controller Area Network (CAN) transceiver for raw vehicle telemetry.
    • Storage & Power:
      • Onboard solid-state drive (SSD) capable of storing months of uncompressed video (estimated 100 petabytes per 100,000 miles raw).
      • CAN-controlled power board monitors vehicle status; automatically activates/deactivates based on ignition status to prevent battery drain.
      • 4G wireless module enables remote health monitoring (drive capacity, temperature, power usage).
    • Thermal Resilience: Tested cameras functioned correctly after exposure to 58–75°C (typical summer car temps) and 127°C (extreme stress test).
    • Optics: Custom cases allow interchangeable CS/CS-type lenses (fisheye and zoom) for expanded internal field of view.

  • Operational Logic & Reliability

    • Automated Triggering: System autonomously starts/stops recording based on CAN bus activity (ignition on/off) or specific triggers (door unlock, vehicle in drive).
    • Failure Management:
      • If camera cables disconnect, the subsystem attempts multiple restarts; persistent failure triggers a system-wide shutdown to prevent corrupted data gaps.
      • Primary failure mode identified: unplugged camera cables resulting in lost recording windows.
    • Cost Safety: Design ensures $0.01% of vehicle value impact (system cost ~$1k vs. vehicle cost >$100k), with wiring routed to prevent vehicle damage.

  • Data Synchronization & Pipeline

    • Clock Precision: Real-time clock maintains accuracy within 2 parts per million (max drift ~7ms over a 1.5-hour drive).
    • Synchronization Process:
      • Timestamps assigned to every sensor stream (video, GPS, IMU, CAN) at collection.
      • Post-offload pipeline uses these timestamps to align 30fps video with higher-frequency IMU/CAN data.
      • Synchronization is the critical prerequisite for all subsequent computer vision and deep learning algorithms.
    • Offloading Method: No remote wireless offloading; hard drives are physically swapped and connected locally to computers, then remotely copied to the data cluster.
    • Processing Workflow:
      1. Camera captures raw image and compresses to H.264 on-board (offloading CPU burden from Ryder).
      2. Data recorded to SSD (stored ~1 month for FastTrack subjects; ~6 months for Naturalistic Driving Study subjects).
      3. Post-offload cleaning removes corrupt data and fixes configuration errors.
      4. Cleaned, synchronized data enters deep learning training and manual annotation pipelines.

  • Data Volume & Metrics

    • Collected Volume: Billions of video frames across the fleet.
    • Storage Footprint: Compressed H.264 data totals nearly 300 terabytes.
    • Compression Efficiency: On-board H.264 encoding allows the use of lightweight single-board computers by shifting CPU-intensive processing to the cameras.

  • Future Development & Strategic Direction

    • Hardware Upgrade: Transitioning from current single-board computer to NVIDIA Jetson TX2 to support higher compute power and real-time processing.
    • Adaptive Data Collection:
      • Moving from "record everything" to selective recording based on onboard real-time analysis.
      • Goal: Discard "boring" driving data to prioritize edge cases and critical human-interaction events (glance allocation, cognitive load, smartphone usage).
    • Research Focus: Understanding human behavior as a supervisor in semi-autonomous environments to improve safety protocols and system design.
    • Internal Environment Emphasis: Prioritizing rich internal sensor data (driver face, body pose, glances) over solely external roadway sensing.