Efficient Computing for Deep Learning, Robotics, and AI (Vivienne Sze) | MIT Deep Learning Series

Compute and Environmental Impact Trends

• Deep learning compute requirements have increased by over 300,000 times in recent years to drive accuracy improvements.
• The carbon footprint of training neural networks exceeds that of a trans-continental flight or an average human life by orders of magnitude.
• Moore's Law and Dennard Scaling have slowed or ended, causing transistor efficiency trends to flatten over the past decade.

Edge Computing Drivers and Constraints

• Moving computation from the cloud to the edge is necessary due to limited communication infrastructure, data privacy/security needs (e.g., healthcare), and latency requirements for autonomous navigation.
• Self-driving vehicles consume over 2,000 watts for computation alone, generating significant heat requiring water cooling and large physical footprints.
• Portable devices are limited to under 1 watt of power for computation due to battery size, weight, and heat dissipation constraints, contrasting with embedded platforms that often consume over 10 watts.

Hardware Efficiency Fundamentals

• Data movement dominates power consumption in AI systems; moving data is significantly more expensive than the compute operations themselves.
• Reading a 32-bit value from off-chip DRAM consumes 640 picojoules, which is orders of magnitude higher than the energy for a single floating-point multiply.
• A single Multiply-and-Accumulate (MAC) operation requires four memory accesses (weights, activations, partial sums, and output), creating a 4:1 memory-to-compute ratio.

Data Flow Architectures and IRIS Chip

• Traditional data flows like "weight stationary" (TPU, NVIDIA NVDLA) minimize weight movement but increase activation and partial sum traffic.
• MIT developed the "Row Stationary" data flow in the IRIS chip to balance data movement across all types (weights, activations, partial sums).
• The IRIS chip reduces global buffer accesses by 100x and off-chip memory accesses by over 1,000x compared to standard architectures.
• Compared to a mobile GPU, the IRIS chip achieves an order-of-magnitude reduction in energy consumption.

Algorithmic Optimization Strategies

• Standard pruning (removing small magnitude weights) yields a 2x energy reduction, whereas "energy-aware pruning" (removing high-energy weights) yields a 3.7x reduction for the same accuracy.
• The number of MACs is a poor predictor of latency; networks with identical MAC counts can have up to a 2x latency difference.
• The NetAdapt framework uses empirical latency and energy measurements to automatically adapt neural network dimensions (e.g., channel reduction) to specific hardware budgets.
• NetAdapt achieves up to a 1.7x speedup over manually designed networks for the same accuracy.

System-Level Acceleration Beyond DNNs

• FAST Framework: Accelerates super-resolution by up to 16x by exploiting temporal correlations in compressed video, processing only a subset of frames while reconstructing the rest using video structure data.
• Navion Chip: A specialized chip for Visual Inertial Odometry (VO) that reduces energy consumption by 2–3 orders of magnitude compared to CPUs, consuming under 25 milliwatts.
• Robotics Exploration: A specialized memory architecture enables computing Shannon's mutual information for robot exploration on a 200m x 200m map in under one second, a 100x speedup over CPUs.

Healthcare and Sensor Applications

• Eye-tracking algorithms on consumer smartphones (iPhone) can measure saccade latency with accuracy comparable to clinical phantoms, enabling low-cost, in-home monitoring of neurodegenerative diseases.
• Efficient algorithms allow depth estimation using Time-of-Flight (ToF) sensors active only 11% of the time (once every ninth frame), maintaining <0.7% mean relative error while drastically reducing sensor power.

Key Takeaways

• Efficient computing is critical for extending AI beyond the cloud to robotics and healthcare, enabling privacy and low latency.
• Cross-layer design integrating specialized hardware, algorithmic pruning, and data-aware architecture is essential to overcome the "memory wall."
• Future progress requires moving beyond counting MACs to optimizing for actual data movement, latency, and energy consumption.