Lecture
Efficient Computing for Deep Learning, Robotics, and AI (Vivienne Sze) | MIT Deep Learning Series
- Deep learning compute requirements have increased exponentially over the past few years, growing by over 300,000 times to drive task accuracy, leading to severe environmental carbon footprints exceeding transatlantic flights or an average human life.
- Migration of compute from the cloud to the edge is necessary to avoid reliance on weak communication infrastructure, ensure privacy for sensitive healthcare data, and meet latency constraints for interactive applications like autonomous navigation at high speeds.
- Self-driving cars currently consume over 2,000 watts for computation, generating significant heat requiring water cooling, while shrinking device form factors to portable sizes will make power challenges more severe due to limited battery size, weight, and cost.
- Embedded platforms for current applications consume over 10 watts, an order of magnitude higher than the sub-watt limit for handheld devices due to heat dissipation, as transistor efficiency has flattened for the past decade with slowed Moore's law and Dennard scaling.
- Data movement dominates power consumption in deep learning, with reading a 32-bit value from off-chip DRAM consuming 640 picojoules, an order of magnitude higher than reading from on-chip SRAM, and in worst cases being two orders of magnitude more expensive than the computation itself.
- Deep neural networks require several hundred million operations and weights, representing a two to three orders of magnitude increase in complexity compared to video compression, with weighted sum computations accounting for over 90% of operations.
- Convolutional neural networks are more computationally complex due to increased dimensionality involving channels, filters, and batch sizes, and models are trending deeper with more weights, creating a bottleneck where memory access limits high throughput despite parallelism in multiplies and accumulates.
- Processing a 700 million MAC network like AlexNet could require three billion DRAM accesses if no data reuse is exploited, whereas a 1-kilobyte local memory beside the multiply-and-accumulate engine allows for cheap data access that is 200 times less energy-intensive than off-chip DRAM.
- Different data flow approaches balance different data movements: "weight stationary" minimizes weight energy but increases input/activation movement, "output stationary" minimizes partial sum movement but increases weight/activation movement, and "input stationary" focuses on minimizing input data movement.
- The IRIS chip's row stationary data flow is expected to reduce accesses to the global buffer by 100 times and off-chip memory by over 1,000 times, predicting an order of magnitude reduction in energy consumption compared to a mobile GPU.
- Transitioning from state-of-the-art HOG features (pre-2011) to AlexNet or VGG increases energy consumption by two to three orders of magnitude, leaving even specialized hardware far less efficient than video compression on phones.
- Counting weights or MACs is not a reliable metric for energy consumption; for GoogLeNet, weights alone account for only 22% of overall energy, while "energy aware pruning" is predicted to yield a 3.7x reduction in energy compared to the 2x reduction from traditional magnitude-based pruning for the same accuracy.
- Latency is not directly correlated with the number of multiplies and accumulates, as networks with the same latency can have a 3x swing in MAC counts, prompting the development of the NetAdapt algorithm to automatically tailor networks to specific platforms for given latency or energy budgets.
- NetAdapt can produce networks up to 1.7x faster than manually designed approaches for the same accuracy, and combining it with compact network design and depth-wise decomposition is expected to increase the frame rate for depth estimation by an order of magnitude.
- Hardware designers require flexible architectures to support various algorithmic approaches like pruning and reduced precision, as specialized DNN hardware often suffers from low array utilization when processing efficient networks like depth-wise layers lacking channel reuse.
- The second version of IRIS supports a wide range of filter shapes and sparsities, achieving an order of magnitude improvement in speed and energy efficiency through a hierarchical mesh on-chip network enabling support for multicast to unicast data delivery.
- The "FAST" framework for super-resolution is predicted to accelerate any algorithm by an order of magnitude using structured redundancy in compressed video, achieving a 16x acceleration by transferring processing to 16 frames with only a slight drop in quality.
- The Navion chip performs complete visual inertial odometry on-chip at below 25 milliwatts, achieving a two to three orders of magnitude reduction in energy consumption compared to mobile or desktop CPUs by reducing data movement and applying low-cost compression.
- Specialized hardware for mutual information computation provides a 100x speed up compared to a CPU at a tenth of the power, enabling the calculation of Shannon's mutual information for robot exploration on a 200m x 200m map at 0.1m resolution in under a second.
- Eye movement evaluations for neurodegenerative diseases can be performed on consumer-grade cameras like an iPhone 6, showing distributions similar to phantom cameras, while depth estimation algorithms can use a monocular camera with depth sensors active only 11% of the time to achieve a mean relative error of 0.7 percent.
- These depth estimation algorithms can run on a VGA resolution at 30 frames per second on a Cortex-A7 processor, enabling AI to extend beyond the cloud by reducing communication costs, enabling privacy, and providing low latency.
- Educational materials, including slides, a book on efficient neural net processing with Joel Emmer, a 90-minute NeurIPS tutorial on YouTube, and a 6.825 course at MIT, are expected to be available soon.