Jan 27, 2018, 4:34 PM ETLecture

MIT 6.S094: Computer Vision

Computer vision relies heavily on deep learning, specifically supervised learning where neural networks map raw sensory data (pixel values) to human-annotated ground truth labels.
The primary challenge in autonomous driving perception is that input data often lacks sufficient context, specifically temporal dynamics and semantic understanding of dynamic scenes.
Standard image classification ignores the "context" and "humor" of a scene (e.g., a cat dressed as a monkey), failing to capture dynamic information that is crucial for robust perception.
The SegFuse competition is introduced as a cutting-edge initiative to advance deep dynamic driving scene segmentation, aiming to produce publication-worthy research.
Unlike previous competitions like Deep Traffic which explore deep reinforcement learning, SegFuse focuses on the intersection of computer vision and temporal dynamics for autonomous perception.

Illumination and Pose Variability: Changes in lighting and object orientation cause significant numerical differences in pixel data, making recognition difficult for current networks.
Intra-class Variability: Objects within the same class (e.g., different types of cats or dogs) exhibit high visual diversity, while variability between different classes can be minimal.
Occlusion: The 3D nature of the world leads to objects blocking one another, requiring algorithms to identify objects based on partial visibility (e.g., recognizing a cat from just an ear or leg).
Loss of Spatial Relationships: Convolutional Neural Networks (CNNs) utilize spatial invariance, which discards the specific spatial relationships between features (e.g., the arrangement of eyes and nose), potentially leading to errors in pose variation.
Data Scarcity in Temporal Domains: High-quality, pixel-level manual annotation for video data is prohibitively expensive (approx. 90 minutes per image), making fully segmented video datasets a major open problem.

AlexNet and VGGNet: Early successful architectures utilized uniform, stacked convolutional layers; VGGNet (16-19 layers) achieved high accuracy but required a massive parameter count (138 million).
GoogleNet (Inception): Introduced modules combining multiple convolution sizes (1x1, 3x3, 5x5) to capture features at different scales efficiently, reducing parameters while maintaining performance.
ResNet: Implemented residual blocks allowing networks to be trained much deeper by passing raw input directly through layers, bypassing the difficulty of training very deep networks; it achieved a 4% top-5 error rate on ImageNet, surpassing human error (5.1%).
SENet (Squeeze-and-Excitation): Achieved a significant error reduction (approx. 25%) by parameterizing channel weights, allowing the network to adaptively recalibrate feature responses based on input content.
Capsule Networks: A developing architecture (based on Jeff Hinton's work) that aims to preserve spatial hierarchies and pose relationships, addressing the information loss inherent in standard CNN pooling operations.

Fully Convolutional Networks (FCN): Repurposed classification networks by removing fully connected layers and adding decoders to produce pixel-level heatmaps, marking the start of dense semantic segmentation.
SegNet: Adopted the encoder-decoder framework specifically for driving contexts, utilizing skip connections to improve resolution compared to naive upsampling.
Dilated Convolutions: Utilize a gridded approach to expand the receptive field without reducing spatial resolution, preserving high-frequency textures while capturing broader context.
Conditional Random Fields (CRFs): Applied as a post-processing step to smooth segmentation outputs by analyzing underlying image intensities and refining boundaries.
Hybrid Dilated Convolution: Employs variable dilation rates to create smoother upscaling results by preventing certain input pixels from receiving disproportionately high attention.
Learnable Upsampling: The most critical advancement identified is parameterizing the upscaling filters (upsampling convolution) rather than using fixed methods like bilinear interpolation.

Objective: Improve frame-by-frame segmentation (produced by a current state-of-the-art network) by fusing it with optical flow data to create temporally consistent dynamic segmentation.
Data Provided:
- Original video footage (1080p and 8K 360) from Cambridge driving.
- Ground truth segmentation for a training set (10,000 images) annotated via Mechanical Turk.
- Pre-computed segmentation masks from a state-of-the-art FCN-based network.
- Dense optical flow maps generated using FlowNet 2.0.
Optical Flow Context: FlowNet 2.0 (Dec 2016) combines "Simple" and "Correlation" architectures to estimate pixel movement direction and magnitude between consecutive frames (30 fps), enabling the propagation of semantic information over time.
Evaluation Metric: Participants must submit code that reduces the discrepancy between the propagated segmentation and the ground truth provided.
Resources: Starter code (Python) and datasets are available at cars.mit.edu/sightfuse.
Expected Outcome: High-performing entries are likely to yield novel ideas leading to publications and global advancements in perception.

Capsule Networks and GANs: An online-only lecture covering these topics will be released separately due to their technical complexity.
Next Lecture: Scheduled for the following week on "Deep Learning for Sensing the Human."
Industry Event: A Waymo presentation is scheduled for the next day at 1 PM in Stata 321.
Research Opportunity: The field of dynamic scene segmentation remains largely open, particularly regarding the fusion of spatial texture with temporal motion data for robotic applications.