Lecture
Deep Learning State of the Art (2019)
- The year 2018 is predicted to be recognized as the pivotal year for natural language processing, analogous to the 2012 ImageNet breakthrough for computer vision.
- BERT development is expected to generate a significant leap in benchmarks and expand the application of natural language processing to solve new tasks.
- Encoder-decoder architectures utilizing recurrent neural networks are anticipated to remain effective for machine translation and processing arbitrary-length input and output sequences.
- Self-attention mechanisms are forecast to enable encoders to identify optimal input sequence aspects by analyzing the full context for specific words.
- The Transformer architecture, leveraging self-attention in encoders and attention in decoders, is expected to capture rich context for generating contextually appropriate output sequences.
- Significant community effort is hoped for in improving data augmentation techniques to maximize learning from limited data samples.
- Transfer learning is projected to allow data augmentation policies derived from ImageNet to be applied to entirely different datasets, though this requires further investigation.
- NVIDIA is expected to continue heavy investment in training deep neural networks with synthetic data to achieve state-of-the-art performance using small samples of real images.
- AutoML is anticipated to minimize human involvement by automating data augmentation and annotation tasks.
- The Dawn Bench competition is predicted to facilitate solutions to fundamental deep learning problems for academia and independent researchers lacking massive computational resources.
- Google DeepMind's BigGAN is expected to produce incredible high-resolution images through scaling model capacity and batch size rather than relying on novel breakthrough ideas.
- Video-to-video synthesis improvements are forecast to ensure temporal consistency, preventing jumpy output and enabling applicability to various mapping tasks.
- DeepLab v3 plus is expected to maintain state-of-the-art performance in semantic segmentation through multi-scale processing via dilated convolutions.
- AlphaZero is predicted to be remembered a century from now as a key historical moment in AI and general intelligence.
- OpenAI's Dota 2 team is expected to return with developments following the loss at the 2018 International.
- It is suggested that it will take a considerable amount of time before AI can achieve victory in general team Texas No Limit Hold'em.
- TensorFlow 2.0 is expected to release in 2019 with features including eager execution and other developments.
- Geoffrey Hinton posits that the future of deep learning depends on a graduate student deeply suspicious of current methods, specifically regarding the limitations of backpropagation.
- By 2019 and beyond, the audience is expected to define the actual breakthroughs and state of the art in the field.