Conference Presentation, Keynote
Sequence to Sequence Deep Learning (Quoc Le, Google)
Lex FridmanQuoc Le, Hugo Larochelle, Andrej Karpathy, Richard Socher, Sherry Moore, Ruslan Salakhutdinov, Andrew Ng, John Schulman, Pascal Lamblin, Adam Coates, Alex Wiltschko, Yoshua Bengio, Shubho Sengupta
- Plans include developing sequence-to-sequence learning systems for applications such as email smart reply, translation, image captioning, summarization, speech transcription, and Q&A, with a production implementation already existing in Google Inbox.
- Training objectives involve running stochastic gradient descent for approximately one million iterations or steps to process email data and update parameters, with training often stopping at 400 steps due to computational expense, though deep networks of six to eight layers are expected to improve results.
- System architecture employs 2,000-dimensional vectors for vocabulary representation, hidden states of 100 dimensions requiring a couple of hours to train, and aims for the largest possible decoder size despite potential speed reductions.
- Variability in input and output sequence lengths is addressed by using recurrent networks with variable node counts per sample, predicting tokens sequentially until an end token is generated, and utilizing attention mechanisms to re-examine hidden states with dynamic coefficients rather than fixed orderings.
- Technical constraints include the necessity of auto-differentiation tools in TensorFlow, Torch, or Theano, handling gradient explosion or imploding by clipping values at 10, and managing variable-size inputs via normalization or packages like Stanford NLP when datasets are small.
- Specific performance strategies involve combining algorithms to filter yes/no responses based on thresholds before generating full replies, using schedule sampling to mitigate derailing risks, and applying heuristic beam search diversity algorithms for selecting top answers.
- Model limitations and risks include potential social friction from binary "yes/no" responses lacking nuance, slow execution on cell phones, and the current inability of the basic autoregressive solution to know previous predictions without recurrence.
- Comparative predictions indicate the sequence-to-sequence model achieves state-of-the-art results in translation on WMT datasets but currently underperforms compared to CTC for speech recognition, with online speech decoding remaining a work in progress.
- Data requirements for training range from three to five million sentence pairs to reach state-of-the-art performance, up to tens of millions of pairs for larger translation corpora, with potential for handling over 20,000 choices and multiple languages using vocabularies around 100,000.
- Future research directions suggest advancements in memory and operation augmentation within the near future, the use of skip-thoughts for document-level generalization, and combining modalities or unsupervised learning to address common sense reasoning.
- Implementation details note that initial systems lack user embeddings to remain general, while future iterations may incorporate user vectors to resolve context issues, and that pre-training word vectors or using dropout can improve performance on scarce data.
- Optimization approaches explore reinforcement learning for sequence-level training to improve global metrics like BLEU, though human preference may still favor standard model outputs, and joint training with autoencoders is expected to work well.