Sequence to Sequence Deep Learning (Quoc Le, Google)

Motivation and Basic Setup

• The speaker addresses the problem of auto-replying to emails (e.g., "yes" or "no") using a sequence-to-sequence (Seq2Seq) learning framework.
• Initial approach involves converting email content into fixed-length 20,000-dimensional vectors via frequency counting (bag-of-words), which loses word order information.
• The task is formulated as a binary classification problem solvable via logistic regression and stochastic gradient descent (SGD), updating a weight matrix $W$ to maximize the probability of the correct reply.

Limitations of Bag-of-Words and Introduction of Recurrent Networks

• Bag-of-words representations fail to capture the sequential ordering of words, which is critical for semantics.
• To address this, Recurrent Neural Networks (RNNs) are introduced to preserve ordering information through hidden states $h_t$.
• In an RNN, the hidden state at time $t$ depends on the previous hidden state $h_{t-1}$ and the current word vector, utilizing shared matrices $U$ and $V$.
• Training requires backpropagation through time to update parameters, often facilitated by auto-differentiation toolkits like TensorFlow, Torch, or Theano.

Sequence-to-Sequence Architecture

• The problem is generalized to mapping variable-length inputs to variable-length outputs (Encoder-Decoder framework).
• Encoder: An RNN processes the input sequence (e.g., an email) and compresses it into a fixed-length context vector.
• Decoder: A second RNN predicts output tokens sequentially, conditioned on the encoder's context and previously predicted tokens.
• The training objective maximizes the conditional probability of the correct output token $y_t$ given all previous inputs and outputs (autoregressive property).
• Special "end" tokens are used to signal the termination of the output sequence.

Decoding Strategies

• Greedy Decoding: Selects the most likely token at each step; fast but prone to error accumulation and suboptimal global sequences.
• Beam Search: Maintains $K$ candidate sequences (e.g., $K=3$ or $10$) at each step, calculating joint probabilities to select the most likely sequence.
• To prevent exponential growth in candidates, beams with low probabilities are pruned (truncated beam search).
• Practical beam sizes of 3 to 10 are sufficient for high-quality results in production systems like Gmail Smart Reply.

Handling Vocabulary and Training Challenges

• Out-of-Vocabulary (OOV): Words outside the vocabulary (e.g., "Barack Obama") are mapped to an "unknown" token; sub-word segmentation or character-level modeling can mitigate this.
• Schedule Sampling: During training, the model is sometimes fed its own predicted outputs instead of ground truth to prevent exposure bias and allow recovery from errors.
• Personalization: User-specific behavior is incorporated by adding a user embedding vector as the initial hidden state of the encoder.

Attention Mechanisms

• The fixed-length context vector bottleneck in standard Encoder-Decoder models limits performance on long sequences.
• Attention allows the decoder to dynamically focus on relevant parts of the input sequence at each decoding step.
• The mechanism computes a weighted sum of encoder hidden states, where weights (attention scores) are determined by the compatibility between the current decoder state and encoder states.
• This approach achieves state-of-the-art results in machine translation (e.g., WMT datasets) by handling one-to-many mappings and non-linear alignments.

Advanced Architectures and Extensions

• LSTM and GRU: Long Short-Term Memory and Gated Recurrent Unit cells are used to mitigate vanishing gradient problems and capture long-term dependencies.
• Deep RNNs: Stacking multiple RNN layers (e.g., 4 to 8 layers) improves representation power and translation quality.
• Gradient Clipping: Gradients are clipped at a specific threshold (e.g., magnitude of 10) to prevent exploding gradients during training.
• Augmented Memory: Research focuses on augmenting RNNs with external memory (e.g., Neural Turing Machines, Memory Networks) for explicit read/write operations, enhancing capabilities in Question Answering.
• Neural Programmers: Augmenting networks with arithmetic operations (addition/subtraction) to perform symbolic reasoning on numerical data.

Applications and Practical Considerations

• Applications: Seq2Seq with attention is applied to machine translation, image captioning, speech transcription, summarization, and conversational AI.
• Speech Recognition: While attention improves transcription, Connectionist Temporal Classification (CTC) remains superior for many speech tasks due to its handling of many-to-one mappings and online decoding capabilities.
• Online Decoding: For real-time speech or search, models can be adapted to decode block-by-block rather than waiting for the full input.
• Data Requirements: Effective training typically requires millions of sentence pairs (e.g., 3–5 million for English-German translation); transfer learning and pre-trained embeddings help when data is scarce.
• Multilingual Support: A single model can handle multiple languages if the vocabulary is expanded and user embeddings capture specific language behaviors.

Future Directions and Research

• Unsupervised Learning: Techniques like Skip-Thoughts and document-level autoencoders are explored to learn representations without labeled data.
• Common Sense Reasoning: Addressing the lack of world knowledge in text by integrating multimodal data or soft rule-based systems.
• Global Optimization: Emerging research uses sequence-level training (e.g., reinforcing BLEU scores) rather than per-step prediction to align with human preferences.
• Context Resolution in Q&A: Personalized user embeddings and injected context tokens are proposed to resolve ambiguities in large corpus questions.