Deep Learning for Natural Language Processing (Richard Socher, Salesforce)

Core Concepts and Definitions

• Natural Language Processing (NLP) is defined as the intersection of computer science, AI, and linguistics, with the primary goal of enabling computers to perform useful tasks (e.g., question answering) via language understanding.
• Full human-level language understanding is considered "AI-complete," requiring the integration of visual inputs, thought processes, and situational world knowledge.
• Traditional NLP analyzes language hierarchically: speech $\rightarrow$ phonemes $\rightarrow$ morphology $\rightarrow$ syntax $\rightarrow$ semantics $\rightarrow$ discourse.
• Deep learning bypasses explicit morphological and syntactic analysis, directly mapping inputs to semantically useful representations, reducing the need for heavy linguistic inspiration in system design.
• NLP difficulty stems from ambiguity (e.g., "I made her duck" has multiple meanings) and the necessity of coreference resolution (e.g., determining who "she" refers to based on verb context).

Representations: Word Vectors

• Discrete representations like WordNet taxonomies are insufficient for capturing linguistic nuance, new vocabulary (e.g., "Ninja"), and domain-specific updates requiring constant human labor.
• Word vectors utilize distributional semantics, representing words as continuous n-dimensional vectors based on their co-occurrence statistics in large corpora.
• Word2Vec (2013): Predicts context words given a center word (or vice versa) using a simple neural network with an inner product and softmax; allows for fast, online training.
• GloVe (2014): Combines global co-occurrence matrix statistics with neural network objectives to efficiently predict log probabilities of word pairs; trained on massive datasets like Common Crawl.
• Word vectors of size 100–300 dimensions capture semantic analogies linearly (e.g., vector("king") - vector("man") + vector("woman") $\approx$ vector("queen")).
• Syntactic relationships (e.g., superlatives: slow $\rightarrow$ slower $\rightarrow$ slowest) also emerge as consistent linear transformations within these vector spaces.
• Performance in word vector tasks scales directly with data volume; 300-dimensional vectors generally yield optimal results for semantic and syntactic relationships.

Sequence Modeling: Recurrent Neural Networks (RNNs)

• Standard RNNs tie weights across time steps, allowing the model to condition the current hidden state on all previous words in a sequence.
• Language modeling is a key application, where the model predicts the next word to minimize perplexity (average log likelihood of the next word).
• Standard RNNs suffer from gradient vanishing and the inability to retain long-term sentiment information (e.g., "wonderful" at the start of a long review) due to continuous state modification by intermediate content words.
• Gated Recurrent Units (GRUs) address these limitations using two gates:
  • Reset Gate: Determines how much of the previous hidden state to forget when computing new content.
  • Update Gate: Balances the contribution of the new state versus the previous state.
• GRUs allow specific dimensions of the hidden state to be preserved across many time steps, enabling the model to capture long-range dependencies and complex sequence patterns.

Advanced Architectures and Research

• Pointer-Sentinel Mixture Model: A new architecture designed to predict words never seen during training by allowing the model to "point" to previous context words in addition to standard softmax predictions.
  • Achieves a perplexity of 70 on language modeling, down from 78 (Wojciech Zaremba) and 87 (Tomas Mikolov).
  • Addresses the limitation of standard softmax layers, which cannot predict out-of-vocabulary words at test time.
• Dynamic Memory Networks (DMNs): An end-to-end trainable architecture aimed at reducing diverse NLP tasks (QA, sentiment, POS tagging) to a unified question-answering framework.
  • Comprises four modules: Input (GRU), Question (GRU), Episodic Memory (Meta-GRU with attention), and Answer (GRU + Softmax).
  • Iterates over input sentences multiple times ("episodes"), using attention mechanisms to selectively aggregate relevant facts for complex reasoning (e.g., transitive inference).
  • Demonstrates state-of-the-art performance on the Facebook bAbI dataset (logical reasoning), sentiment analysis, and Part-of-Speech tagging.
• Visual Question Answering (VQA): The DMN architecture successfully adapts to computer vision by replacing word vectors with CNN-derived region vectors.
  • The model learns to attend to specific image regions relevant to the question (e.g., identifying "green" trees or a "surfboard").
  • Performance indicates the model can overcome linguistic priors (e.g., correctly identifying green bananas despite "banana" usually being associated with yellow).
  • Limitations include struggles with temporal reasoning (e.g., distinguishing "about to hit" vs. "just hit" a ball) without explicit visual cues.

Future Challenges and Open Questions

• Unified Joint Models: Current obstacles include the lack of a single architecture achieving state-of-the-art across all tasks and the difficulty of multi-task learning without "catastrophic forgetting."
• Data Scarcity: In specific domains (e.g., healthcare), systems require large datasets of question-answer pairs; the recommended approach is manually curating data or bootstrapping from existing logs (e.g., failed search queries).
• Robustness: Systems are currently sensitive to adversarial inputs and cannot handle false premises (e.g., asking about a "man" in an image of only a girl) without explicit training on such errors.
• Knowledge Bases: Integrating external knowledge graphs (e.g., Freebase, WordNet) with text reasoning remains an active but under-explored research area for complex common-sense inference.
• Interpretability: The primary method for understanding model logic is visualizing attention scores to identify which input parts (words or image regions) influenced the final decision.