Theano Tutorial (Pascal Lamblin, MILA)

• Core Functionality: Theano is a symbolic expression compiler that enables the definition of mathematical expressions using NumPy syntax, supporting basic operations (min, max, addition) as well as complex neural network structures.
• Graph Manipulation: The system allows for graph substitutions, cloning, and replacement to facilitate automatic (symbolic) differentiation and the application of the R operator for forward differentiation.
• Optimization: The compilation process performs graph optimizations for numerical stability (e.g., fusing log(1+x) into a stable operation) and computational efficiency, including in-place operations and device-specific transfers (CPU/GPU).
• Runtime Execution: Optimized graphs are compiled into C++ or CUDA code, which is dynamically generated, compiled, and imported back into Python to ensure high performance while avoiding context-switching overhead.
• Historical Context: Theano has existed for over eight years, originating from a small team at MILA (formerly LISA) and growing to include global contributors, serving as the backend for libraries like Keras, Blocks, and Lasagne, and supporting probabilistic programming in PyMC3.
• Symbolic Variables:
  • Input Variables: Defined with fixed dimensions and data types but flexible shapes; memory layout is not fixed between calls.
  • Shared Variables: Symbolic variables that hold persistent values across function calls, typically used for model parameters (weights/biases) that can be updated.
• Automatic Differentiation: The theano.grad function backpropagates through the graph to compute gradients without explicitly constructing Jacobian matrices, returning symbolic expressions for gradients and update rules (e.g., gradient descent).
• Function Compilation: theano.function compiles specific subsets of a graph (e.g., prediction only, or training with updates), allowing implicit inputs for shared variables and explicit outputs for costs and gradients.
• GPU Integration: Users can select devices (CPU, CUDA, specific GPU) via configuration files or environment variables; shared variables default to GPU memory, and float32 or float16 data types are recommended for performance.
• Loop Handling: The scan operator encapsulates recurrent computations (like LSTMs), allowing loops in the computation graph while supporting backpropagation through time (BPTT).
• Debugging Tools: Theano provides py.print for graph visualization, DebugPrint for textual graph inspection, and test value injection to detect shape mismatches during graph construction.
• Future Roadmap: Planned improvements include better 3D convolution support, faster graph optimization, expanded data parallelism, completion of basic RNN operations, and wrapping more cuDNN operations.
• Distribution Challenges: Due to tight integration with Python's memory management (refcounting) and reliance on C++/CUDA backends, distributing models as standalone binaries is not currently supported; Docker containers are the recommended deployment method.

Practical Examples Covered

• Logistic Regression on MNIST: Demonstrated a basic implementation using symbolic variables, defined a cross-entropy loss, compiled a training function with updates, and visualized learned filters.
• ConvNet (LeNet) Architecture: Illustrated the composition of convolutional, pooling, and fully connected layers to build a deeper network, handling arbitrary image sizes and optimizing parameters.
• LSTM for Character Generation: Showcased the use of the scan operator for sequence modeling, including handling variable sequence lengths via masking, pre-processing with Fuel, and generating text character-by-character.

Q&A Highlights

• Error Traceback: Disabling optimizations reveals the specific line of Python code where a shape mismatch error occurred, aiding in debugging symbolic graph definitions.
• Model Distribution: The speaker noted that disentangling Theano from the Python runtime is a significant project; currently, Docker is the standard solution for sharing models without requiring local compiler installations.