Torch Tutorial (Alex Wiltschko, Twitter)

Presentation Scope:

• The talk covers practical usage of the Torch framework and Lua, followed by a deep dive into Autograd and the shared concepts of deep learning libraries.
• All examples and tutorials are available via a specific GitHub repository, including a self-paced CNN training notebook.
• Material was developed in collaboration with Sumit Chintala (Facebook), Hugo Larochelle, and Ryan Adams (Twitter).

Torch and Lua Ecosystem:

• Language Choice: Torch uses Lua, which offers LuaJIT (just-in-time compilation) performance comparable to C for loops while maintaining a small, understandable codebase (~10,000 lines of C).
• Interoperability: Lua is designed for embedding and interop with C libraries, a feature copied by Python's FFI, Julia, and used in software like World of Warcraft, Adobe Lightroom, and Redis.
• Origin Story: The choice of Lua stemmed from the difficulty of running Python on embedded hardware (e.g., for wearable CNNs), where Lua had existing success.
• Community Model: The ecosystem is community-driven rather than industry-owned, fostering high-quality, cutting-edge research implementations (e.g., GANs, image captioning).
• GPU Support: Provides first-class, interactive GPU computation via cuTorch, allowing seamless data transfer and operation without manual CUDA kernel writing.

Torch Core Mechanics:

• Tensor Structure: A tensor is a lightweight pointer/view into raw memory storage, defined by size, stride, and offset rather than a full data copy.
• Memory Efficiency: Operations like slicing create new tensor views with different strides/offsets without copying data, meaning modifications to a slice reflect on the original tensor.
• Philosophy:
  • Imperative Programming: Code executes line-by-line with immediate feedback ("things should be immediate").
  • Minimal Abstraction: Code is directly traceable to underlying C implementations (usually 1-2 jumps), aiding performance debugging.
  • Positioning: Positioned more towards research than industry, though Twitter has successfully deployed it in production for serving all incoming media.

Torch Architecture Packages:

• nn: A package for building feedforward networks using "LEGO blocks" (containers); best for linear architectures and standard CNNs.
• optim: Handles stochastic gradient descent and variants (AdaGrad, AdaDelta); requires parameters to be linear in memory.
• autograd: A higher-order function library allowing the composition of any computation (even control flow) while maintaining correct gradient tracking.

Automatic Differentiation (Autograd):

• Core Abstraction: Machine learning relies on automatic differentiation (AD), specifically reverse mode (backpropagation), which is efficient for optimizing millions of parameters.
• Forward Mode Inefficiency: Forward mode AD requires one execution pass per parameter, making it unsuitable for standard neural networks.
• Implementation Strategy:
  • Trace-Based: Autograd records operations in real-time ("spy" functions) to build a dynamic computation graph.
  • Reverse Execution: Gradients are computed by walking the trace backward from the loss to the inputs.
  • Control Flow: Supports dynamic control flow (if statements, variable-length for loops, recursion) because the graph is built dynamically during execution.
  • Custom Gradients: Users can override default derivatives for non-differentiable operations (e.g., floor, quantization) to enable differentiable pipelines.

Library Comparison & Graph Types:

• Granularity Spectrum:
  • High Abstraction: nn, Keras (fixed layers like CNNs).
  • Mid-Abstraction: autograd, Theano, TensorFlow (composition of primitive numeric ops).
  • Custom Modules: Only possible in libraries allowing user-defined derivatives at the primitive level (right side of spectrum).
• Graph Construction Modes:
  • Static/Graph-Based: TensorFlow/Theano build explicit graphs before execution (allowing compiler optimizations but hindering control flow).
  • Just-in-Time (JIT): Autograd/Chainer build graphs dynamically during execution (flexible for control flow, harder to optimize).
  • Hybrid: Some libraries attempt to blend static graphs with runtime control flow extensions (e.g., Theano's scan).

Production & Deployment:

• Training vs. Inference: Autograd's overhead is limited to training; inference uses only the compiled numerical code with no tracking overhead.
• Performance: Inference speed in Autograd is comparable to nn for standard models; custom models may be faster if hand-optimized.
• Integration: Lua is lightweight (kilobytes) and embeddable in C/C++/Java; Twitter uses JVM-integrated Lua via JNI for production serving.
• Serialization: Models can be serialized for deployment in other languages (Python, Java, C++) by extracting weights or embedding a Lua VM.

Future Research Directions:

• Checkpointing: Recomputing intermediate values during forward pass to save memory in reverse-mode AD.
• Mixed Modes: Combining forward and reverse mode AD for diamond-shaped computational graphs.
• Higher-Order Gradients: Efficient calculation of Hessian-vector products for second-order optimization.
• Source-to-Source: Re-investigating text-based gradient compilation rather than just runtime graph execution.

Q&A Highlights:

• Visualization: Lua's plotting is less mature than Python's; relies on external wrappers like Bokeh.js.
• C++ Deployment: No appreciable latency overhead when calling Lua from C/C++ at inference; the Lua VM is small and mature.
• Cross-Language Access: No centralized equivalent to TensorFlow Serving; common approaches include writing custom inference in target languages or embedding the Lua VM.