Conference Presentation, Lecture, Keynote
Torch Tutorial (Alex Wiltschko, Twitter)
Lex FridmanAlex Wiltschko, Hugo Larochelle, Andrej Karpathy, Richard Socher, Sherry Moore, Ruslan Salakhutdinov, Andrew Ng, John Schulman, Pascal Lamblin, Adam Coates, Quoc Le, Yoshua Bengio, Shubho Sengupta
- The session will begin with practical examples before focusing on Torch Autograd and shared concepts across deep learning libraries, with a goal to explain the rationale behind the diversity of existing tools.
- For individuals with scripting experience in MATLAB or Python, the speaker estimates that transitioning to Lua is achievable within a day for basic tasks, with initial productivity expected within a week.
- Forward mode automatic differentiation is predicted to be ineffective for neural networks with millions of parameters, whereas reverse mode is expected to compute gradients for all matrix-based parameters simultaneously.
- Torch Autograd is designed to build compute graphs just-in-time by listening to function calls, with the speaker suggesting a reinvestigation of source-to-source transformations despite the historical reliance on domain-specific runtimes.
- Training with Torch Autograd is expected to incur no speed penalty at test time as the differentiation layer disappears, though highly complex models may still require hand-written code for optimal performance.
- Implementation of higher-order gradients, such as Hessian vector products, is characterized as an expensive and difficult open problem, though currently available in some libraries.
- Optimization strategies like checkpointing for non-linearities such as ReLU may favor recomputation over memory storage during the forward pass, representing a trade-off in resource management.
- While optimized performance modes remain a work in progress, average model training with Torch Autograd is anticipated to be as fast or faster than native NN implementations.
- Inference on embedded systems will not show appreciable overhead from the Lua layer provided the binary is compiled into C code, with low-latency calls to Torch from C code expected to be extremely fast.
- Latency for Torch calls is predicted to be negligible compared to running native Lua, contrasting sharply with wrappers like JNI which are expected to incur significant overhead.
- At Twitter, machine learning constraints are dominated by model complexity and latency rather than HTTP request handling overhead, with the speaker noting Twitter may be the only large-scale production user of Torch for serving all incoming media.
- There is currently no centralized "Torch server" equivalent to TensorFlow Serving, though the decentralized community structure means such a tool could potentially be developed by others.
- The speaker anticipates that forward engagement via social media streams and direct tweets will increase meaningful participation from remote viewers, with commitments to answer questions during and after the session.
- Switching from frameworks like Keras or TensorFlow to Torch is primarily suggested to enable easier performance reasoning, as added compiler infrastructure in other systems can obscure performance bottlenecks.
- Source-to-source transformation using text-based compiler output is no longer considered the gold standard, with the speaker hypothesizing that deeper reinvestigation into this area is worthwhile.
- The speaker maintains that no single "silver bullet" exists in the ecosystem, identifying TensorFlow as the largest platform, Theano as research-goal oriented, and Torch as tending toward research over industry applications.
- Forward-looking statements regarding the number of libraries, their differences, and specific performance claims are presented as hypotheses subject to correction or further verification by the audience.