Interview, Fireside Chat, Webinar, Conference Presentation
The Uncertain Art of Accelerating ML Models with Sylvain Gugger
Jane StreetSylvain Gugger, Ron Minsky, Jeremy Howard, Mark Mandelmann, Mark Mirchandani, Francesc Campoy, Gabriel Sanchez
- Jeremy Howard predicts AI may displace jobs within a two-to-three-year timeframe, though he cites this as a claim from a New York Times article rather than his own assertion.
- Francis Cample anticipates that Jane Street will release a bootcamp on machine learning techniques every few months to educate traders and researchers.
- Support for GPU in the Mojo language is planned for release in a couple of months.
- NVIDIA is expected to soon release new Blackwell GPUs supporting FP4 precision.
- Jane Street's infrastructure will need to evolve to handle data that is significantly noisier and changes rapidly, as strategies viable two years ago may no longer be effective due to market reactions.
- The organization expects to continue building custom models and data loading utilities because few papers address financial time series analysis specifically.
- Tooling must support inference times ranging from 100 nanoseconds to a few hundred microseconds and up to milliseconds.
- Researchers will require access to small, composable building blocks rather than "magic" black-box training loops to allow deep interaction with machine learning internals.
- The field of machine learning is described as a "cooking science" where no one definitively knows the underlying mechanisms, and beginners often fail to realize that outcomes are not guaranteed without testing.
- Strict reproducibility is necessary to avoid developing models that perform well but cannot be understood or reproduced later.
- Training models on trading data is inherently harder than on structured data like text or images, meaning practitioners will never achieve the same level of accuracy.
- The current ecosystem is considered more immature than others, allowing engineers to significantly improve training speed, such as by a factor of four, by fixing obvious inefficiencies in a couple of hours.
- Techniques like gradual resizing and warm-up learning rate schedules are expected to be standard for open-source versions of models like GPT, even if top labs do not explicitly publish their usage.
- Hugging Face is anticipated to remain at the forefront of open-source AI despite challenges regarding the need for code to instantiate shared model weights.
- A shift from PyTorch's eager execution to compiled models via Torch.compile is becoming necessary due to GPU speeds outpacing CPU kernel scheduling.
- CUDA Graphs technology is expected to be useful for removing the overhead associated with launching series of small kernels.
- CUDA Streams can be utilized to parallelize data transfer and computation if the model is written without synchronization points to maintain full GPU utilization.
- Mixed precision training using float16 or bfloat16 is expected to enable models to train two to four times faster by leveraging specialized tensor cores.
- The non-deterministic nature of GPU training due to parallel execution can lead to floating-point results differing from expectations because floating-point arithmetic is not associative.
- The "hog-wild concurrency" approach has fallen somewhat out of favor due to its complete lack of reproducibility.
- A trade-off exists between performance and accuracy when organizing data loading for balanced GPU loads, sometimes necessitating a sacrifice of perfect shuffling.
- The speaker expresses a hope that Jane Street will improve training infrastructure by creating modular, configurable libraries to avoid "ossified" tools and giant training loops with spaghetti code.
- Gradual resizing was noted as still not widely used in the context of the competition, despite its status in other areas.