Conference Presentation, Tutorial
Playing Atari Games in OCaml
- Modern deep learning models are projected to achieve 99.5% accuracy on image classification tasks, surpassing human capability, with future trends heavily relying on transfer and self-supervised learning to reduce data requirements.
- Training plans involve mini-batches of 64 images constrained by up to 48 gigabytes of GPU memory, iterating through 100 epochs over a period of a couple of hours or more to find a minimum.
- While two linear layers with ReLU nonlinearity suffice for digit classification, deeper models are considered necessary for complex tasks like Atari games due to optimization algorithm limitations in finding specific minima, despite the universal approximation theorem.
- Policy gradient algorithms (e.g., A23, A3C, PPO, TRPO) are predicted to surpass Q-learning in training speed for Atari games, with agents requiring multiple hundred games for Pong and multiple days for Space Invaders.
- Consumer-grade GPUs with 8 gigabytes of memory (e.g., RTX 2070) are expected to successfully train simple Atari agents, while OpenAI baselines provide battle-tested implementations to mitigate training bugs.
- Future StarCraft or Dota agents may exploit micro-precision bugs if action-per-minute limits are not enforced, whereas limiting action rates aims to foster human-like behavior and prevent infinite-action exploitation.
- OCaml machine learning code, including Q-learning implementations following the Bellman equation, will be shared on GitHub with PyTorch and TensorFlow bindings, maintaining interfaces close to Python to leverage existing documentation.
- The OCaml ecosystem is expected to remain notably behind Python regarding tutorials and libraries, despite the advantage of accessing years of numerical stability techniques developed in major frameworks.
- Deep learning models currently struggle with transfer learning compared to human adaptability, and financial market applications face significant challenges due to super noisy observations and the necessity for a perfect world simulator.
- Current agents operate in episodic cycles without parameter changes during a game to accelerate early learning, utilizing epsilon-greedy policies set to zero during final evaluations to ensure optimal decisions.
- Specific game dynamics, such as Breakout's incentive to target higher bricks leading to tunnel-digging strategies or Space Invaders' orthogonal movement precision, highlight differences between AI and human performance in timing and strategic depth.