Conference Presentation, Keynote
A Keynote by Thomas Sohmers, Co Founder & CTO of Positron
- Economic Shift: AI inference is projected to become the primary bottleneck and driver for global GDP growth, fundamentally altering the macroeconomic landscape.
- Labor Scaling: Unlike previous industrial revolutions, AI enables the unprecedented scaling of labor, allowing software development to be performed 24/7 at a cost that may eventually drop below $1/hour.
- Cost Analysis: Running a capable open-source model (e.g., DeepSeq R1) on an 8x H200 system costs approximately $14,000 annually, offering a cost-benefit ratio comparable to a junior engineer's salary while operating continuously.
- Market Capex/Opex: Massive capital expenditures on new data centers are justified only if inference costs can be reduced, as inference (not training) is the revenue-generating bottleneck.
- Architectural Mismatch: Traditional GPU optimizations for matrix-matrix multiplication (CNNs) are inefficient for modern Transformer models, which require matrix-vector operations and have an operational intensity 500 times lower (1 flop per byte transferred).
- Hardware Inefficiency: Current GPU systems achieve poor memory bandwidth utilization for Transformers, often utilizing less than 30% of available capacity, with performance dropping into the teens for specific workloads like Llama 70B.
- KV Cache Explosion: Short-term memory (KV cache) grows quadratically with sequence length; as context windows expand from 8k (GPT-3.5) to 128k+ (GPT-4) and 1M+ (Gemini 2.5), memory capacity for the cache now exceeds the memory required for model weights.
- Training Dependency: New training paradigms like Monte Carlo Tree Search for reasoning (e.g., OpenAI O1) require massive inference runs during the training phase, creating a feedback loop where inference capability dictates training speed.
- Positron AI Background: Founded in April 2023, the startup shipped its first hardware within 18 months with $12M in initial funding, recently announcing a $23M raise.
- Performance Claims: Positron's hardware claims a 70% throughput advantage over two NVIDIA H100 GPUs while using significantly less power per token and costing less than half the retail price.
- Memory-First Design: The architecture prioritizes memory bandwidth and capacity over raw floating-point operations to address Transformer bottlenecks.
- Software Compatibility: The system natively ingests standard Hugging Face model files (.pt, .safetensors) without modification, ensuring drop-in compatibility with the NVIDIA CUDA ecosystem.
- Power Efficiency: Hardware cards operate at 150 watts, enabling deployment in air-cooled data centers without requiring the liquid cooling infrastructure mandated by high-power NVIDIA systems.
- Concurrent Inference: The Atlas server (4U, <2kW) supports concurrent execution of multiple models, allowing a single unit to host dozens of different models (e.g., fine-tunes, Phi, Mistral) simultaneously rather than dedicating entire GPUs to single models.
- Product Roadmap:
- Current: Atlas server shipping today, supporting up to 512 billion parameters concurrently.
- 2025: Titan system releasing next-generation silicon with 1 terabyte of DRAM per chip (8TB total), enabling support for models with trillions of parameters.
- Strategic Vision: The company posits that the future of AI acceleration will be "positronic rather than electronic," emphasizing memory-centric architectures over traditional compute-centric designs.