How to Scale AI Application Inference 100x ft. Fireworks’ Lin Qiao

Strategic Shift in Inference Alignment

• Future inference optimization requires aligning data distribution from application workloads with the data distribution used in model training.
• Current industry practice relies heavily on off-the-shelf models, creating a significant gap between model capabilities and specific product knowledge.
• Successful scaling depends on engineering systems that infuse application-specific data into the model rather than treating inference as an isolated component.

The Three-Dimensional Scaling Law

• The future of inference scaling is defined by the simultaneous optimization of three competing dimensions: quality, speed, and user concurrency (cost).
• Enterprise demands often require maximizing all three dimensions simultaneously, creating a multi-dimensional optimization problem specific to each application.
• The proposed solution involves "co-optimization" where post-training and inference stages are combined to drive massive model acceleration.

Economic and Operational Goals

• The primary objective is to reduce current inference costs by 10x to 100x, transforming high-cost "icebergs" into sustainable business models.
• Lowering the cost barrier is intended to allow applications that have achieved product-market fit to scale indefinitely.

Complexity of Inference Systems

• Solving the three-dimensional optimization problem requires navigating a combinatorial explosion of over 100,000 variables.
• Key optimization vectors include:
  • Token prediction batching (e.g., predicting 10 tokens at once rather than one).
  • Numerical and positional alignment with application-specific data distributions.
  • Hardware selection based on specific SKU advantages (FLOPS vs. memory bandwidth).
  • Model sharding and cross-host distributed inference strategies.
  • Kernel selection and specialized tuning mechanisms for quality.

Fireworks Platform Capabilities

• Fireworks operates a virtual cloud infrastructure that abstracts GPU management and selects hardware based on specific application data distributions.
• The platform supports diverse hardware vendors and SKUs to optimize for different model shapes and forms.
• It offers developer-facing tools for self-service experimentation, including reinforcement tuning driven by production data.
• Recent customer scaling milestones include:
  • A food chain company expanding from one AI shop feature to 1,000 shops in three months.
  • A software development firm scaling AI features from 100,000 to 25 million developers within three months.

Future Outlook

• The industry is moving toward heavy customization of inference infrastructure tailored to specific application needs.
• Next-generation innovation depends on making this complex optimization accessible, high-quality, and easy to deploy for app developers.