a16z Podcast | A Conversation With the Inventor of Spark

Spark Definition and Value Proposition

• Spark is software designed for processing large volumes of data on a cluster.
• It features a powerful programming model supporting advanced analytics, including machine learning, graph computation, and stream processing.
• The system prioritizes ease of use, addressing the complexity and poor performance of predecessors like MapReduce.

Origins and Motivations

• Matej Zaharia developed Spark while at UC Berkeley after observing limitations in early Hadoop/MapReduce implementations at companies like Facebook (circa 2007).
• MapReduce was primarily designed for batch jobs taking tens of minutes to hours (e.g., nightly web indexing), which was insufficient for interactive, real-time data exploration.
• Facebook's challenge involved massive data scale combined with a need for many non-technical users to interact with data without relying solely on data specialists.

Key Announcements from Spark Summit

• IBM Investment: IBM announced backing Spark by dedicating developer resources to its improvement and integrating the technology into internal product lines, Watson, and database products.
• Toyota Case Study: Toyota uses Spark to analyze social media feedback in real-time to identify vehicle issues, such as specific brake noises reported by users online, enabling rapid product improvements.
• Broad Adoption: Major organizations presenting at the summit included Netflix, Capital One, Goldman Sachs, and Novartis.

Open Source Ecosystem and Strategy

• Spark is identified as the most active open-source project in data processing.
• Success factors include tackling a clear market need (large-scale data), maintaining a high-quality core team, and fostering a low-barrier entry for community contributors.
• Integration Strategy: The ecosystem has expanded to include projects like Hive, Pig, and Mahout running on Spark, alongside storage systems like MongoDB and Cassandra.
• Architecture Philosophy: Spark is designed to sit atop various storage systems, including Hadoop, while remaining agnostic to allow data access from any environment.

Commercialization Model (Databricks)

• Databricks was founded to harden Spark for commercial use while maintaining full open-source availability.
• The business model focuses on offering Spark as a cloud service rather than gating features behind a paywall, eliminating the traditional tension between open-source contribution and proprietary monetization.
• This approach ensures that all engine improvements remain freely downloadable, with revenue generated via managed cloud infrastructure.

Historical Context: The Netflix Challenge

• PhD student Lester Mackey's team placed second in the Netflix Challenge, a competition offering a $1 million prize to reduce movie recommendation error rates.
• Spark was initially built to support the high-speed algorithm processing required for this type of large-scale recommendation engine.

Future Outlook

• The trend toward interactive, real-time data analysis is driving the shift away from traditional batch-only architectures.
• Future integration is expected between IBM's non-cloud solutions (consulting, Watson) and Spark to create a unified data processing environment.