newsfilter.io

a16z Podcast | Big Data Goes Really Big

a16zPrat Moghe, Peter Levine, Michael Copeland
Watch on YouTube →
  • The industry is shifting from Big Data 1.0 to 2.0, characterized by a transition from on-premise infrastructure and relational stacks to cloud-based SaaS models that democratize access beyond the Fortune 2000 to governments, social services, and hundreds of thousands of smaller organizations.
  • This cloud migration aims to reduce deployment cycle times from six to nine months to same-day responses, enabling real-time decision-making, hyper-targeted marketing, and the ability to iterate quickly, potentially cutting costs to one-fifth of current infrastructure expenditures.
  • Current market dynamics involve a $10 billion segment dominated by incumbents with legacy stacks, where organizations lacking rapid data access react to events rather than predicting them, while the data growth rate outpaces current spend deployment capabilities by 200% annually.
  • Organizational culture is expected to flatten with the emergence of a collaborative, Google- or Facebook-like environment where the distance between specialists and executives narrows, and analytics enable trend identification prior to incidents or the reduction of resource deployment in areas like public safety.
  • The technology stack is evolving from SQL (ten years ago) and Hadoop (five years ago) to Spark, with a future trajectory toward application-aware big data systems and machine learning layered over real-time cloud foundations.
  • Risks and barriers include enterprise hesitation driven by the expense of data migration and operational friction from hybrid cloud-on-premise setups, with non-migrating companies facing the prospect of launching initiatives two to three years late and losing competitive relevance.
  • Future challenges will include overcoming the "Big Data 3.0" problem once the current 2.0 cloud transformation is solved, necessitating that the immediate priority is establishing powerful real-time systems in the cloud rather than maintaining separate technology stacks.