The Data Science Revolution

Conference Context and Participants

• Event: O'Reilly Media's Strata conference (2014), focused on the transformative power of big data.
• Moderator: Tim O'Reilly, CEO of O'Reilly Media, who noted data as "intel" for next-generation applications.
• Participants introduced:
  • Frank Adante (Rubicon Project): Co-founder/product architect; company automates advertising buying/selling, processes 3 trillion monthly transactions reaching 97% of the U.S. internet audience.
  • Tim Domke (eBay): Head of Big Data; eBay enabled $212 billion in e-commerce last year ($1 in $5 of U.S. e-commerce), utilizing data for risk management and external partner baselining.
  • Keith Collins (SAS): SVP/CIO; SAS holds ~35% global analytics market share (next five competitors combined do not equal this); company undergoing fourth infrastructure disruption to move to in-memory, large-scale compute clusters.
  • Michael Woods (IASDI): Senior data scientist; company applies algebraic topology for geometric data interpretation; currently serving seven of the top ten pharmaceutical companies and driving personalized medicine initiatives.

Disruption and Real-World Transformations

• Agriculture: Monsanto's acquisition of The Climate Corp (~$1B) combines weather data with robotics (Precision Planting) for self-driving tractors and micro-targeting of fertilizers/water; now competing directly with John Deere by supplying "smart guts" for equipment.
• Transportation: Google's self-driving cars utilize data from Google Street View (lane width, stoplight distance) to solve traffic light recognition as a "simple AI problem" once the location is known.
• Retail/Commerce: eBay identified counterintuitive correlations, such as swimwear purchases correlating more with cell phone accessories and home decor than traditional demographics (e.g., geographic location).
• Healthcare: Insurers use non-health purchase data (beer, cigarettes) to predict patient health outcomes; precision medicine aims to identify gene-specific drug efficacy rather than general population averages.
• Industrial: Shift from scheduled maintenance to predictive maintenance based on real-time sensor data (e.g., helicopter engine alerts) and usage patterns in sandy environments.
• Government: Simple text message interventions reduce jail overcrowding by reminding defendants of court dates, saving system costs.
• Advertising: Rubicon Project connects 100,000 advertisers to 600 million consumers within an 80-millisecond window (non-perishable inventory), automating what was previously a manual, spreadsheet-based process.

Technical Shifts: From Volume to Complexity and Computation

• Real-Time vs. Historical: Real-time analytics require historical baselines for validation; the value lies in "operationalizing" insights into decision-making processes (e.g., credit denial, drug efficacy).
• Data Complexity: Distinguished between "big data" (volume) and "complex data" (variety); petabytes of uniform sensor data offer little value compared to terabytes of varied data revealing failure signatures.
• Algorithmic Cross-Pollination: Successful data science often involves applying algorithms from unrelated domains (e.g., gene sequencing algorithms used for financial market state identification; collaborative filtering used in circuit design).
• Computational Availability: Cloud computing has democratized access to large-scale compute clusters, allowing smaller companies to perform analyses previously reserved for major corporations.

Data Privacy, Security, and Ownership

• First-Party Data: Rubicon Project distinguishes between proprietary first-party data (owned by the website) and aggregate anonymous data used for broader market understanding.
• Metadata Risks: NSA surveillance highlighted that metadata (who talks to whom) provides significant intelligence without content interception.
• Security Models: Shift from signature detection to anomaly detection over time to identify data leaks (e.g., phishing via home computers leading to internal breaches).
• Physical Security: "Suits and Spooks" conference example demonstrated physical access to power grid monitoring equipment (e.g., stealing passwords from a laptop left in a car) as a critical vulnerability in the Industrial Internet.
• Privacy Debate: Tension between HIPAA-style absolute protection and the modern reality of data-for-service exchange; consensus on "don't be creepy" but acknowledging bad actors exist.
• Future Model: Trend toward enterprises owning their data processing and sharing only "insights" (conclusions) rather than raw transactional data.

Methodological Challenges: Correlation vs. Causality

• Spurious Correlations: Relying on correlations without causality carries significant risk (e.g., financial loss, patient death); robust external validation procedures are mandatory.
• Reactive Systems: Some data systems change behavior based on the belief in their own predictions (reflexive knowledge), creating a third category beyond simple correlation or causation.
• Scope of Analysis: Data is strong at answering "what" and "when," but rarely "why" without external rationale; the line between solvable and unsolvable problems is dynamic (e.g., earthquake prediction).
• Role of Intuition: Data scientists must balance algorithmic findings with domain expertise; the "data talking to itself" model replaces traditional hypothesis-driven research in high-data environments.

Economic Models and Future Outlook

• Data as an Asset: Current consumer payment for data occurs via non-cash exchanges (loyalty programs, free services like Google); direct cash compensation for personal data is not expected to become the dominant model.
• Self-Service Analytics: While tools are becoming more accessible, the "data scientist" role remains critical for asking the right questions and interpreting results; the goal is embedding analytics into business processes ("black boxes").
• Trust Issues: The defining question of the 21st century will be "whose black box do you trust?" as analytics become baked into autopilot systems.
• Social Good: Emphasis on the societal value of data sharing (e.g., water distribution, DNA research) to drive adoption in sectors like pharmaceuticals where trust is currently a barrier.