Chris Re

• Deep Dive is a "dark data" system designed to convert unstructured inputs (emails, web pages, tables, images) into structured, SQL-style databases.
• The system achieves extraction, transformation, and loading (ETL) quality that surpasses human annotators in specific settings while reducing development effort by orders of magnitude.
• Deep Dive unifies extraction, integration, and cleaning as a single statistical inference problem rather than treating them as distinct stages.
• The approach utilizes random variables and factor graphs to represent uncertain choices (e.g., entity resolution, OCR grouping), allowing the system to probabilistically determine the most likely database entries.
• User programs specify only features, random variables, and correlations, decoupling high-level logic from the underlying inference algorithms.
• To support this high abstraction, the team built scalable inference engines that exploit modern hardware parallelism (SIMD, multi-core, NUMA).
• The system prioritizes hardware efficiency over strict sequential consistency, utilizing "Hogwild!" algorithms that remove locking mechanisms to achieve linear speedups on sparse problems.
• A "Hogwild!" implementation allows the system to achieve 10 to 1,000 times the speed of competitor systems by relaxing consistency constraints without degrading statistical convergence under specific sparsity conditions.

Case Studies and Performance Metrics

• PaleoDeepDive (Biodiversity/Climate)

  • Goal: Construct a global fossil record by extracting data from scientific PDFs.
  • Baseline: Previous efforts (PaleoDB) required ~300 volunteers over a decade (2 person-decades) with 84% accuracy on location canonicalization.
  • Efficiency: A single graduate student built the system, which processes 10x more documents in under an hour.
  • Data Volume: The system extracts roughly 100x more data from the same papers compared to human readers, who often miss subjective taxonomic opinions.
  • Accuracy: Machine accuracy on formation extraction exceeds human volunteers, who suffer from inconsistent conventions and errors when applying outdated vs. current naming conventions.

• Law Enforcement Application

  • Goal: Identify human trafficking victims by analyzing web text advertising sexual services.
  • Method: The system builds a "prostitution pricing model" to flag anomalous patterns indicative of trafficking.
  • Outcome: The system achieves high precision and has triggered countermeasures from criminal elements, including the initiation of real arrests.
  • Status: Pilots are expanding to additional law enforcement agencies.

Technical Trade-offs and Trends

• Statistical vs. Hardware Efficiency: The architecture balances the number of algorithmic steps required for convergence (statistical efficiency) against the exploitation of hardware primitives (hardware efficiency).
• Sequential vs. Parallel: Traditional stochastic gradient descent is natively sequential; Deep Dive's approach converts these into massively parallel operations by removing synchronization locks.
• Future Development: The team continues to refine consistency relaxation techniques to improve energy efficiency and performance, moving beyond simple "parlor tricks" to robust engineering solutions.
• Open Source: All code and implementation details for Deep Dive are available in open-source repositories.
• Recognition: The approach won a major NLP competition in collaboration with Chris Manning's group.