Interview, Fireside Chat
Building a Data Warehouse From Scratch with Jacob Baskin
- Anticipates a shift in technical design priorities where practical simplicity and user comprehensibility are valued over theoretical perfection, reducing the relevance of strict incentive compatibility principles in real-world applications.
- Foresees the new Superstore data warehouse accommodating petabytes of data that exceed single-computer capacity, delivering superior performance for large-scale read operations compared to Postgres through asynchronous writes and columnar compression.
- Plans to construct a data warehouse analytics system internally rather than relying on SaaS or open-source solutions to maintain full control over feature prioritization for specific finance use cases not well-supported by external communities.
- Projects that an initial storage investment of 10 petabytes made in 2022 will prove sufficient, despite the high financial commitment being based on a design document from a small team.
- Identifies CockroachDB as a scalability bottleneck for high-throughput writes to single tables due to its row-division consensus mechanism, necessitating a replacement with the ARIA state machine replication system.
- Predicts the migration to ARIA will increase throughput by funneling data through a single core, accepting a single point of failure risk resulting in potential downtime of approximately one minute every four years.
- Highlights critical infrastructure risks including NFS locking semantics that may crash storage appliances during concurrent file creation and Hive scaling challenges that could inadvertently function as a denial-of-service machine.
- Outlines plans to resolve data transfer inefficiencies by implementing BitTorrent-style distribution for code propagation and to modernize the Hive interface toward a declarative computation graph akin to Polars or Spark.
- Describes future scheduling improvements that model job urgency via time-based utility curves rather than static bids, acknowledging that this addition transforms the scheduling problem into an NP-hard complexity alongside existing gang scheduling requirements.
- Anticipates that the explosion of data volume and machine learning applications will continuously generate new use cases for the system, particularly as it becomes capable of efficiently querying terabytes of data.
- Forecasts that future GPU clusters will demand topologically aware scheduling to optimize inter-GPU connectivity and manage data movement across specific data centers.
- Explains the strategic decision to restrict direct access to underlying Parquet files to enable a robust access control layer with ACLs, avoiding the insufficient feature set of Unix file system controls.