Building a Data Warehouse From Scratch with Jacob Baskin

Jacob Baskin's Background and Mechanism Design

• Baskin began coding in childhood, initially planned to study psychology, but switched to computer science at Brown University to pursue mechanism design and the intersection of economics and algorithms.
• Mechanism design involves creating systems (like auctions) where participants are incentivized to truthfully report their preferences to achieve socially optimal resource allocation.
• Baskin learned in academia that incentive-compatible mechanisms are valuable, but realized in industry that explaining the mechanism clearly to users is often more critical than theoretical optimality.
• He found that incorporating user budget constraints into mechanisms often breaks simple incentive-compatibility, leading him to prioritize simplicity and explainability over pure theoretical rigor.

Career at Google and AdExchange

• Baskin joined Google in 2008 to work on AdExchange, a two-sided market connecting publishers (ad space sellers) and advertisers (ad buyers).
• Google acquired DoubleClick to leverage its enterprise relationships with large publishers and advertisers, complementing Google's strength in search advertising infrastructure.
• The team integrated DoubleClick's business insights with Google's scalable infrastructure (MapReduce, BigTable, GFS) to build a system handling global ad traffic.
• AdExchange utilized real-time bidding (RTB), querying advertisers' systems for every ad impression and injecting HTML snippets to fetch ad images directly from the winner's server.
• To address security and privacy concerns regarding tracking pixels and cookies, Google implemented user identifier anonymization, allowing advertisers to recognize past users without receiving raw PII.
• Google's engineering philosophy of containing state in few components (e.g., stateful cores, stateless web servers) was adapted for ads, though with a 250ms latency budget significantly higher than high-frequency trading systems.

Transition to Urban Infrastructure Startup

• Baskin joined a startup aiming to manage municipal curbside rules by building a platform to digitize physical parking signs and rules.
• The team used augmented reality (AR) visual odometry and smartphone mapping to pinpoint sign locations with high accuracy, overcoming GPS limitations in urban canyons.
• The company built custom parsers to interpret the spatially complex and city-specific language of parking signs, which varied significantly across jurisdictions.
• Despite solving difficult technical problems, the startup failed as a business due to an inability to monetize the domain-specific data effectively.

Jane Street and the "SuperStore" Data Warehouse

• Baskin joined Jane Street in 2015 to work on database infrastructure, specifically replacing a monolithic, single-machine Postgres cluster ("TraderDB") that struggled with scale and user-induced performance issues.
• The legacy Postgres system suffered from long-running transactions causing lock contention and versioning bloat, which degraded performance for all users.
• The team designed "SuperStore," a distributed analytical database that prioritized large read queries over transactional consistency between tables.
• Asynchronous Writes: SuperStore commits writes to a log immediately but materializes data asynchronously to optimize compression and columnar layout, separating write acknowledgment from data availability.
• No Direct Parquet Access: The team denied user requests to directly read underlying Parquet files to enforce centralized access control, usage logging, and schema management, preventing the deprecation hell seen in previous systems.
• Metadata Store Trade-off: SuperStore initially relied on CockroachDB for metadata, which proved to be a bottleneck for write throughput due to the latency of distributed consensus (Raft) for frequent small row updates.
• Architecture Pivot: The team is currently replacing CockroachDB with "ARIA," a finance-style state machine replication system that achieves high throughput by centralizing the sequencer on a single core, accepting a single point of failure in exchange for performance.

The "Hive" Compute Cluster

• The Hive is Jane Street's distributed compute cluster, scaling to hundreds of thousands of CPU cores and over 10,000 GPUs for neural network training and large-scale simulation.
• Growth in demand is driven primarily by machine learning training (GPU-heavy) and data preparation/analysis (CPU-heavy).
• NFS Scalability Issues: The team had to implement safeguards to prevent Hive jobs from taking down storage appliances via NFS directory locking semantics, where creating thousands of files simultaneously can block the entire service.
• Scheduling Mechanism Design: The current resource allocation uses a second-price auction where users bid dollars per CPU/GPU hour.
• Utility Curve Gap: The current auction model fails to capture "urgency" (the decay of value over time), forcing users to guess optimal submission times rather than expressing their true time-sensitive preferences.
• Optimization Strategy: The team aims to replace the current "bang for buck" bidding with a mechanism that models the derivative of user utility over time, solving a complex NP-hard gang-scheduling problem.
• Data Distribution: To efficiently launch code across thousands of workers, the team is implementing BitTorrent-style peer-to-peer distribution, allowing workers that receive code chunks to immediately serve them to other workers, reducing load on the primary source.
• API Evolution: The team is shifting from ad-hoc task graphs to a declarative computation graph API (inspired by Polars and Spark) to enable global optimization of data movement and computation placement across the cluster.