Interview, Keynote
Things That Don't Scale, The Software Edition – Dalton Caldwell and Michael Seibel
- Core Philosophy: Founders should "turn the water on" to identify broken pipes (failures) rather than attempting to pre-verify every component before launch; scalability problems are solved by earning the privilege of solving them after establishing product-market fit.
- The 90-10 Solution: Invented by Paul Buhite (Gmail inventor), this principle advocates achieving 90% of the user benefit with only 10% of the development work to accelerate validation.
- Gmail Origins (Paul Buhite):
- Buhite created the first Gmail version by injecting his own email into the existing Google Groups UI.
- He built features incrementally based on immediate usage needs (e.g., "I need to write an email" → built writing functionality).
- Physical Maintenance: When the system crashed, Buhite fixed a corrupted hard drive by physically entering the server room with a screwdriver.
- Invite System: The viral invite limit was not a growth hack but a forced constraint due to saturated hard drive capacity.
- Release Hurdles: Google initially hesitated to release the product due to concerns over excessive hardware consumption.
- Facebook's Early Scaling Strategy:
- Architecture: Facebook utilized a "copy-and-paste" architecture with separate PHP instances, MySQL databases, and Memcache instances for each university (e.g., separate servers for Harvard, Stanford, etc.).
- Rationale: This avoided the impossible task of scaling a single users table to hundreds of millions of people on the poor hardware and database technology available at the time.
- URL Structure: User login was hardcoded to specific URLs (e.g.,
harvard.thefacebook.com), requiring years of engineering to later unify into a global users table. - Hardware: The initial infrastructure consisted of cheap "micro" servers physically located in a Santa Clara data center.
- Twitch (Justin.tv) Workarounds:
- Static Caching: During traffic spikes (20x steady state), popular channels were converted to static pages to prevent application server crashes; the video player and chat remained dynamic.
- Stream Propagation: To handle celebrity broadcasts causing server failure, the system pre-propagated the stream to necessary video servers before turning the broadcast live for users, resulting in a few-second delay but stable delivery.
- Free Peering Hack: When an ISP refused free peering, Twitch blocked video access for users from that ISP after 10 minutes, citing the inability to serve them, effectively forcing the ISP to negotiate.
- Community Translation: To achieve global localization without high costs, Twitch used a Reddit-style volunteer platform where community members translated UI strings.
- iMeme's "Video-No-Video" Hack:
- The music platform played audio files by reusing open-source video code, encoding music as
.flvfiles with zero video bitrate and a blank video field.
- The music platform played audio files by reusing open-source video code, encoding music as
- Friendster vs. MySpace:
- Friendster: Attempted to calculate "two degrees of separation" in real-time via MySQL threads, leading to system instability.
- MySpace Solution: Replaced live calculations with a static display indicating whether a person was in a user's direct friend list or just an "extended network" connection.
- Google's MapReduce Genesis:
- The Crisis: In ~2001, Google's batch re-indexing process failed and took three weeks to run; the search engine effectively stopped updating new content for 3–4 months.
- User Impact: Users remained unaware of the stale data as search results continued to return previously indexed content.
- The Solution: The "broken" system forced the creation of MapReduce to parallelize crawling and indexing tasks.
- Legacy: This under-duress innovation led to Hadoop and is now a foundational technology for nearly every major internet company.
- Pre-Cloud Infrastructure Realities:
- Early startups (pre-AWS) were required to physically drive to data centers to reboot servers during outages.
- Hardware limitations (slow disks, bad MySQL, poor scaling) necessitated "dirty" hacks rather than elegant, scalable architecture.
- Forward-Looking Conclusion:
- Successful companies like Apple (Wozniak hand-soldering) and Airbnb (founders taking photos) prove that non-scalable, manual, "ugly" solutions are often the necessary first step to validate demand.
- Engineers should prioritize building what users want immediately and defer solving scalability issues until the product proves successful.