Fireside Chat, Conference Presentation
Jennifer Tejada, CEO&Chairperson, PagerDuty: Never Miss a Beat Building Reliable Experiences with AI
- Major incidents impacting customer experience have increased by 43% in the last year, with a single major incident costing an average of nearly $1 million due to disruptions in revenue, profitability, and operating margins.
- PagerDuty, a 19-year-old public company, serves over two-thirds of the Fortune 100 and half of the Fortune 500 to help mature operations through automated digital resilience.
- Spotify maintains a resilient system despite having approximately 100 million lines of custom code and relying on roughly 1 billion lines of dependent code.
- To manage scale, Spotify shifted its operational philosophy from treating code as "pets" to treating it as "cattle," enabling large-scale transformation and standardization.
- Spotify introduced internal structure to counter previous chaos, utilizing its internal developer portal, Backstage, to enforce standards, frameworks, and consistent technology usage across the fleet.
- The organization leverages AI and LLMs to perform fleet-wide code transformations at scale to address resiliency, security, and cost issues, trading potential "blast radius" risks for speed and predictability.
- Spotify hired an internal marketing team to drive change management and defragment the technology ecosystem, a move supported by the CFO to improve operational resilience.
- Unlike SRE models, Spotify utilizes an "ops and squad" model where every developer is responsible for the operational status of the 20–30 code components they generate.
- PagerDuty's strategy involves an open API ecosystem connecting to 700+ tools to automatically detect, correlate, and orchestrate incidents, with a vision to evolve from event detection directly to code remediation before end-user notification.
- Spotify developed a custom AI chatbot trained on "golden data sets" and standards to ensure both human developers and AI agents receive high-quality context and are nudged toward standardization.
- Backstage has been commercialized as an external product called "Portal," which includes an integrated AI agent named "Ica," marking Spotify's shift toward a B2B infrastructure provider.
- The decision to open-source and commercialize Backstage was driven by the realization that Spotify's scale challenges were universal and that SaaS offers higher margins than the low-margin music industry.
- Spotify maintains a single cloud provider to ensure simplicity and structure, avoiding the fragmentation that can lead to unmanageable tech debt.
- Engineering satisfaction regarding tech stack health is actively tracked, and a Backstage plugin named "Soundcheck" provides real-time indicators when teams deviate from established operational "golden paths."
- Tyson Singer notes that while Spotify values speed, the introduction of strict standards and the removal of excessive autonomy were necessary trade-offs to manage the chaos of rapid growth.