Conference Presentation
Bryan Cantrill: The Hurricane's Butterfly: Debugging Pathologically Performing Systems
- The speaker anticipates that delivering a talk on debugging during outages will trigger a "conga line of outages" with immediate system failures, and warns that if their phone remains on, their joint system may exhibit pathological behavior for years.
- There is an expectation that software will frequently fail fatally without understanding the cause, and that as post-singularity is reached, robots may take over but could potentially self-repair first.
- Predictions indicate that systems will eventually perform terribly after operating well for six months to five years, and that low-level problems can induce latency explosions throughout the stack, a dynamic amplified by increasingly deep and tall software layers built since 1997.
- The outlook suggests that fixing scalability issues creates a sequence where resolving one problem immediately exposes the next, and that improvements to the system are difficult to model, meaning changes may have unpredictable consequences.
- Debugging is described as non-mechanical with no fixed recipe, where hypotheses are the final stage rather than the first, and where the "stupid system" is inherently dynamic and moving during the investigation.
- It is predicted that without sufficient observability, engineers will guess or make changes by accident, and that correlation does not imply causation, meaning success can sometimes stem from resetting processes rather than actual debugging.
- The speaker expects a shift from traditional monitoring to observability within DevOps, emphasizing that aggregating data eliminates critical time information and that disaggregated data is necessary to see patterns hidden by aggregation.
- Visualizations are predicted to provoke further questions and disconfirm hypotheses rather than provide direct answers, while tools are seen as assistance only; the actual cognitive work remains difficult and human-driven.
- The future outlook highlights that humans remain superior to computers at discovering de novo patterns, but debugging rewards persistence and resilience over intuition, which can be detrimental, and that every executing instruction is ultimately traceable to human direction.
- Risks include the amplification of performance pathologies due to layers that are not airtight sealed, the difficulty in modeling system improvements, and the temptation to identify problems that are not the actual root cause.