Diagnosing performance problems in modern datacenters and distributed systems is challenging, as the root cause could be contained in any one of the system’s numerous components or, worse, could be a result of interactions among them. As distributed systems continue to increase in complexity, diagnosis tasks will only become more challenging. There is a need for a new class of diagnosis techniques capable of helping developers address problems in these distributed environments. As a step toward satisfying this need, this dissertation proposes a novel technique, called request-flow comparison, for automatically localizing the sources of performance changes from the myriad potential culprits in a distributed system to just a few potential ones. Request-flow comparison works by contrasting the workflow of how individual requests are serviced within and among every component of the distributed system between two periods: a non-problem period and a problem period. By identifying and ranking performance-affecting changes, request-flow comparison provides developers with promising starting points for their diagnosis efforts. Request workflows are obtained with less than 1% overhead via use of recently developed end-to-end tracing techniques. To demonstrate the utility of request-flow comparison in various distributed systems, this dissertation describes its implementation in a tool called Spectroscope and describes how Spectroscope was used to diagnose real, previously unsolved problems in the Ursa Minor distributed storage service and in select Google services. It also explores request-flow comparison’s applicability to the Hadoop File System. Via a 26-person user study, it identifies effective visualizations for presenting request-flow comparison’s results and further demonstrates that request-flow comparison helps developers quickly identify starting points for diagnosis.This dissertation also distills design choices that will maximize an end-to-end tracing infrastructure’s utility for diagnosis tasks and other use cases.

Ensuring the health of a distributed system with frequent updates is complicated. Many tools exist to improve developers’ comprehension and productivity in this task, but room for improvement exists. Based on previous research within request flow comparison, we propose a system design for using distributed tracing data in the process of reviewing code changes. The design is evaluated from the perspective of system performance and developer productivity using a critical production system at a large software company. The results show that the design has minimal negative performance implications while providing a useful service to the developers. They also show a positive but statistically insignificant effect on productivity during the evaluation period. To a large extent, developers adopted the tool into their workflow to explore and improve system understanding. This use case deviates from the design target of providing a method to compare changes between software versions. We conclude that the design is successful, but more optimization of functionality and a higher rate of adoption would likely improve the effects the tool could have. / Att säkerställa stabilitet i ett distribuerat system med hög frekvens av uppdateringar är komplicerat. I dagsläget finns många verktyg som hjälper utvecklare i deras förståelse och produktivitet relaterat till den här typen av problem, dock finns fortfarande möjliga förbättringar. Baserat på tidigare forskning inom teknik för att jämföra protokollförfrågningsflöden mellan mjukvaruversioner så föreslår vi en systemdesign för ett nytt verktyg. Designen använder sig av data från distribuerad tracing för att förbättra arbetsflödet relaterat till kodgranskning. Designen utvärderas både prestanda och produktivitetsmässigt under utvecklingen av ett affärskritiskt produktionssystem på ett stort mjukvaruföretag. Resultaten visar att designen har mycket låg inverkan på prestandan av systemet där det införs, samtidigt som den tillhandahåller ett användbart verktyg till utvecklarna. Resultaten visar också på en positiv men statistiskt insignifikant effekt på utvecklarnas produktivitet. Utvecklarna använde primärt verktyget för att utforska och förbättra sin egen förståelse av systemet som helhet. Detta användningsområde avvek från det ursprungliga målet med designen, vilket var att tillhandahålla en tjänst för att jämföra mjukvaruversioner med varandra. Från resultaten drar vi slutsatsen att designen som helhet var lyckad, men mer optimering av funktionalitet och mer effektivt införande av verktyget i arbetsflödet hade troligtvis resulterat i större positiva effekter på organisationen.