Regression testing is an essential part of developing and maintaining software. It helps verify that changes to the software have not introduced any new bugs, and that the functionality still works as intended. However, for this verification to be valid, the executed tests must be assumed to be deterministic, i.e. produce the same output under the same circumstances. Unfortunately, this is not always the case. A test that exhibits non-deterministic behavior is said to be flaky. Flaky tests can severely inhibit the benefits of regression testing, as developers must figure out whether a failing test is due to a bug in the system under test (SUT) or test flakiness. Moreover, the non-deterministic nature of flaky tests poses several problems. Not only are the failures difficult to reproduce and debug, but developers are more likely to ignore the outcome of flaky tests, potentially leading to overlooked bugs in the SUT. The aim of this thesis was to investigate the applicability of execution tracing techniques as a means of providing root cause analysis for flaky tests in the randomness and network categories. This involved reproducing and studying flakiness, as well as implementing and evaluating a prototype with the ability to analyze runtime behavior in flaky tests. To gain a better understanding of reproducibility and common traits among flaky tests in the selected categories, a pre-study was conducted. Based on the outcome of the pre-study and findings in related literature, the network category was dropped entirely, and two techniques were chosen to be implemented. The implementation process resulted in the FlakyPy tool, a plugin for pytest that provides root cause analysis aimed at randomness flakiness. When run against a dataset of 22 flaky tests, the tool was able to identify potential root causes in 15 of these. This serves as an indication that execution tracing has the potential of detecting possible root causes in flaky randomness tests in Python. However, more research is needed to evaluate how developers perceive the usefulness of such tools.

With today’s focus on Continuous Integration, test cases are used to ensure the software’s reliability when integrating and developing code. Test cases that behave in an undeterministic manner are known as flaky tests, which threatens the software’s reliability. Because of flaky test’s undeterministic nature, they can be troublesome to detect and correct. This is causing companies to spend great amount of resources on flaky tests since they can reduce the quality of their products and services. The aim of this thesis was to develop a usable tool that can automatically detect flakiness in the Randomness category. This was done by initially locating and rerunning flaky tests found in public Git repositories. By scanning the resulting pytest logs from the tests that manifested flaky behaviour, noting indicators of how flakiness manifests in the Randomness category. From these findings we determined tracing to be a viable option of detecting Randomness as a cause of flakiness. The findings were implemented into our proposed tool FlakyReporter, which reruns flaky tests to determine if they pertain to the Randomness category. Our FlakyReporter tool was found to accurately categorise flaky tests into the Randomness category when tested against 25 different flaky tests. This indicates the viability of utilizing tracing as a method of categorizing flakiness.