Context: Exploratory Testing (ET) and Scripted Testing (ST) are two of the more commonly practiced manual testing approaches in industry. ST is a traditional testing approach in which testing is carried out by executing pre-designed test cases. While in ET, learning, test designing and test execution are carried out simultaneously. In many instances, ET and ST complement each other very well in projects; however, proponents of ET claim that ET is more cost-beneficial in comparison to ST. Moreover, a few studies have indicated that ET is more effective in defect detection. Nevertheless, to the best of our knowledge, no study has been conducted to compare the costs and benefits of these two approaches. Objectives: The aim of this study was to conduct a qualitative Cost-Benefit Analysis (CBA) of ET in comparison with ST. By comparing and analyzing these two testing approaches, this study attempts to aid in decision-making with respect to how resources should be allocated for ET and ST for certain projects. Methods: The factors of costs and benefits of ET and ST were identified by conducting six semi-structured interviews in industry. Based on the analysis of these factors, a CBA model is proposed. The academic and industrial evaluation of the proposed CBA model was performed by conducting five interviews with researchers and practitioners. In addition, a qualitative CBA of a process of ET, Session-Based Testing Management (SBTM), and a process of ST, Test-Case Based Testing (TCBT) is conducted by collecting data through questionnaires and interviews with industry practitioners. A total of 22 questionnaire responses and seven interviews were analyzed. Results: By analyzing the identified cost and benefit factors, a CBA model was developed based on the testing phases stated in the ISO/IEC 29119 standard. A qualitative CBA of the SBTM process in comparison with the TCBT process was conducted by applying the CBA model in a questionnaire. The following findings were gathered from the CBA: • The differences of the SBTM and TCBT processes are identified by an analysis of the activities performed in various organizations, which the respondents belonged to. • The results of the analysis and comparison, of the costs (in terms of effort) and benefits (quality of the testing activities) of these two testing processes, are presented with respect to each testing phase. • The factors that impact the costs and benefits of using SBTM and TCBT, are summarized and discussed in this report. • The scenarios, in which SBTM and TCBT can be more cost-beneficial, are identified based on practitioners’ opinions. Conclusions: According to the survey results, industry practitioners consider SBTM as more cost-beneficial in comparison with TCBT, particularly in the test design, implementation and test execution phases. However, industry practitioners also stressed that ET should not be considered as a replacement for ST. In some contexts, testing objectives are better achieved through a more scripted approach, while, in other contexts, testing objectives will benefit more from the ability to create and improve tests as they are being executed. Whether a testing approach is valuable or cost-beneficial also depends on the context of project and the required benefits.

Context: Exploratory testing (ET) is an approach to test software with a strong focus on personal skills and freedom of the tester. ET emphasises the simultaneous design and execution of tests with minimal test documentation. Test practitioners often claim that their choice to use ET as an important alternative to scripted testing is based on several benefits ET exhibits over the scripted testing. However, these claims lack empirical evidence as there is little research done in this area. Moreover, ET is usually considered an ad-hoc way of doing testing as everyone does it differently. There have been some attempts in past to provide structure to ET. Session based test management (SBTM) is an approach that attempts to provide some structure to ET and gives some basic guidelines to structuring the test sessions. However, these guidelines are still very abstract and are very open to individuals' interpretation. Objective: The main objective of this doctoral thesis is to support practitioners in their decisions about choosing exploratory versus scripted testing. Furthermore, it is also aimed to investigate the empirical evidence in support of ET and find ways to structure ET and classify different levels of exploration that drive the choices made by exploratory testers. Another objective of this thesis is to provide a decision support system to select levels of exploration in overall test process. Method: The findings presented in this thesis are obtained through a controlled experiment with participants from industry and academia, exploratory surveys, interviews and focus groups conducted at different companies including Ericsson AB, Sony Mobile Communications, Axis Communications AB and Softhouse Consulting Baltic AB. Results: Using the exploratory survey, we found three test techniques to be most relevant in context of testing software systems and in particular heterogeneous systems. The most frequently used technique mentioned by the practitioners is ET which is not a much researched topic. We also found many interesting claims about ET in grey literature produced by practitioners in the form of informal presentations and blogs but these claims lacked any empirical evidence. Therefore, a controlled experiment was conducted with students and industry practitioners to compare ET with scripted testing. The experiment results show that ET detects significantly more critical defects compared to scripted testing and is more time efficient. However, ET has its own limitations and there is not a single way to use it for testing. In order to provide structure to ET, we conducted a study where we propose checklists to support test charter design in ET. Furthermore, two more industrial focus group studies at four companies were conducted that resulted in a taxonomy of exploration levels in ET and a decision support method for selecting exploration levels in ET. Lastly, we investigated different problems that researchers face when conducting surveys in software engineering and have presented mitigation strategies for these problems. Conclusion: The taxonomy for levels of exploration in ET, proposed in this thesis, provided test practitioners at the companies a better understanding of the underlying concepts of ET and a way to structure their test charters. A number of influence factors elicited as part of this thesis also help them prioritise which level of exploration suits more to their testing in the context of their products. Furthermore, the decision support method provided the practitioners to reconsider their current test focus to test their products in a more effective way.