Context. Colossal systems that are evolving are primarily system of systems (SOS). The system of systems are characteristic of functionally independent subsystems. These subsystems exhibit heterogeneity in terms of software or hardware. Each subsystem may reflect heterogeneity in dimensions such as the system complexity, system configuration, programming language and platforms, etc. Exploratory testing (ET) is perceived to be the best for testing such systems. An enhancement to exploratory testing is the session-based test management (SBTM) where several activities form a part of each session. These activities are mainly dependent on tester and the test charter of that session. There is lack of information in existing literature regarding a standard framework to design test charters for exploratory testing which forms the main area of focus of this thesis research. Objectives. Firstly, to investigate the design of test charters in general. Secondly, to find out the factors influencing the design of test charters. Lastly, to develop a framework to design effective test charters in SOS context. Methods. A mixed method approach that incorporates both qualitative and quantitative research methods is used. This research includes the quantitative leg of the online survey along with the interviews and literature review that are qualitative in nature. Literature review has been chosen to investigate the test charter design in general. Besides, interviews and online surveys  have been used to research regarding the factors and test charter framework.  Snowball sampling method and convenience sampling method have been used to sample the research data. Moreover, thematic analysis method is used for analyzing the qualitative data while descriptive statistics is used for quantitative data analysis. Results. The design aspects of test charter are documented, the factors influencing test charter design and the framework for effective test charter design for exploratory testing are presented. Conclusions. The thesis objectives are fulfilled. The findings on how the test charters are generally designed have helped in gaining insight on the primary elements that constitute a test charter design. Further, investigating the factors influencing the test charter design has helped in knowing the main elements affecting the test charter design. Finally, the main contribution of this thesis, the developed flexible test charter framework for exploratory testing encapsulates variables that should be considered, controlled or varied systematically during the course of testing. It is deemed to act as a guideline for practitioners for effective test charter design.

Context: Exploratory testing (ET) is an approach to test software with a strong focus on personal skills and freedom of the tester. ET emphasises the simultaneous design and execution of tests with minimal test documentation. Test practitioners often claim that their choice to use ET as an important alternative to scripted testing is based on several benefits ET exhibits over the scripted testing. However, these claims lack empirical evidence as there is little research done in this area. Moreover, ET is usually considered an ad-hoc way of doing testing as everyone does it differently. There have been some attempts in past to provide structure to ET. Session based test management (SBTM) is an approach that attempts to provide some structure to ET and gives some basic guidelines to structuring the test sessions. However, these guidelines are still very abstract and are very open to individuals' interpretation. Objective: The main objective of this doctoral thesis is to support practitioners in their decisions about choosing exploratory versus scripted testing. Furthermore, it is also aimed to investigate the empirical evidence in support of ET and find ways to structure ET and classify different levels of exploration that drive the choices made by exploratory testers. Another objective of this thesis is to provide a decision support system to select levels of exploration in overall test process. Method: The findings presented in this thesis are obtained through a controlled experiment with participants from industry and academia, exploratory surveys, interviews and focus groups conducted at different companies including Ericsson AB, Sony Mobile Communications, Axis Communications AB and Softhouse Consulting Baltic AB. Results: Using the exploratory survey, we found three test techniques to be most relevant in context of testing software systems and in particular heterogeneous systems. The most frequently used technique mentioned by the practitioners is ET which is not a much researched topic. We also found many interesting claims about ET in grey literature produced by practitioners in the form of informal presentations and blogs but these claims lacked any empirical evidence. Therefore, a controlled experiment was conducted with students and industry practitioners to compare ET with scripted testing. The experiment results show that ET detects significantly more critical defects compared to scripted testing and is more time efficient. However, ET has its own limitations and there is not a single way to use it for testing. In order to provide structure to ET, we conducted a study where we propose checklists to support test charter design in ET. Furthermore, two more industrial focus group studies at four companies were conducted that resulted in a taxonomy of exploration levels in ET and a decision support method for selecting exploration levels in ET. Lastly, we investigated different problems that researchers face when conducting surveys in software engineering and have presented mitigation strategies for these problems. Conclusion: The taxonomy for levels of exploration in ET, proposed in this thesis, provided test practitioners at the companies a better understanding of the underlying concepts of ET and a way to structure their test charters. A number of influence factors elicited as part of this thesis also help them prioritise which level of exploration suits more to their testing in the context of their products. Furthermore, the decision support method provided the practitioners to reconsider their current test focus to test their products in a more effective way.