Exploratory testing (ET) is an agile approach towards software testing. It is simultaneous learning, testing, reporting of problems and then generating new tests based on the learning. The aim of this thesis is to investigate exploratory testing in the industry. This thesis was proposed by Sogeti AB in Lund. Therefore the research questions were formulated and finalized according to their requirements and consent. Initially, a literature survey was conducted to study the different concepts of ET. After that a descriptive case study was conducted to investigate ET practices in an industrial environment. The research methodology used is qualitative. It comprises of ten semistructured interviews with industrial practitioners including both ET testers and customers having different perspectives. The study explores the misconceptions about ET and also identifies its claimed pros and cons. The investigation also studies its suitability with different types of testing and its effective combination with other techniques. In addition to the interviews, a survey was conducted to further investigate and analyze our findings with a larger sample of 25 practitioners. A framework for a session-based exploratory testing as practiced by the industrial partner is presented as part of our results. ET has different opinions and it is hard to say whether they are misconceptions or not. The most prominent advantage of ET from the testers view point is utilization of tester’s creativity and experience however, customers think the biggest benefit of ET approach is its ability to provide focused testing. The major disadvantage according to the testers resulted to be, difficulty in finding testers with appropriate experience, skill set and knowledge. On the other hand Inability to produce decision material was the biggest concern of the customers. The results also showed that ET approach was preferred to be used in combination with other testing techniques. Combination of Risk Based Testing and ET was reported to be used the most. ET was considered very suitable for situations where the testing requirements included learning of the system, time constraints and complementary testing. It was considered unsuitable for testing critical systems. / +46- (0)762005979

Context. Colossal systems that are evolving are primarily system of systems (SOS). The system of systems are characteristic of functionally independent subsystems. These subsystems exhibit heterogeneity in terms of software or hardware. Each subsystem may reflect heterogeneity in dimensions such as the system complexity, system configuration, programming language and platforms, etc. Exploratory testing (ET) is perceived to be the best for testing such systems. An enhancement to exploratory testing is the session-based test management (SBTM) where several activities form a part of each session. These activities are mainly dependent on tester and the test charter of that session. There is lack of information in existing literature regarding a standard framework to design test charters for exploratory testing which forms the main area of focus of this thesis research. Objectives. Firstly, to investigate the design of test charters in general. Secondly, to find out the factors influencing the design of test charters. Lastly, to develop a framework to design effective test charters in SOS context. Methods. A mixed method approach that incorporates both qualitative and quantitative research methods is used. This research includes the quantitative leg of the online survey along with the interviews and literature review that are qualitative in nature. Literature review has been chosen to investigate the test charter design in general. Besides, interviews and online surveys  have been used to research regarding the factors and test charter framework.  Snowball sampling method and convenience sampling method have been used to sample the research data. Moreover, thematic analysis method is used for analyzing the qualitative data while descriptive statistics is used for quantitative data analysis. Results. The design aspects of test charter are documented, the factors influencing test charter design and the framework for effective test charter design for exploratory testing are presented. Conclusions. The thesis objectives are fulfilled. The findings on how the test charters are generally designed have helped in gaining insight on the primary elements that constitute a test charter design. Further, investigating the factors influencing the test charter design has helped in knowing the main elements affecting the test charter design. Finally, the main contribution of this thesis, the developed flexible test charter framework for exploratory testing encapsulates variables that should be considered, controlled or varied systematically during the course of testing. It is deemed to act as a guideline for practitioners for effective test charter design.

Context: Software testing is an integral part of software development life cycle. To improve the quality of software there are different testing approaches practiced over the years. Traditionally software testing is carried out by following approach focusing on prior test design. While exploratory testing is an approach to test software where the tester does not require to follow a specific test design. But rather, exploratory testing should facilitate the tester in testing the complete system comprehensively. Exploratory testing is seen by some, as a way to conduct simultaneous learning, test design and execution of tests simultaneously. While others point to exploratory testing enabling constant evolution of tests in an easy manner. Objectives: In this study we have investigated the field of exploratory testing in literature and industry to understand its perception and application. Further among the stated claims by practitioners, we selected defect detection efficiency and effectiveness claim for empirical validation through an experiment and survey. Methods: In this study, a systematic literature review, interview, experiment and survey are conducted. In the systematic review a number of article sources are used, including IEEE Xplore, ACM Digital Library, Engineering village, Springer Link, Google Scholar and Books database. The systematic review also includes the gray literature published by the practitioners. The selection of studies was done using two-phase and tollgate approach. A total of 47 references were selected as primary studies. Eight semi-structures interviews were conducted with industry practitioners. Experiment had total 4 iterations and 70 subjects. The subjects were selected from industry and academia. The experimental design used was one factor with two interventions and one response variable. Results: Based on our findings from literature review and interviews, the understanding of exploratory testing has improved over the period but still lacks empirical investigation. The results drawn from experimental and survey data shows that exploratory testing proved effective and efficient in finding more critical bugs in limited time. Conclusions: We conclude that exploratory testing has a lot of potential and much more to offer to testing industry. But more empirical investigation and true facts and figures are required to motivate the testing industry to adapt it. We have reported a number of advantages, disadvantages, challenges and factors in this study. We further investigated the claims stated by the ET practitioners through an experiment and survey. The statistical tests were conducted on the collected data to draw meaningful results. We found statistical significance difference in number of true defects found. Using exploratory testing approach testers found far more defects than test case based testing. Although, there was no statistical significance difference between the two approaches for false defects. / Slutsatser: Vi anser att det experimentella tester har stor potential och mycket mer att erbjuda testning industrin. Men mer empirisk undersökning och sann fakta och siffror är skyldiga att motivera testning industrin att anpassa den. Vi har rapporterat en rad fördelar, nackdelar, utmaningar och faktorer i denna studie. Vi undersökte vidare fordringar anges av ET utövare genom ett experiment och undersökning. De statistiska test genomfördes på insamlade data för att dra meningsfulla resultat. Vi fann statistisk signifikans skillnaden i antalet sann fel som upptäcks. Använda utforskande testning strategi testare fann långt fler fel än testfall baserat testning. Även om det inte fanns någon statistisk signifikans skillnad mellan de två synsätten för falska defekter. / 0046 73 651 8048

Context. Software testing is one of the crucial phases in software development life cycle (SDLC). Among the different manual testing methods in software testing, Exploratory testing (ET) uses no predefined test cases to detect defects. Objectives. The main objective of this study is to test the effectiveness of ET in detecting defects at different software test levels. The objective is achieved by formulating hypotheses, which are later tested for acceptance or rejection. Methods. Methods used in this thesis are literature review and experiment. Literature review is conducted to get in-depth knowledge on the topic of ET and to collect data relevant to ET. Experiment was performed to test hypotheses specific to the three different testing levels : unit , integration and system. Results. The experimental results showed that using ET did not find all the seeded defects at the three levels of unit, integration and system testing. The results were analyzed using statistical tests and interpreted with the help of bar graphs. Conclusions. We conclude that more research is required in generalizing the benefits of ET at different test levels. Particularly, a qualitative study to highlight factors responsible for the success and failure of ET is desirable. Also we encourage a replication of this experiment with subjects having a sound technical and domain knowledge. / H.no.2-7-644, Mukrumpura, Karimnagar, Pincode:500001, India, Phone number: +91-9908644775

Software testing is a key phase in software development lifecycle. Testing objectives corresponds to the discovery and detection of faults, which can be attained by utilizing manual or automated testing approaches. In this thesis, we are mainly concerned with the manual test approaches. The most commonly used manual testing approaches in the software industry are the Exploratory Testing (ET) approach and the Test Case Based Testing (TCBT) approach. TCBT is primarily used by software testers to formulize and guide their testing tasks and set the theoretical principles for testing. On the other hand ET is simultaneous learning, test design, and test execution. Software testing might benefit from an intelligent combination of these approaches of testing however there is no proof of any formal process that accommodates the usage of both test approaches in a combination. This thesis presents a process for Mix Testing (MT) based on the strengths and weaknesses of both test approaches, identified through a systematic literature review and interviews with testers in a software organization. The new process is defined through the mapping of weaknesses of one approach to the strengths of other. Static validation of the MT process through interviews in the software organization suggested that MT has ability to resolve the problems of both test approaches to some extent. Furthermore, MT was validated by conducting an experiment in an industrial setting. The analysis of the experimentation results indicated that MT has better defect detection than TCBT and less than ET. In addition, the results of the experiments also indicate that MT provides equal functionality coverage as compared to ET and TCBT.

Mjukvarutestning har, sedan introduktionen av datorer, varit föremål för forskning. Idag, när datorer och mikroprocessorer alltmer integreras i produkter som omger oss i vårt dagliga liv, så ökar vikten av effektiv och korrekt mjukvarutestning. Tidigare forskning visar att utforskande testning är en effektiv metod för att upptäcka programvarubuggar. Men, med tids- och resursaspekterna i åtanke finns det andra metoder som kan vara effektivare. En möjlig sådan metod är parvis testning. Litteraturen visar att även denna metod har bra potential för att kunna identifiera programvarubuggar. Det finns dock inte så mycket forskning om jämförelsen av dessa två metoder, vilket är anledningen till att denna studie genomfördes. Denna förklarande fallstudie utvärderar hur utforskande testning presterar jämfört med parvis testning. Aspekter som beaktades i utvärderingsprocessen var antalet upptäckta defekter, allvarlighetsgraden av de hittade defekterna och vilka typer av defekter som hittades. Data samlades in genom undersökningar, intervjuer, deltagande observationer och direkta observationer. Med all denna data som samlats in, drogs slutsatsen att svaret på denna rapports forskningsfrågor är tvetydiga. Det finns fördelar med båda teknikerna, och beroende på förutsättningarna för utforskande testning så kan parvis testning prestera likvärdigt. En sak märktes dock under studien som inte var en del av den ursprungliga omfattningen, och det var styrkan som utforskande testning har som ett läroverktyg. / Software testing has been a subject of research since the introduction of computers. Today, when computers and microprocessors are increasingly integrated into the products surrounding us in our daily lives, the importance of effective and accurate software testing increases. Previous research shows that exploratory testing is an effective method for detecting software bugs. Still, with the time and resource aspects considered, there are other potentially more time and resource-effective methods. One such possible method is the pairwise testing method. The literature also shows that this method is effective for finding software bugs. There is, however, not that much research about the comparison of these two methods, which is why this study was conducted. This explanatory case study evaluates how exploratory testing performs compared with pairwise testing. Aspects considered in the evaluation process were the number of detected defects, the severity distribution of the found defects, and what types of defects were found. The data was collected through surveys, interviews, participant observations, and direct observations. With the data collected, it was concluded that the answers to the research questions of this study are ambiguous. There are benefits with both of the techniques, and depending on the exploratory testing conditions, the pairwise technique can perform comparably as the exploratory testing. However, one thing noticed during the study that was not part of the original scope was the strength of exploratory testing as a learning tool. Lastly, some hypotheses were stated, supported by the collected data.

Context: Exploratory Testing (ET) and Scripted Testing (ST) are two of the more commonly practiced manual testing approaches in industry. ST is a traditional testing approach in which testing is carried out by executing pre-designed test cases. While in ET, learning, test designing and test execution are carried out simultaneously. In many instances, ET and ST complement each other very well in projects; however, proponents of ET claim that ET is more cost-beneficial in comparison to ST. Moreover, a few studies have indicated that ET is more effective in defect detection. Nevertheless, to the best of our knowledge, no study has been conducted to compare the costs and benefits of these two approaches. Objectives: The aim of this study was to conduct a qualitative Cost-Benefit Analysis (CBA) of ET in comparison with ST. By comparing and analyzing these two testing approaches, this study attempts to aid in decision-making with respect to how resources should be allocated for ET and ST for certain projects. Methods: The factors of costs and benefits of ET and ST were identified by conducting six semi-structured interviews in industry. Based on the analysis of these factors, a CBA model is proposed. The academic and industrial evaluation of the proposed CBA model was performed by conducting five interviews with researchers and practitioners. In addition, a qualitative CBA of a process of ET, Session-Based Testing Management (SBTM), and a process of ST, Test-Case Based Testing (TCBT) is conducted by collecting data through questionnaires and interviews with industry practitioners. A total of 22 questionnaire responses and seven interviews were analyzed. Results: By analyzing the identified cost and benefit factors, a CBA model was developed based on the testing phases stated in the ISO/IEC 29119 standard. A qualitative CBA of the SBTM process in comparison with the TCBT process was conducted by applying the CBA model in a questionnaire. The following findings were gathered from the CBA: • The differences of the SBTM and TCBT processes are identified by an analysis of the activities performed in various organizations, which the respondents belonged to. • The results of the analysis and comparison, of the costs (in terms of effort) and benefits (quality of the testing activities) of these two testing processes, are presented with respect to each testing phase. • The factors that impact the costs and benefits of using SBTM and TCBT, are summarized and discussed in this report. • The scenarios, in which SBTM and TCBT can be more cost-beneficial, are identified based on practitioners’ opinions. Conclusions: According to the survey results, industry practitioners consider SBTM as more cost-beneficial in comparison with TCBT, particularly in the test design, implementation and test execution phases. However, industry practitioners also stressed that ET should not be considered as a replacement for ST. In some contexts, testing objectives are better achieved through a more scripted approach, while, in other contexts, testing objectives will benefit more from the ability to create and improve tests as they are being executed. Whether a testing approach is valuable or cost-beneficial also depends on the context of project and the required benefits.

Context: Exploratory testing (ET) is an approach to test software with a strong focus on personal skills and freedom of the tester. ET emphasises the simultaneous design and execution of tests with minimal test documentation. Test practitioners often claim that their choice to use ET as an important alternative to scripted testing is based on several benefits ET exhibits over the scripted testing. However, these claims lack empirical evidence as there is little research done in this area. Moreover, ET is usually considered an ad-hoc way of doing testing as everyone does it differently. There have been some attempts in past to provide structure to ET. Session based test management (SBTM) is an approach that attempts to provide some structure to ET and gives some basic guidelines to structuring the test sessions. However, these guidelines are still very abstract and are very open to individuals' interpretation. Objective: The main objective of this doctoral thesis is to support practitioners in their decisions about choosing exploratory versus scripted testing. Furthermore, it is also aimed to investigate the empirical evidence in support of ET and find ways to structure ET and classify different levels of exploration that drive the choices made by exploratory testers. Another objective of this thesis is to provide a decision support system to select levels of exploration in overall test process. Method: The findings presented in this thesis are obtained through a controlled experiment with participants from industry and academia, exploratory surveys, interviews and focus groups conducted at different companies including Ericsson AB, Sony Mobile Communications, Axis Communications AB and Softhouse Consulting Baltic AB. Results: Using the exploratory survey, we found three test techniques to be most relevant in context of testing software systems and in particular heterogeneous systems. The most frequently used technique mentioned by the practitioners is ET which is not a much researched topic. We also found many interesting claims about ET in grey literature produced by practitioners in the form of informal presentations and blogs but these claims lacked any empirical evidence. Therefore, a controlled experiment was conducted with students and industry practitioners to compare ET with scripted testing. The experiment results show that ET detects significantly more critical defects compared to scripted testing and is more time efficient. However, ET has its own limitations and there is not a single way to use it for testing. In order to provide structure to ET, we conducted a study where we propose checklists to support test charter design in ET. Furthermore, two more industrial focus group studies at four companies were conducted that resulted in a taxonomy of exploration levels in ET and a decision support method for selecting exploration levels in ET. Lastly, we investigated different problems that researchers face when conducting surveys in software engineering and have presented mitigation strategies for these problems. Conclusion: The taxonomy for levels of exploration in ET, proposed in this thesis, provided test practitioners at the companies a better understanding of the underlying concepts of ET and a way to structure their test charters. A number of influence factors elicited as part of this thesis also help them prioritise which level of exploration suits more to their testing in the context of their products. Furthermore, the decision support method provided the practitioners to reconsider their current test focus to test their products in a more effective way.