None of today’s large scale systems could function without the reliable availability of a varied range of network communications capabilities. Whilst software, hardware and communications technologies have been advancing throughout the past two decades, the methods commonly used by industry for testing large scale systems which incorporate critical communications interfaces have not kept pace. This thesis argues for the need for a specifically tailored framework to achieve effective testing of communications-critical large scale systems (CCLSS). The thesis initially discusses how generic test approaches are leading to inefficient and costly test activities in industry. The thesis then presents the form and features of an alternative CCLSS domain-specific test framework, develops its ideas further into a detailed and structured test approach for one of its layers, and then provides a detailed example of how this framework can be applied using a real-life case study. The thesis concludes with a qualitative as well a simulation-based evaluation of the framework’s benefits observed during the case study and an evaluation by expert external participants considering whether similar benefits can be realised if the framework is adopted for the testing of other comparable systems. Requirements data from a second CCLSS is included in the evaluation by external participants as a second smaller case study.

Testing is the dominating method for quality assurance of industrial software. Despite its importance and the vast amount of resources invested, there are surprisingly limited efforts spent on testing research, and the few industrially applicable results that emerge are rarely adopted by industry. At the same time, the software industry is in dire need of better support for testing its software within the limited time available. Our aim is to provide a better understanding of how test cases are created and applied, and what factors really impact the quality of the actual test. The plethora of test design techniques (TDTs) available makes decisions on how to test a difficult choice. Which techniques should be chosen and where in the software should they be applied? Are there any particular benefits of using a specific TDT? Which techniques are effective? Which can you automate? What is the most beneficial way to do a systematic test of a system? This thesis attempts to answer some of these questions by providing a set of guidelines for test design, including concrete suggestions for how to improve testing of industrial software systems, thereby contributing to an improved overall system quality. The guidelines are based on ten studies on the understanding and use of TDTs. The studies have been performed in a variety of system domains and consider several different aspects of software test. For example, we have investigated some of the common mistakes in creating test cases that can lead to poor and costly testing. We have also compared the effectiveness of different TDTs for different types of systems. One of the key factors for these comparisons is a profound understanding of faults and their propagation in different systems. Furthermore, we introduce a taxonomy for TDTs based on their effectiveness (fault finding ability), efficiency (fault finding rate), and applicability. Our goal is to provide an improved basis for making well-founded decisions regarding software testing, together with a better understanding of the complex process of test design and test case writing. Our guidelines are expected to lead to improvements in testing of complex industrial software, as well as to higher product quality and shorter time to market.

Test efficiency measures the cost-effectiveness of a test organisation and it is measured by dividing the number of defects found in a test by the effort needed to perform the test. This thesis project investigated whether the Mobile Positioning Centre (MPC) site at Ericsson AB could improve their test efficiency or not. The purpose of the project was to identify areas that could increase the test efficiency by investigating state of the art literature and evaluating the test process at MPC. The evaluation identified unit testing and debugging as the areas at MPC where the test efficiency could be increased the most. The project work resulted in an implementation proposal containing a number of actions that would increase the test efficiency at MPC. Primarily, the actions comprised an improved test tool environment; both enhancements for the existing tools and design suggestions for new test tools. The implementation proposal also included suggestions for how to integrate the test tool improvements with the organisation and processes at MPC. / Uppsatsen utvärderar testprocessen på en Ericsson site (the Mobile Positioning Centre).