<p>The products The Company manufactures are used in an area of the industry where errors leading to a stop in production can be quite expensive. Therefore are testing of the products important and the tests can also give indications about the quality of the products.</p><p>The Company is in a phase where they are developing a new product line to support all existing and future products. In this phase, it was decided that all products will use a common framework for unit testing and a common build system for all products. One part of the thesis was to investigate and evaluate different frameworks for unit testing and tools for a build system. The unit test framework that were evaluated are Cppunit, cfix, NUnit, Boost test library, Unit Testing++ and CxxTest. The result of the evaluation was that CppUnit were recommended. For the build system MSBuild, NAnt, Automated Build Studio and Cruise Control .NET were evaluated. The recommended tools for a build system is MSBuild in combination with Cruise Control .Net if The Company is interested in the functionality Cruise Control .Net has to offer.</p><p>The Company also has an interest in evaluating the current test processes and identify improvements as a part of The Company’ s objective were all products should follow a common test process. In order to identify these improvements a literature study of four test process improvement frameworks (Test Process Improvement, Test Maturity Model Integrated, Minimal Test Practice Framework and Test Improvement Model) were carried out. Out of these four frameworks Test Process Improvement (TPI) were chosen to assist when identifying improvements. With the help of TPI a limited assessment took place to give indications about the test maturity for three of The Company’s products where two of the products had low maturity. Results of the improvement measures can be summed up with the need to harmonize the documents, standardize and document the various processes.</p><p>As a last part of the thesis the possibility to automate testing of two of the products graphical user interfaces with the program Test Complete were investigated. For one of the products the result was that it worked satisfactorily and for the other product it did not work at all. This resulted in recommendations for how The Company should proceed with automated testing of the graphical user interface.</p> / <p>De produkter som företaget tillverkar används i huvudsak inom ett område av branschen där fel som leder till stopp i produktionen kan vara ganska dyrt. Detta gör testning av produkterna viktiga och tester kan också ge indikationer om kvaliteten på produkterna.</p><p>Företaget är i en fas där man utvecklar en ny produktlinje som ska stödja alla befintliga och framtida produkter. I denna fas har man beslutat att alla produkterna ska använda ett gemensant ramverk för enhetstestning och även använda ett gemensamt byggsystem för samtliga produkter. En del av examensarbetet var att undersöka och utvärdera olika ramverk för enhets testning och verktyg för byggsystem. De ramverk som utvärderades var CppUnit, cfix, NUnit, Boost test library, unitTest++ och CxxTest. Utvärderingen ledde fram till att CppUnit rekommenderades till företaget. Verktyg som utvärderades för byggsystem var MSBuild, NAnt, Automated Build Studio och Cruise Control .Net. För byggsystem rekommenderas MSBuild i kombination med Cruise Control .Net ifall företaget är intresserade av den extra funktionalitet som Cruise Control .Net har att erbjuda.</p><p>Företaget har även ett intresse av att utvärdera den nuvarande testprocessen och identifiera förbättringar som ett led i att befintliga produkter skall följa en gemensam testprocess. För att kunna identifiera dessa förbättringar utfördes en litteraturstudie över fyra stycken test process förbättrings ramverk (Test Process Improvent, Test Maturity Model integrated, Minimal Test Practice Framework och Test Improvement Model). Utav dessa fyra ramverk så valdes Test Process Improvement (TPI) ut som en hjälp för att identifiera förbättringar. Med hjälp av TPI utfördes en begränsad bedömning av företagets mogenhetsgrad på tre produkter, där två av produkterna har en låg mogenhetsgrad. Resultat av förbättringsåtgärderna kan sammanfattas med att man bör harmonisera dokument, standardisera och dokumentera olika processer.</p><p>Som en sista del i examensarbetet så utvärderades möjligheten att automatisera testning på två av produkternas grafiska användargränssnitt med programmet TestComplete. För en av produkterna blev resultatet att det fungerade tillfredställande och för den andra produkten så fungerade det inte alls. Resultatet blev rekommendationer för hur företaget borde gå vidare med automatisering av testnig på det grafiska användargränssnittet.</p>

The products The Company manufactures are used in an area of the industry where errors leading to a stop in production can be quite expensive. Therefore are testing of the products important and the tests can also give indications about the quality of the products. The Company is in a phase where they are developing a new product line to support all existing and future products. In this phase, it was decided that all products will use a common framework for unit testing and a common build system for all products. One part of the thesis was to investigate and evaluate different frameworks for unit testing and tools for a build system. The unit test framework that were evaluated are Cppunit, cfix, NUnit, Boost test library, Unit Testing++ and CxxTest. The result of the evaluation was that CppUnit were recommended. For the build system MSBuild, NAnt, Automated Build Studio and Cruise Control .NET were evaluated. The recommended tools for a build system is MSBuild in combination with Cruise Control .Net if The Company is interested in the functionality Cruise Control .Net has to offer. The Company also has an interest in evaluating the current test processes and identify improvements as a part of The Company’ s objective were all products should follow a common test process. In order to identify these improvements a literature study of four test process improvement frameworks (Test Process Improvement, Test Maturity Model Integrated, Minimal Test Practice Framework and Test Improvement Model) were carried out. Out of these four frameworks Test Process Improvement (TPI) were chosen to assist when identifying improvements. With the help of TPI a limited assessment took place to give indications about the test maturity for three of The Company’s products where two of the products had low maturity. Results of the improvement measures can be summed up with the need to harmonize the documents, standardize and document the various processes. As a last part of the thesis the possibility to automate testing of two of the products graphical user interfaces with the program Test Complete were investigated. For one of the products the result was that it worked satisfactorily and for the other product it did not work at all. This resulted in recommendations for how The Company should proceed with automated testing of the graphical user interface. / De produkter som företaget tillverkar används i huvudsak inom ett område av branschen där fel som leder till stopp i produktionen kan vara ganska dyrt. Detta gör testning av produkterna viktiga och tester kan också ge indikationer om kvaliteten på produkterna. Företaget är i en fas där man utvecklar en ny produktlinje som ska stödja alla befintliga och framtida produkter. I denna fas har man beslutat att alla produkterna ska använda ett gemensant ramverk för enhetstestning och även använda ett gemensamt byggsystem för samtliga produkter. En del av examensarbetet var att undersöka och utvärdera olika ramverk för enhets testning och verktyg för byggsystem. De ramverk som utvärderades var CppUnit, cfix, NUnit, Boost test library, unitTest++ och CxxTest. Utvärderingen ledde fram till att CppUnit rekommenderades till företaget. Verktyg som utvärderades för byggsystem var MSBuild, NAnt, Automated Build Studio och Cruise Control .Net. För byggsystem rekommenderas MSBuild i kombination med Cruise Control .Net ifall företaget är intresserade av den extra funktionalitet som Cruise Control .Net har att erbjuda. Företaget har även ett intresse av att utvärdera den nuvarande testprocessen och identifiera förbättringar som ett led i att befintliga produkter skall följa en gemensam testprocess. För att kunna identifiera dessa förbättringar utfördes en litteraturstudie över fyra stycken test process förbättrings ramverk (Test Process Improvent, Test Maturity Model integrated, Minimal Test Practice Framework och Test Improvement Model). Utav dessa fyra ramverk så valdes Test Process Improvement (TPI) ut som en hjälp för att identifiera förbättringar. Med hjälp av TPI utfördes en begränsad bedömning av företagets mogenhetsgrad på tre produkter, där två av produkterna har en låg mogenhetsgrad. Resultat av förbättringsåtgärderna kan sammanfattas med att man bör harmonisera dokument, standardisera och dokumentera olika processer. Som en sista del i examensarbetet så utvärderades möjligheten att automatisera testning på två av produkternas grafiska användargränssnitt med programmet TestComplete. För en av produkterna blev resultatet att det fungerade tillfredställande och för den andra produkten så fungerade det inte alls. Resultatet blev rekommendationer för hur företaget borde gå vidare med automatisering av testnig på det grafiska användargränssnittet.

Testförbättringsmodeller har utvecklats och blivit fler sedan 90-talet. Dessa modeller strävar efter att ge team och organisationer struktur inom testarbete. TMM, som är en av de äldsta testförbättingsmodellerna, togs fram i början av 90-talet och har sedan gett inspiration till flera testförbättringsmodeller med mognadsnivåer. Två av dem är TMMi och TIM. Modellerna, som båda ursprungligen är bearbetningar av TMM, har olika strukturer när det kommer till hur olika delar av testfunktionen förbättras. I en fallstudie har TMMi och TIM jämförts mot varandra för att få fram skillnaderna i modellerna samt rekommendationer hos ett team som arbetar med flera konsultuppdrag. TIM utvärderas även om hur aktuell den är i ett agilt konsultföretag.Enkät och intervju användes för att samla in data och modellerna användes för att generera rekommendationer till teamet.Det framkommer att TIM kan appliceras på en agil konsultverksamhet, främst för att TIM lägger fram vad som ska göras och det agila arbetssättet lägger fram hur det ska göras.Den främsta skillnaden som upptäcktes mellan TIM och TMMi ligger i deras struktur gällande mognadsnivåer. TIM uppmuntrar till en balanserad förbättring genom individuell bedömning av testprocessområden (KPA), så att det studerade teamet har en bättre förståelse för vilka av de som främst behöver förbättras, till skillnad från TMMi som är striktare vid sin bedömning.TIM i förhållande till TMMi har rekommendationer som riktar sig till enskilda testare på en lägre mognadsnivå, vilket påverkar graden av självständighet för testning. TMMi:s rekommendationer betonar att röktester bör utföras av teamet i större utsträckning än vad som sker för tillfället. / Test improvement models have evolved and become more common since the 1990s. These models strive to give teams and organizations structure in their testing work. TMM, one of the oldest models, was developed in the early 90's and has since provided inspiration for several test improvement models with maturity levels thereafter. Two of them are TMMi and TIM. Both models, which are further worked on from TMM originally, have two different structures when it comes to how different parts of the test function are improved. In a case study, TMMi and TIM have been compared against each other to identify the differences in the models and recommendations from a team working on several consulting assignments. TIM is also evaluated on how current it is in an agile consulting company. The questionnaire and interview were used to collect data and the models were used to generate recommendations for the team. It appears that TIM can be applied to an agile consulting business, mainly because TIM focuses on what to do and the agile methodology focuses on how to do it. The main difference found between TIM and TMMi lies in their structure regarding maturity levels. TIM encourages balanced improvement through separate assessment of key process areas (KPA), so that the studied team has a better understanding of which of them needs to be improved primarily, unlike TMMi, which is stricter in its assessment. TIM in relation to TMMi has recommendations aimed at individual testers at a lower maturity level, which affects the degree of independence for testing. TMMi's recommendations emphasize that smoke tests should be performed by the team to a greater extent than is currently happening.

Effectiveness of system testing, in specific phases such as design verification, requirements validation, test planning and generation, system integration and system testing are considered. Software as well as hardware test issues are reviewed. Metrics related to system testing are specified. The current system testing processes in a large Turkish military electronic systems manufacturer are reviewed, specific problems are identified and UML-based behavioral testing is proposed as an improved process. The current process and the proposed process are compared in terms of test coverage, test effectiveness and test effort metrics.

This report discusses the importance of test process evaluation in order to improve a test model and to provide developer- and management feedback. The report results in a test evaluation framework, developed in cooperation with a department at Ericsson Software Technology in Karlsrona. The framework is a result of discussions with the developers regarding performed testing, studying defect types from past projects and by analyzing the result from a small survey answered by some of the developers at Ericsson. The overall project aim was to evaluate performed testing in order to improve the test model. This requires a good insight of the test process, which is provided by the developed test evaluation framework. The test process is visualized by extracting test process data, making it possible to achieve the project aim. The project aim can be divided into the three following areas: Firstly to evaluate if the current test model is followed as expected, for example are all test techniques used according to the test model? Secondly to evaluate how well the test model fulfills predefined expectations, i.e. is a defect detected with the expected test technique and in the expected test phase? Finally to evaluate if there are any problematic defects that should receive extra attention during a project such as if one or several defect types occurs more frequently than others? The framework is based on another framework, Orthogonal Defect Classification [Chillarege92], combined with the research area Test Technique Effectiveness. The aim of this combination was to support the developed framework. Further a specific part of the framework is focusing on developer- and management feedback. / Dan Bengtsson Västra Stationstorget 7 222 37 Lund

In this thesis, the overall conclusion is that combination strategies, (i.e., test case selection methods that manage the combinatorial explosion of possible things to test), can improve the software testing in most organizations. The research underlying this thesis emphasizes relevance by working in close relationship with industry. Input parameter models of test objects play a crucial role for combination strategies. These models consist of parameters with corresponding parameter values and represent the input space and possibly other properties, such as state, of the test object. Test case selection is then defined as the selection of combinations of parameter values from these models. This research describes a complete test process, adapted to combination strategies. Guidelines and step-by-step descriptions of the activities in process are included in the presentation. In particular, selection of suitable combination strategies, input parameter modeling and handling of conflicts in the input parameter models are addressed. It is also shown that several of the steps in the test process can be automated. The test process is validated through a set of experiments and case studies involving industrial testers as well as actual test problems as they occur in industry. In conjunction with the validation of the test process, aspects of applicability of the combination strategy test process (e.g., usability, scalability and performance) are studied. Identification and discussion of barriers for the introduction of the combination strategy test process in industrial projects are also included. This research also presents a comprehensive survey of existing combination strategies, complete with classifications and descriptions of their different properties. Further, this thesis contains a survey of the testing maturity of twelve software-producing organizations. The data indicate low test maturity in most of the investigated organizations. Test managers are often aware of this but have trouble improving. Combination strategies are suitable improvement enablers, due to their low introduction costs.

As more systems are becoming embedded hardware-based, challenges regarding software safety and considerable consequences of their failure arise. Various safety standards assure certain safety aspects of systems, addressing areas including testing. The safety standards chosen for this thesis are ISO/IEC/IEEE 29119-2 &amp; 3, IEC 61508-1 &amp; 3, ISO 13849-1 &amp; 2, and ISO/IEC/IEEE 12207:2017. This thesis tackles the problem of compliance with safety standards by utilising a lightweight assessment method, leading to recommendations for improving the test process of an industrial control system. A case study is performed on an automation company to achieve the objectives of this thesis. The method used for the qualitative data analysis results in recommendations regarding the compliance of the company's test process with selected safety standards. As the final step, the execution of a focus group research leads to the industrial evaluation of the recommendations and assessment results. The company's development process fully complies with 22% and fails to comply with 58% of the extracted requirements from the selected safety standards. Furthermore, the thesis results in recommendations for improving the test process of an industrial control system. As a result of performing the case study, a method for a lightweight assessment of the development process of industrial control systems is achieved. The generic method follows five steps, firstly tabulating the data to attain assessment criteria and items, used by the assessment step to get a compliance degree per requirement. The analysis step comes next to shed light on areas of strength and weakness, leading to recommendations. The final step evaluates and refines the recommendations according to the results of a focus group. Further development of the method used in this thesis can lead to a generic method for assessing development processes, concerning safety standards, using limited resources.  The results of this generic method can lead to recommendations for test process improvements of control systems via safety standards.

Context: Software testing is a crucial area in software development. It has been found that there is little evidence on how testing is performed in software automotive domain. Challenges in this domain have not yet been fully evaluated. Furthermore, interactions between test procedures, and other activities of software development are left unexposed. Assessment of test process is an important part of improvement initiatives. Objectives: In this study we perform an in-depth investigation to identify challenges which lead to wastes in test process in the context of automotive software domain. To this end, we also identified strengths that add value in test process. Methods: We performed a Case study at a Sweden based large automotive organization. To this end, we conducted semi-structured interviews and used Grounded Theory method for interview data analysis. We used Value Stream Mapping (VSM), a Lean methodology tool for test process assessment. Furthermore, we conducted Systematic Literature Reviews to look for evidence of similar problems and solutions offered in peer-reviewed literature. Results: We grouped the identified challenges and strengths into 10 and 4 categories respectively. We also identified 7 kinds of waste and 4 kinds of values based using VSM. Based on systematic literature reviews, we proposed 7 solution proposals, an agile process model with practices. Conclusions: We conclude that VSM is an efficient tool in eliciting improvement potentials in software testing context using qualitative data. In regard to solution proposals, we propose further empirical work to evaluate the solution proposals themselves and tailor to company needs. However, the results obtained through this study can be compared to test processes at other companies in automotive domain. / kasoju.abhinaya@gmail.com

The quality is probably the most significant property of a successful software product. As experience with many software projects has already shown, leaving out testing and quality management from software development process can result in vast and critical customer issues, which usually invoke additional expenses for the software production company. In the course of time, software testing as a discipline has therefore seized an important position among other software development activities. And due to the fact that the software, thanks to rising customer demands and growing competing products portfolio, is getting more complex, the more advanced software testing procedures need to be implemented. Test strategy and iteration planning, implementation of multiple test types into the test suite, test automation, evaluation, administration and maintenance: these activities are commonly required within larger software development projects. Formalized software testing is, nevertheless, being introduced also to projects using agile management techniques which, by management decision or target user industry, are supposed to deliver products of highest quality. This initiates the high focus on the software testing discipline. The purpose of this thesis is to design and implement a new system of software testing within a real project in the CertiCon a.s. company, for which software development is the main business activity. Current issues and gaps for possible improvement, gathered both by project management and the author himself, are analyzed in the first major part of this work and transformed into testing system requirements, which should cover both process changes as well as implementation/development of needed software supporting the process. The second part of thesis is covering the design of the new software testing system, focusing mainly on the area of test management and selection of appropriate software management tool. The final part of the thesis is aimed at implementation of the designed solution within a real organization environment and evaluates its benefits against the previously set requirements.

Test coverage is an important parameter of analyzing how well the product is being tested in any domain within the IT industry. Unit testing is one of the important processes that have gained even more popularity with the rise in Test driven development (TDD) culture.This degree project, conducted at NASDAQ Technology AB, analyzes the existing unit tests in one of the products, and compares various coverage models in terms of quality. Further, the study examines the factors that affect code coverage, presents the best practices for unit testing, and a proven test process used in a real world project.To conclude, recommendations are given to NASDAQ based on the findings of this study and industry standards. / Testtäckning är en viktig parameter för att analysera hur väl en produkt är testad inom alla domäner i IT-industrin. Enhetstestning är en av de viktiga processerna som har ökat sin popularitet med testdriven utveckling. Detta examensarbete, utfört på NASDAQ Technology AB, analyserar de befintliga testen i en av produkterna, och jämför olika kvalitetsmodeller. Vidare undersöker undersökningen de faktorer som påverkar koddekning, presenterar de bästa metoderna för enhetstestning och en beprövad testprocess som används i ett verkligt världsprojekt. Avslutningsvis ges rekommendationer till NASDAQ baserat på resultaten från denna studie och industristandarder.