This thesis focuses on test management tools and automated testing. The project covers analysis of existing open source tools and proposes its own BDD orientated test management tool in the form of a web service. The project aims to specify and design this application based on the process of Behaviour driven development. The resulting application TestBuDDy allows for test library management. Changes on the test library are projected onto a remote repository of software under test (SUT) and triggers a test run (the test library is being run against SUT by the BDD framework). TestBuDDy is able to save the test run results, parse them into a report and generate and group found issues. The application also allows requirement management and user management. The application is integrated with the CI/CD tool Gitlab CI, the BDD framework JBehave and the issue tracker JIRA. The application is designed to help testers during their work and also to be expandable within the open source community.

Tato práce představuje optimalizační techniky zaměřené na proces sběru výkonnostních dat v rámci výkonnostní analýzy a profilování programů v nástroji Perun.   Rozšíření architektury a implementace těchto nových optimalizačních technik v nástroji Perun (a převážně pak v jeho modulu Tracer) zlepšuje jeho škálovatelnost a umožňuje tak provádět výkonnostní analýzu i nad rozsáhlými projekty.   Zaměřujeme se především na zvýšení přesnosti sběru dat, redukci množství instrumentovaných bodů programu, omezení časové režie procesu sběru dat a výkonnostního profilování, snížení objemu sbíraných dat a velikosti výsledného výkonnostního profilu.   Optimalizace je dosažena pomocí aplikace statistických metod, množství technik statické a dynamické analýzy (případně jejich kombinací) a využitím pokročilých možností a schopností nástrojů SystemTap a eBPF.   Na základě vyhodnocení provedeného na dvou vybraných projektech a množství experimentů můžeme konstatovat, že se nám úspěšně podařilo dosáhnout značné optimalizace u téměř všech sledovaných metrik a kritérií.

S vývojem informací, komunikací a technologií se moderní průmyslové řídicí systémy (ICS) potýkají čím dál více s otázkami automatického testováni pro zabezpečení stability a bezpečnosti systému. Z tohoto důvodu se testování stalo jednou z nejdůležitějších částí životního cyklu všech softwarů. V této diplomové práci budu zvažovat možnost využití stávajících zkušebních metod a nástrojů pro získání dostatečné jakosti a bezpečnosti softwaru v kontinuálních integračních systémech.

Multirepo model přístupu ke správě a verzování zdrojového kódu, jež zahrnuje použití mnoha oddělených repozitářů verzovacích systémů, je poslední dobou často zmiňován v odborné literatuře. Jednou z jeho nevýhod je množství zdlouhavých, nezajímavých a repetitivních úkonů, které je nutno provádět při hromadných operacích tvořících transakce napříč těmito repozitáři. Multirepo repozitáře navíc umožňují využití široké škály technologií, což jen umocňuje riziko lidské chyby, ke které při ručně prováděných hromadných operacích může dojít. V rámci této práce je navrženo, implementováno a otestováno řešení pro automatizaci operací prováděných napříč množstvím repozitářů uspořádaných v multirepo modelu, což s nimi uživatelům zlepšuje zkušenost.

The notion of change is one of the more pronounced challenges API developers face, since a change to published APIs means a change for its’ consumers. API Versioning is a commonly used strategy to mitigate the risks of changing APIs, however, the strategy introduces a possibility of having practically innumerable number of API versions published at once, all possibly used by different consumers, creating considerable overhead for API developers to manage. API Versioning and API version management are well-researched disciplines. Nevertheless, Yasmin et al. (2020) have identified a research gap in the support for API consumers to made aware of API deprecations in a proactive and automatic fashion. API Versioning in the context of microservices– which the literature seldom addresses– may serve as an interesting solution space for answering the call of Yasmin et al. (2020), due to the architecture’s strong ties to continuous integration– a practice concerned with automatically and proactively ensuring software quality. Concisely, this thesis is set out to answer Yasmin et al.’s (2020) call for research by utilizing the underexplored contextualization of microservices and continuous integration. Using a Design Science Research strategy, this thesis proposes a model detailing how deprecated RESTful web API version usage can be automatically and proactively detected in microservices using continuous integration. The model is subsequently proven to be instantiable via a proof of concept, designed for and situated at a national Swedish bank. Through a series of in-vitro simulations at the bank, the proof of concept is demonstrated to successfully being able to automatically and proactively detect deprecated RESTful web API version usage. An evaluation of the solution shows that the proof of concept– and by extent the model– fulfills the demarcated research objectives of automaticity, proactivity, generality, and autonomy. Thus, the proposed solution is determined to be both novel and useful, successfully answering the call of Yasmin et al. (2020) in the context of microservices and continuous integration.

With today’s focus on Continuous Integration, test cases are used to ensure the software’s reliability when integrating and developing code. Test cases that behave in an undeterministic manner are known as flaky tests, which threatens the software’s reliability. Because of flaky test’s undeterministic nature, they can be troublesome to detect and correct. This is causing companies to spend great amount of resources on flaky tests since they can reduce the quality of their products and services. The aim of this thesis was to develop a usable tool that can automatically detect flakiness in the Randomness category. This was done by initially locating and rerunning flaky tests found in public Git repositories. By scanning the resulting pytest logs from the tests that manifested flaky behaviour, noting indicators of how flakiness manifests in the Randomness category. From these findings we determined tracing to be a viable option of detecting Randomness as a cause of flakiness. The findings were implemented into our proposed tool FlakyReporter, which reruns flaky tests to determine if they pertain to the Randomness category. Our FlakyReporter tool was found to accurately categorise flaky tests into the Randomness category when tested against 25 different flaky tests. This indicates the viability of utilizing tracing as a method of categorizing flakiness.

The Eiffel Protocol provides traceability downstream and upstream of all activities that transpire inside the CI/CD pipeline. The traceability achieved by the Eiffel Protocol comes with great benefits even though it does not cover all development activities as it pertains only to the CI/CD pipeline. Our research aims to explore the idea of extending the Eiffel Protocol to cover all activities and discuss what benefits could be seen, especially in the scope of reducing the number of integration failures. A literature study was first carried out to find the root causes of these failures. After the literature study, we conducted a focus group session to gather data about the potential benefits and problems of an extension, what analyses could be drawn, and how it can affect integration errors. Our results show that an extension is beneficial as analyses that can be made with the generated data can tackle some of the biggest issues found in software development teams, especially in larger organizations. The complexity, cost involved and the time needed to see a return on investment does however weigh it down. Thus, while it is beneficial it is not enough for organizations to consider it as a priority to integrate with their environments when thinking of the costs involved to do so. Further implementation solutions need to be researched before it shows its worth.

Denna rapport behandlar skapandet av en webbapplikation för filtreraring och visualisering av data i Pythonramverket Dash. Rapporten ger en översikt av gruppens arbetsmetodik och projektets utveckling. Webbapplikationen utvecklades inom kursen TDDD96 Kandidatprojekt i programvaruutveckling av nio studenter som studerar civilingenjör Datateknik och civilingenjör Mjukvaruteknik. Detta uppdrag fick projektgruppen av företaget Ericsson. Projektets resultat blev en fungerade webbapplikation efterfrågad funktionalitet. Resultatet och arbetsmetodiken, bland annat testdriven utveckling, diskuteras i rapporten med fokus på hur utvecklingsprocessen förbättrades. Rapportens slutsatser är att ramverket Dash lämpar sig för webbutveckling i ett mindre projekt, särskilt för datavisualisering, och att produkten skapar värde för kunden.

An application build describes compiling and linking the source code of a developed application to libraries and executables. A Continuous Integration (CI) build executes such a build after the source code has been changed and tries to integrate the changes into the existing application. Such CI builds are executed automatically and include automated software tests, which give the developer the assurance that the changes are technically correct. When the time between the discovery of a test failure and the notification to the developer about it is too long, the development process will be impacted negatively and the beneficial effects of CI decrease. Even though several companies already have CI systems that display all events of a single CI build on a terminal during runtime, bigger applications often involve several CI builds in a single CI pipeline to integrate code changes. Observing the events of these CI builds during runtime might require concurrent monitoring of several different terminals. This thesis overcomes this issue by developing a Proof of Concept (PoC) which streams the test failures of a whole CI pipeline in real-time to the developer. To show the feasibility of real-time failure event streaming of CI builds, the PoC is implemented within Spotify’s CI for clientfacing applications. The issues highlighted by this initial PoC will help to refine the whole CI practice. Furthermore, the faster feedback cycles realised by this PoC will lead to a productivity, efficiency and happiness increase for the involved developers and, eventually, higher quality of the developed software. / Ett applikationsbygge beskriver kompilering och länkning av källkod för en utvecklad applikation till bibliotek och körbara filer. Ett Kontinuerlig Integrerings (CI)-bygge kör en sådan bygge efter att källkoden har ändrats och försöker integrera ändringarna i den befintliga applikationen. Sådana CIbyggen exekveras automatiskt och inkluderar automatiserade mjukvarutester, som ger utvecklaren en försäkran om att ändringarna är tekniskt korrekta. När tiden mellan upptäckten av ett testfel och meddelandet till utvecklaren om det är för lång kommer utvecklingsprocessen att påverkas negativt och de fördelaktiga effekterna av CI minskar. Även om flera företag redan har CIsystem som visar alla händelser av ett enskilt CI-bygge i en terminal under körning, involverar större applikationer ofta flera CI-byggen i en och samma CI-pipeline för att integrera kodändringar. Att observera händelserna i dessa CI-byggen under körning kan kräva jämlöpande övervakning av flera olika terminaler. Den här avhandlingen övervinner detta problem genom att utveckla en PoC som strömmar testfelen för en hel CI-pipeline i realtid till utvecklaren. För att visa genomförbarheten av strömning av felhändelser i realtid av CIbyggnader implementeras PoC i Spotifys CI för klientvända applikationer. De problem som lyfts fram av denna första PoC kommer att bidra till att förfina hela CI-praxisen. Dessutom kommer de snabbare återkopplingscyklerna som realiseras av denna PoCatt leda till ökad produktivitet, effektivitet och glädje för de inblandade utvecklarna och, så småningom, högre kvalitet på den utvecklade mjukvaran.

Continuous Integration (CI) techniques are widely adopted in web and application development but have received limited attention in the embedded software domain. This thesis investigates the application of CI techniques in embedded software development through a case study at Cobolt AB, a company specializing in optoelectronics. The study aims to identify suitable CI techniques, assess implementation efforts, and evaluate the impact of CI adoption in this domain. A CI service is implemented using Jenkins as the automation server, following an iterative development and deployment process. The service incorporates multi-target compilation, automated unit testing, test reporting, visual CI feedback, and trunk-based development. These techniques prove effective for embedded software with a modular firmware architecture. However, automated system testing encounters limitations due to the need for manual interaction with hardware targets. Challenges encountered during implementation, such as learning CI tools, managing build tool dependencies, and addressing manual input requirements for system testing, are overcome through iterative implementation, distributed build architecture, and selective test automation. Developers’ resistance to CI adoption diminishes as they experience the positive impact of the CI service. CI adoption in the embedded domain brings benefits such as fast bug detection, increased developer motivation, higher confidence in code quality, and encouragement for standardization n. No noticeable negative impacts are observed. Future research should focus on integrating hardware-in-the-loop simulation systems for comprehensive automated system testing, exploring validation on multiple hardware targets, and studying the vertical scaling capabilities of distributed build architectures with Jenkins. / Kontinuerlig integration (CI) tekniker används i stor utsträckning inom webboch applikationsutveckling, men har fått begränsad uppmärksamhet inom inbyggd programvarudomän. Denna avhandling undersöker tillämpningen av CI-tekniker inom inbyggd programvaruutveckling genom en fallstudie vid Cobolt AB, ett företag specialiserat på optoelektronik. Studien syftar till att identifiera lämpliga CI-tekniker, bedöma implementeringsinsatser och utvärdera effekten av CI-användning inom detta område. En CI-tjänst implementeras med Jenkins som automatiseringsserver, efter en iterativ utvecklings- och distribueringsprocess. Tjänsten inkluderar kompilering för flera målenheter, automatiserad enhetstestning, testrapportering, visuell CI-återkoppling och utveckling baserad på huvudgrenen. Dessa tekniker visar sig vara effektiva för inbyggd programvara med en modulär firmware-arkitektur. Dock begränsas automatiserad systemtestning av behovet av manuell interaktion med hårdvarumål. Utmaningar som uppstår under implementeringen, såsom att lära sig CIverktyg, hantera byggverktygsberoenden och hantera manuella indatakrav för systemtestning, övervinner genom iterativ implementering, distribuerade byggarkitekturer och selektiv testautomatisering. Utvecklarnas motstånd mot CI-användning minskar när de upplever de positiva effekterna av CI-tjänsten. CI-användning inom inbyggd programvaruutveckling medför fördelar som snabb upptäckt av fel, ökad utvecklar motivation, högre förtroende för kodkvalitet och främjande av standardisering. Inga märkbara negativa effekter observeras. Framtida forskning bör fokusera på att integrera hårdvaru-i-loop simulering för omfattande automatiserad systemtestning, utforska validering på flera hårdvarumål och studera de vertikala skalningsmöjligheterna hos distribuerade byggarkitekturer med Jenkins.