As cloud-native applications become more distributed, complex, and unpredictable with the adoption of microservices and other new architectural components, traditional monitoring solutions are inadequate in providing end-to-end visibility and proactively identifying deviations from expected behaviour before they become disruptive to services. In response to these challenges, observability-driven development (ODD) is proposed as a new methodology that leverages tools and practices to observe the state and detect the behaviour of systems. Unlike the leading IT giants developing their proprietary tools and platforms, some non-IT companies and smaller organizations still have difficulty adopting observability-driven development. Proprietary development demands extensive resources and manpower, while connecting to third-party platforms may compromise data security. This thesis proposed an end-to-end observability pipeline that is composed of merely open-source components. The pipeline collects and correlates metrics, logs, and traces to facilitate software development and help troubleshoot in production. The pipeline is designed to be adaptive and extensible so that companies can adopt it as the first step towards observability-driven development, and customize it to meet their specific requirements. / Molnbaserade applikationer blir alltmer distribuerade, komplexa och oförutsägbara med införandet av mikrotjänster och andra nya arkitektoniska komponenter. Detta resulterar i att traditionella övervakningslösningar blir alltmer inadekvata. De traditionella lösningarna tillhandahåller inte tillräcklig överskådlighet över dessa applikationer (end-to-end) för proaktiv identifiering av avvikelser från förväntat beteende innan de börjar påverka tjänsterna negativt. Som svar på dessa utmaningar föreslås observerbarhetsdriven utveckling (ODD) som en ny metod som utnyttjar verktyg och praxis för att observera tillståndet och upptäcka systemens beteende. Till skillnad från de ledande IT-jättarna som utvecklar sina egna verktyg och plattformar, har vissa icke-IT-företag och mindre organisationer fortfarande svårt att ta till sig observerbarhetsdriven utveckling. Egenutvecklad mjukvara kräver omfattande resurser och arbetskraft, medan anslutning till tredjepartsplattformar kan äventyra datasäkerheten. Den här avhandlingen bidrar med en end-to-end lösning som enbart baserats på öppen källkod. Pipelinen samlar in data från loggar och korrelerar dessa mätvärden för att underlätta mjukvaruutveckling och hjälpa till att felsöka i produktionen. Pipelinen är designad för att vara anpassningsbar och utvidgningsbar så att företag kan använda den som ett första steg mot observerbarhetsdriven utveckling och anpassa den för att möta deras specifika krav.

As infrastructures and software grow in complexity the need to keep track of things becomes important. It is the job of log aggregation solutions to condense log data into a form that is easier to search, visualize, and analyze. There are many log aggregation solutions out there today with various pros and cons to fit the various types of data and architectures. This makes the choice of selecting a log aggregation solution an important one. This thesis analyzes two full-stack log aggregation solutions, Elastic stack and SigNoz, with the goal of evaluating how the ingestion and storage components of the two stacks perform with smaller and larger amounts of data. The evaluation of these solutions was done by ingesting log files of varying sizes into them while tracking their performance. These performance metrics were then analyzed to find similarities and differences. The thesis found that SigNoz featured a higher CPU usage on average, faster processing times, and lower memory usage. Elastic stack was found to do more processing and indexing on the data, requiring more memory and storage space to allow for more detailed searchability of the ingested data. This also meant that there was a larger storage space requirement for Elastic stack than SigNoz to store the ingested logs. The hope of this thesis is that these findings can be used to provide insight into the area and aid those choosing between the two solutions in making a more informed decision.