The volumes of data which Big Data applications have to process are constantly increasing. This requires for the development of highly scalable systems. Microservices is considered as one of the solutions to deal with the scalability problem. However, the literature on practices for building scalable data-intensive systems is still lacking. This thesis aims to investigate and present the benefits and drawbacks of using microservices architecture in big data systems. Moreover, it presents other practices used to increase scalability. It includes containerization, shared-nothing architecture, data sharding, load balancing, clustering, and stateless design. Finally, an experiment comparing the performance of a monolithic application and a microservices-based application was performed. The results show that with increasing amount of load microservices perform better than the monolith. However, to cope with the constantly increasing amount of data, additional techniques should be used together with microservices.

The evolution of software applications from single desktops to sophisticated cloud-based systems is challenging. In particular, applications that involve massive data sets, such as geospatial applications and data science applications are challenging for domain experts who are suddenly constructing these sophisticated code bases. Relatively new software practices, such as Microservice infrastructure and DevOps, give us an opportunity to improve development, maintenance and efficiency for the entire software lifecycle. Microservices and DevOps have become adopted by software developers in the past few years, as they have relieved many of the burdens associated with software evolution. Microservices is an architectural style that structures an application as a collection of services. DevOps is a set of practices that automates the processes between software development and IT teams, in order to build, test, and release software faster and increase reliability. Combined with lightweight virtualization solutions, such as containers, this technology will not only improve response rates in cloud-based solutions but also drastically improve the efficiency of software development. This thesis studies two applications that apply Microservices and DevOps within a domain-specific application. The advantages and disadvantages of Microservices architecture and DevOps are evaluated through the design and development on two different platforms---a batch-based cloud system, and a general purpose cloud environment. / Graduate

Sowohl das Architekturmuster der Microservices als auch das „Internet of Things“ (IoT) folgen den Prinzipien verteilter Systeme. Eine Anwendung von Microservices im IoT ist daher naheliegend, allerdings stellt sich hierbei die Frage,welche Integrationsform dafürambesten geeignet ist. Aus diesem Grund besteht das Ziel dieser Arbeit darin, verschiedene Kommunikationskonzepte von Microservices auf ihre Eignung für den Bereich des IoTs zu überprüfen. In der Arbeit werden dafür verschiedene Kommunikationskonzepte vorgestellt und anhand von diversen Kriterien miteinander auf qualitative Weise verglichen. Die Kriterien setzen sich aus allgemeinen Herausforderungen im Umgang mit verteilten Systemen sowie aus kennzeichnenden Eigenschaften für IoT-Systeme, wie bspw. Skalierbarkeit, Erweiterbarkeit und Ressourcennutzung zusammen. Ergänzend dazu fließen weitere Aspekte der Softwareentwicklung in den Vergleich ein. Anhand der Vergleichsergebnisse wird schließlich eine Bewertung durchgeführt, welche als Grundlage für die Entwicklung von Microservices im Bereich IoT dienen kann. Zum Ende wird das Resultat durch die Konzeption einer eigenen Anwendung erprobt. Der Vergleich und die anschließende Bewertung zeigen, dass sich asynchrone Microservices aufgrund ihrer Eigenschaften am besten für den Einsatz im IoT eignen. Dabei wurde insbesondere auf die die Kommunikationsvariante des Messagings mittels des Protokolls MQTT eingegangen. Aufbauend auf den Ergebnissen dieser Arbeit könnten Forschungen im Bereich moderner IoT-Plattformen angestellt werden. Weiterhin würde sich die Erweiterung bestehender Microservice-Anwendungen durch zusätzliche Services für Datensammlung und intelligente Auswertung dieser als Forschungsthema anbieten.:1 Einleitung und Intention 1.1 MakroSolutions GmbH 1.2 Stand der Wissenschaft und Technik 1.3 Ziel der Arbeit 1.4 Aufbau der Arbeit 2 Definitionen und theoretische Grundlagen 2.1 Verteilte Systeme 2.1.1 Services 2.2 Internet of Things 2.2.1 Industrial Internet of Things 2.3 Microservices 2.3.1 Definition und Eigenschaften 2.3.2 Vorteile im Umgang mit Microservices 2.3.3 Herausforderungen im Umgang mit Microservices 2.3.4 Anwendungsgebiete 2.3.5 Abgrenzung zu monolithischer Architektur 2.3.6 Abgrenzung zu Service-Orientierter Architektur (SOA) 3 Kommunikation der Microservices 3.1 Mikro- und Makroarchitektur 3.2 Integrationsformen 3.3 Gemeinsame Datenbanknutzung und Datenreplikation 3.4 Synchrone Microservices 3.5 Asynchrone Microservices 4 Vergleich und Bewertung 4.1 Wahl der Vergleichskriterien 4.1.1 Fallacies of distributed computing 4.1.2 CAP-Theorem 4.1.3 Internet of Things 4.1.4 Weitere qualitative Kriterien 4.2 Vergleich der Kommunikationskonzepte 4.2.1 Fehlertoleranz und Resilienz 4.2.2 Performance und Ressourcennutzung 4.2.3 Datenkonsistenz und Verfügbarkeit 4.2.4 Skalierbarkeit und Erweiterbarkeit 4.2.5 Kopplung 4.2.6 Komplexität 4.2.7 Tests 4.2.8 Sicherheit 4.3 Bewertung in Bezug auf das Internet der Dinge (IoT) 5 Konzeption der Software 5.1 Ausgangspunkt 5.2 Anforderungen 5.3 Vorüberlegungen 5.4 Konzeption 6 Auswertung und Ausblick 6.1 Zusammenfassung und Interpretation der Ergebnisse 6.2 Beschränkungen und Ausblick für weitere Forschung

Microservices is a new and trendy architecture in software development and amongst its features isthe ability to open up for teams to more freely choose the tech stack and programming language thatbest fits their needs. This feature, termed language diversity for the purposes of this study, is describedin literature as a key to optimization and flexibility but is also ripe with concerns of complexity. Inthis study the author seeks to explore what language diversity could mean for the Swedish IT giantSkatteverket, the Swedish Tax Agency, from an organizational standpoint and to understand how itcan align with Skatteverket’s goals. To do so the author has performed a case study consisting of aquestionnaire sent out to tech workers in four different sections, and interviews with two keyindividuals in the organization. The results show that a significant number of respondents(p-value=0.003), consider that language diversity would improve Skatteverket’s attractiveness as anIT employer, the effects of which could be a competitive edge on the job market. It was also shownthat most (p-value<0.001) believed it would lead to at least some problems for the organization.Amongst the respondents, more experienced tech workers showed a tendency (p-value=0.06) tobelieve it would have less of a positive impact on Skatteverket’s image and were more likely tobelieve that the problems it would lead to would be greater. Overall, the study showed that languagediversity could offer other rewards than those proclaimed in literature, and that the modernizationfactor of it could play a big role for Skatteverket.

The development of applications that use microservice architecture patterns is increasingrapidly, and this architecture is proven to be successful in many different areas, especiallyin cloud computing. The reason microservices and cloud computing are a great matchis the possibility of scaling and deploying individual services, which positively affects thecost and utilization. This architecture pattern includes some challenges for the devel-opers, such as placement optimisation and knowledge about how the applications aredeployed.This study intends to clarify how the applications in a multi-cluster environment are de-ployed. A service model was created, describing how applications built with microservicearchitecture patterns communicate to each other and which microservices the applicationcontains. More specifically, this can be seen as the first step of placement optimisationthat will be developed in the future. The test cases used to produce the service modelshave various characteristics, such as control planes, where applications were deployed,and numbers of replicas. These kinds of characteristics were varied so that the servicemodels could be relied on and such that the model created works independent of howthe deployment model is created. The created service models show that the applicationtopology is not restricted for the reverse engineering method to work. Independent ofthe number of control planes or replicas, this method worked. Furthermore, the servicemodels created for each test case gave the correct outcome for each application regardingmicroservices and the connections between each microservice.

Modern technology is rapidly and efficiently expanding, and by looking at the largest companies by market cap, one will find enterprises like Apple, Microsoft, Alphabet, and Meta. Given the complexity of modern software architecture, there arises a necessity for a software architecture that is both scalable and adaptable. This demand has given rise to the adoption of microservices as a preferred approach for building complex and distributed applications. However, managing microservices effectively is a difficult task. Therefore, Google created an orchestration tool called Kubernetes (K8). The primary purpose of this thesis is to extend the information about the characteristics of a K8 cluster by monitoring its performance in various scenarios. There is substantial documentation about how K8 works and why it is used. However, insufficient information exists regarding the performance of K8 in different scenarios.  Extensive testing has transpired to extend the information about the characteristics of K8. Parameters such as the number of Pods, containers, mounts, and CPU cores have been thoroughly tested. Additionally, parameters such as container load, CPU limitation, container distribution, and memory allocation have been examined. The core result will include startup time and CPU utilization. The startup time is essential in a K8 cluster because of its ephemeral characteristics, meaning each Pod is short-lived and will restart frequently. CPU utilization testing is essential to analyze how K8 allocate resources and perform with different amounts of resources.  The results show that the most significant parameters regarding startup time are, as one might expect, the number of containers, CPUs, Pods, and the load in each Pod. However, the complexity of the Pod, for instance, the number of mount points, has significantly less effect on the cluster than expected. Regarding CPU utilization, the results show that K8 does lower CPU usage if possible, resulting in equal CPU usage even with different numbers of CPUs. The most significant CPU usage parameter is the load of the application. Finally, this thesis work has filled some gaps in how a K8 cluster behaves during various circumstances, for instance, varying numbers of Pods, containers, or CPUs. One must consider several aspects while designing a K8 cluster. However, all aspects have not been considered, and the usage of K8 increases daily. Therefore, this thesis will hopefully be one of many reports investigating how a K8 cluster behaves and what to consider when building a cluster.

The microservice architecture enables organizationsto shorten development cycles and deliver cloud-native applicationsrapidly. However, it also brings security concerns thatneed to be addressed by developers. Therefore, security testingin microservices becomes even more critical. Recent researchpapers indicate that security testing of microservices is oftenneglected for reasons such as lack of time, lack of experience inthe security domain, and absence of automated test environments.Even though several security scanning tools exist to detectcontainer, containerized workload management (Kubernetes),and network issues, none individually is sufficient to cover allsecurity problems in microservices. Using multiple scanning toolsincreases the complexity of analyzing findings and mitigatingsecurity vulnerabilities. This paper presents a fully automatedtest tool suite that can help developers address security issuesin microservices and resolve them. It targets to reduce timeand effort in security activities by encapsulating open-sourcescanning tools into one suite and providing improved feedback.The developed security scanning suite is named Pomegranate.To develop Pomegranate, we employed Design Science andconducted our investigation in Ericsson. We have evaluated ourtool using a static approach. The evaluation results indicate thatthe Pomegranate could be helpful to developers by providingsimplified and classified outputs for security vulnerabilities inmicroservices. More than half of the practitioners who give usfeedback found Pomegranate helpful in detecting and mitigatingsecurity problems in microservices. We conclude that a fullyautomated test tool suite can help developers to address mostsecurity issues in microservices. Based on the findings in thispaper, the direction for future work is to conduct a dynamicvalidation of Pomegranate in a live project.

Au vu des dernières avancées dans le domaine de l’ingénierie web, ainsi qu’avec la baisse de coût du cloud computing, les architectures orientées services sont rapidement devenues la solution prépondérante pour fournir des services à valeur ajoutée aux clients. Suite à cette tendance, la composition de services tiers est devenue un paradigme de référence pour le développement d’applications robustes et riches, ou encore pour l’automatisation de processus métiers. Avec la disponibilité de centaines de milliers de services et APIs web, la réalisation de telles intégrations devient lourde et fastidieuse quand effectuée manuellement. Par ailleurs, chaque client peut exiger des besoins et politiques d’intégration différentes, ce qui complexifie davantage la tâche. De plus, fournir une telle solution qui soit à la fois robuste et scalable est une tâche non-triviale. Il est donc primordial d’étudier comment coordonner de manière efficace les interactions entre les services web existants. Ainsi, cette thèse vise à étudier les problématiques liées à la composition de services web dans le contexte des pratiques de développement web modernes. Nous présentons un cadre architectural permettant la spécification de compositions de services web grâce à une approche orientée langage, et montrons comment supporter leur exécution de manière scalable grâce à MEDLEY, une plateforme légère et orientée événements. / In light of the recent advances in the field of web engineering, along with the decrease of cost of cloud computing, service-oriented architectures rapidly became the leading solution in providing valuable services to clients. Following this trend, the composition of third-party services has become a successful paradigm for the development of robust and rich distributed applications, as well as automating business processes. With the availability of hundreds of thousands of web services and APIs, such integrations become cumbersome and tedious when performed manually. Furthermore, different clients may require different integration requirements and policies, which further complexifies the task. Moreover, providing such a solution that is both robust and scalable is a non-trivial task. Therefore, it becomes crucial to investigate how to efficiently coordinate the interactions between existing web services. As such, this thesis aims at investigating the underlying challenges in web service composition in the context of modern web development practices. We present an architectural framework to support the specification of web service compositions using a language-based approach, and show how we support their execution in a scalable manner using MEDLEY, a lightweight, event-driven platform.

Multi-core processors require a program to be decomposable into independent parts that can execute in parallel in order to scale performance with the number of cores. But parallel programming is hard especially when the program requires state, which many system programs use for optimization, such as for example a cache to reduce disk I/O. Most prevalent parallel programming models do not support a notion of state and require the programmer to synchronize state access manually, i.e., outside the realms of an associated optimizing compiler. This prevents the compiler to introduce parallelism automatically and requires the programmer to optimize the program manually. In this dissertation, we propose a programming language/compiler co-design to provide a new programming model for implicit parallel programming with state and a compiler that can optimize the program for a parallel execution. We define the notion of a stateful function along with their composition and control structures. An example implementation of a highly scalable server shows that stateful functions smoothly integrate into existing programming language concepts, such as object-oriented programming and programming with structs. Our programming model is also highly practical and allows to gradually adapt existing code bases. As a case study, we implemented a new data processing core for the Hadoop Map/Reduce system to overcome existing performance bottlenecks. Our lambda-calculus-based compiler automatically extracts parallelism without changing the program's semantics. We added further domain-specific semantic-preserving transformations that reduce I/O calls for microservice programs. The runtime format of a program is a dataflow graph that can be executed in parallel, performs concurrent I/O and allows for non-blocking live updates.

No description available.