[pt] Contexto: As operações de aprendizagem automática (MLOps) surgiram como um conjunto de práticas que combina desenvolvimento, testes e operações para implementar e manter aplicações de aprendizagem automática. Objetivo: Nesta dissertação, iremos avaliar os benefícios e limitações da utilização dos princípios de MLOps no contexto de modelos supervisionados online, que são amplamente utilizados em aplicações como a previsão meteorológica, tendências de mercado e identificação de riscos. Método: Aplicámos dois métodos de investigação para avaliar os benefícios dos MLOps para aplicações de aprendizagem automática online supervisionada: (i) desenvolvimento de um projeto prático de aprendizagem automática supervisionada para aprofundar a compreensão do problema e das possibilidades de utilização dos princípios MLOps; e (ii) duas discussões de grupo de foco sobre os benefícios e limitações da utilização dos princípios MLOps com seis programadores de aprendizagem automática experientes. Resultados: O projeto prático implementou uma aplicação de aprendizagem automática de regressão supervisionada utilizando KNN. A aplicação utiliza informações sobre as rotas das linhas de autocarros públicos do Rio de Janeiro e calcula a duração da viagem de autocarro com base na hora de partida do dia e no sentido da viagem. Devido ao âmbito da primeira versão e ao facto de não ter sido implementada em produção, não sentimos a necessidade de utilizar os princípios MLOps que esperávamos inicialmente. De facto, identificámos a necessidade de apenas um princípio, o princípio do controlo de versões, para alinhar as versões do código e dos dados. O grupo de discussão revelou que os programadores de aprendizagem automática acreditam que os benefícios da utilização dos princípios MLOps são muitos, mas que não se aplicam a todos os projectos em que trabalham. A discussão revelou que a maioria dos benefícios está relacionada com a prevenção de passos manuais propensos a erros, permitindo restaurar a aplicação para um estado anterior e ter um pipeline robusto de implementação automatizada contínua. Conclusões: É importante equilibrar as compensações do investimento de tempo e esforço na implementação dos princípios de MLOps, considerando o âmbito e as necessidades do projeto. De acordo com os especialistas, esse investimento tende a compensar para aplicativos maiores com implantação contínua que exigem processos automatizados bem preparados. Por outro lado, para versões iniciais de aplicações de aprendizagem automática, o esforço despendido na implementação dos princípios pode alargar o âmbito do projeto e aumentar o tempo de execução. / [en] Context: Machine Learning Operations (MLOps) has emerged as a set of practices that combines development, testing, and operations to deploy and maintain machine learning applications. Objective: In this dissertation, we will assess the benefits and limitations of the use of MLOps principles in the context of online supervised models, which are widely used in applications such as weather forecasting, market trends, and risk identification. Method: We applied two research methods to assess the benefits of MLOps for supervised online machine learning applications: (i) developing a practical supervised machine learning project to deepen the understanding of the problem and of the MLOps principles usage possibilities; and (ii) two focus group discussions on the benefits and limitations of using the MLOps principles with six experienced machine learning developers. Results: The practical project implemented a supervised regression machine learning application using KNN. The application uses information on Rio de Janeiro s public bus line routes and calculates the bus trip duration based on the trip departure time of the day and trip direction. Due to the scope of the first version and given that it was not deployed into production, we didn t feel the need to use the MLOps principles we were expecting at first. Indeed, we identified the need for only one principle, the versioning principle, to align versions of the code and the data. The focus group revealed that machine learning developers believe that the benefits of using MLOps principles are many but that they do not apply to all the projects they worked on. The discussion brought up that most of the benefits are related to avoiding error-prone manual steps, enabling it to restore the application to a previous state, and having a robust continuous automated deployment pipeline. Conclusions: It is important to balance the trade-offs of investing time and effort in implementing the MLOps principles considering the scope and needs of the project. According to the experts, this investment tends to pay off for larger applications with continuous deployment that require well-prepared automated processes. On the other hand, for initial versions of machine learning applications, the effort taken into implementing the principles might enlarge the scope of the project and increase the time needed to deploy a first version to production.

As ML (Machine Learning) and deep neural networks get more used in many systems,the need to understand and test such systems becomes more actual. When designing a newsystem that contains ML models, the safety of this system becomes inevitably important.This rises the need to discuss a strategy to deal with the potential problems and weak-nesses in such a system. This thesis provides findings from literature and illustrates thepotential strategies in the area of image recognition in a comprehensive way. Lastly, theresult presented in this thesis shows that using an ML component in a complex softwaresystem with high safety requirements requires adopting software methodologies, such asMLOps (Machine learning operations) to monitor such a system and give suggestions tohow to test and verify an ML model integrated into a larger software system.

Steel production companies must utilize new technologies and innovations to stay ahead of a highly competitive market. Recently, there has been a focus on Industry 4.0, which involves the digitalization of production to integrate with newer technologies such as cloud solutions and the Internet of Things (IoT). This results in a greater understanding of processes and data gathered in production, laying the foundation for potential machine learning (ML) implementations. ML models can improve process quality, reduce energy usage to produce more environmentally friendly products, and gain competitive advantages. Implementing several ML models in production can be difficult, as it involves dealing with different datasets and algorithms, moving models into production, and post-deployment maintenance. If these tasks are kept manually, the workload quickly becomes too large to handle effectively. This is why machine learning operations (MLOps) has recently been a popular topic. Automating parts of the ML workflow enables these systems to scale effectively as the number of models increases. This thesis aims to investigate how implementing MLOps practices can help an organization increase its use of ML systems. To do this, an MLOps framework is implemented using Microsoft Azure services together with a dataset from the stakeholder Uddeholm AB. The resulting workflow consists of automated pipelines for data pre-processing, training, and deployment of an ML model, contributing to establishing a scalable ML framework. Automating the majority of the workflow greatly eases the workload for managing the lifecycle of ML models.

MLOps and Kubernetes are two major components of the modern-day information technology landscape, and their impact on the field is likely to grow even stronger in the near future. As a multitude of tools have been developed for the purpose of facilitating effortless creation of cloud native MLOps solutions, many of them have been designed, to varying degrees, to integrate with the Kubernetes system. While numerous evaluations have been conducted on these tools from a general MLOps perspective, this thesis aims to evaluate their qualities specifically within a Kubernetes context, with the focus being on their integration into this ecosystem. The evaluation is conducted in two steps: an MLOps market overview study, as well as an in-depth MLOps tool evaluation. The former represents a macroscopic overview of currently available MLOps tooling, whereas the latter delves into the practical aspects of deploying three Kubernetes native, open source MLOps platforms on cloud-based Kubernetes clusters. The platforms are Kubeflow, Pachyderm, and Polyaxon, and these are evaluated in terms of functionality, usability, vitality, and performance.

Объект исследования: является процесс обучения MLOps. Цель работы: состоит в создании учебно-практического комплекса для дисциплины «Автоматизация машинного обучения» для обучения студентов основным инструментам и технологиям автоматизации машинного обучения с возможностью дальнейшего применения полученных знаний в профессиональной деятельности. Для достижения поставленной цели необходимо решить следующие задачи. Методы исследования включают в себя: Анализ методических документов, сравнение используемых технологий, систематизацию и обобщение данных о существующих онлайн-курсах по дисциплине «Автоматизация машинного обучения»; Анализ современных программных инструментов, позволяющих выполнить и ускорить процесс автоматизации машинного обучения. / Object of study: is the MLOps learning process. The purpose of the work: is to create an educational and practical complex for the discipline “Automation of Machine Learning” to teach students the basic tools and technologies of machine learning automation with the possibility of further application of the acquired knowledge in professional activities. To achieve this goal, it is necessary to solve the following tasks. Research methods include: Analysis of methodological documents, comparison of the technologies used, systematization and synthesis of data on existing online courses in the discipline “Automation of Machine Learning”; Analysis of modern software tools that allow you to perform and speed up the process of automating machine learning.

Many developed machine learning models are not used in production applications as several challenges must be solved to develop and deploy ML models. Manual reimplementation and heterogeneous environments increase the effort required to develop an ML model or improve an existing one, considerably slowing down the overall process. Furthermore, it is required that a model is constantly monitored to ensure high-quality predictions and avoid possible drifts or biases. MLOps processes solve these challenges and streamline the development and deployment process by covering the whole life cycle of ML models. Even if the research area of MLOps, which applies DevOps principles to ML models, is relatively new, several researchers have already developed abstract MLOps process models. Research for cases with multiple collaboration partners is rare. This research project aims to develop an MLOps process for cases involving multiple collaboration partners. Hence, a case study is conducted with the cooperation of Aimo and LNU as a single case. Aimo requires ML models for their application and collaborates with LNU regarding this demand. LNU develops ML models based on the provided data, which Aimo integrates into their application afterward. This case is analyzed in-depth to identify challenges and the current process. These results are required to elaborate a suitable MLOps process for the case, which also considers the handover of artifacts between the collaboration partners. This process is derived from the already existing general MLOps process models. It is also instantiated to generate a benefit for the case and evaluate the feasibility of the MLOps process. Required components are identified, and existing MLOps tools are collected and compared, leading to the selection of suitable tools for the case. A project template is implemented and applied to an ML model project of the case to show the feasibility. As a result, this research project provides a concrete MLOps process. Besides that, several artifacts were elaborated, such as a project template for ML models in which the selected toolset is applied. These results mainly fit the analyzed case. Nevertheless, several findings are also generalizable such as the identified challenges. The compared alternatives and the generally applied method to elaborate an MLOps process can also be applied to other settings. This is also the case for several artifacts of this project, such as the tool comparison table and the applied process to select suitable tools. This case study shows that it is possible to set up MLOps processes with a high maturity level in situations where multiple cooperation partners are involved and artifacts need to be transferred among them.

In the last 5+ years, researchers and the industry have been working hard to adopt MLOps (Machine Learning Operations) to maximize production. The current literature on MLOps is still mostly disconnected and sporadic (Testi et al., 2022). This study conducts mixed-method research, including a literature review, survey questionnaires, and expert interviews to address this gap. The researcher provides an aggregated overview of the necessary principles, components, roles, and the associated architecture and workflows resulting from these investigations. Furthermore, this research furnishes a definition of MLOps and addresses open challenges in the field. Finally, this work proposes a MLOps pipeline to implement product recommendations on the e-commerce platform to guide ML researchers and practitioners who want to automate and operate their ML products.

The maturing capabilities of Artificial Intelligence (AI) and Machine Learning (ML) have resulted in increased attention in research and development on adopting AI and ML in 5G and future networks. With the increased maturity, the usage of AI/ML models in production is becoming more widespread, and maintaining these systems is more complex and likely to incur technical debt when compared to standard software. This is due to inheriting all the complexities of traditional software in addition to ML-specific ones. To handle these complexities the field of ML Operations (MLOps) has emerged. The goal of MLOps is to extend DevOps to AI/ML and therefore speed up development and ease maintenance of AI/ML-based software, by, for example, supporting automatic deployment, monitoring, and continuous re-training of models. This thesis investigates how to construct an MLOps workflow by selecting a number of tools and using these to implement a workflow. Additionally, different approaches for triggering re-training are implemented and evaluated, resulting in a comparison of the triggers with regards to execution time, memory and CPU consumption, and the average performance of the Machine learning model.

Over the last few years, Machine Learning has become a huge field with “Big Tech” companies sharing their experiences building machine learning infrastructure. Feature Stores, used as centralized data repositories for machine learning features, are seen as a central component to operational and scalable machine learning. With the growth in machine learning, there is, naturally, a tremendous growth in data used for training. Most of this data tends to sit in Parquet files in cloud object stores or data lakes and is used either directly from files or in-memory where it is used in exploratory data analysis and small batches of training. A majority of the data science involved in machine learning is done in Python, but the infrastructure surrounding it is not always directly compatible with Python. Often, query processing engines and feature stores end up having their own Domain Specific Language or require data scientists to write SQL code, thus leading to some level of ‘transpilation’ overhead across the system. This overhead can not only introduce errors but can also add up to significant time and productivity cost down the line. In this thesis, we conduct a systems research on the performance of offline feature stores and identify ways that allow us to pull out data from feature stores in a fast and efficient way. We conduct a model evaluation based on benchmark tests that address common exploratory data analysis and training use cases. We find that in the Hopsworks feature store, with the use of state-of-the-art, storage-optimized, format-aware, and vector execution-based query processing engine as well as using Arrow protocol from start to finish, we are able to see significant improvements in both creating batch training data (feature value reads) and creating Point-In-Time Correct training data. For batch training data created in-memory, Hopsworks shows an average speedup of 27x over Databricks (5M and 10M scale factors), 18x over Vertex, and 8x over Sagemaker across all scale factors. For batch training data as parquet files, Hopsworks shows a speedup of 5x over Databricks (5M, 10M, and 20M scale factors), 13x over Vertex, and 6x over Sagemaker across all scale factors. For creating in-memory Point-In-Time Correct training data, Hopsworks shows an average speedup of 8x over Databricks, 6x over Vertex, and 3x over Sagemaker across all scale factors. Similary for PIT-Correct training data created as file, Hopsworks shows an average speedup of 9x over Databricks, 8x over Vertex, and 6x over Sagemaker across all scale factors. Through the analysis of these experimental results and the underlying infrastructure, we identify the reasons for this performance gap and examine the strengths and limitations of the design. / Under de senaste åren har maskininlärning blivit ett stort område där ”Big Tech”-företag delar med sig av sina erfarenheter av att bygga infrastruktur för maskininlärning. Feature Stores, som används som centraliserade datalager för maskininlärningsfunktioner, ses som en central komponent för operativ och skalbar maskininlärning. Med tillväxten inom maskininlärning följer naturligtvis en enorm tillväxt av data som används för utbildning. De flesta av dessa data finns i Parquet-filer som lagras i molnobjektsbutiker eller datasjöar och används antingen direkt från filer eller i minnet där de används i explorativ dataanalys och små utbildningsbatcher. En majoritet av datavetenskapen inom maskininlärning görs i Python, men den omgivande infrastrukturen är inte alltid direkt kompatibel med Python. Ofta har motorer för frågebehandling och feature stores sina egna domänspecifika språk eller kräver att datavetare skriver SQL-kod, vilket leder till en viss nivå av `transpileringsoverhead' i hela systemet. Denna overhead kan inte bara leda till fel utan också till betydande tids- och produktivitetskostnader i slutändan. I den här avhandlingen genomför vi en systemstudie av prestandan hos offline feature stores och identifierar sätt som gör att vi kan ta fram data från feature stores på ett snabbt och effektivt sätt. Vi genomför en modellutvärdering baserad på benchmarktester som tar upp vanliga användningsfall för explorativ dataanalys och utbildning. Vi konstaterar att vi i Hopsworks feature store, med hjälp av en toppmodern, lagringsoptimerad, formatmedveten och vektorexekveringsbaserad frågebehandlingsmotor samt Arrow-protokoll från början till slut, kan se betydande förbättringar både när det gäller att skapa batchutbildningsdata (läsa featurevärden) och skapa Point-In-Time Correct-utbildningsdata. För batchutbildningsdata som skapats i minnet visar Hopsworks en genomsnittlig hastighet på 27x över Databricks (5M och 10M skalfaktorer), 18x över Vertex och 8x över Sagemaker över alla skalfaktorer. För batch-träningsdata som parkettfiler visar Hopsworks en hastighetsökning på 5x över Databricks (5M, 10M och 20M skalfaktorer), 13x över Vertex och 6x över Sagemaker över alla skalfaktorer. För att skapa Point-In-Time Correct-träningsdata i minnet visar Hopsworks en genomsnittlig hastighet på 8x över Databricks, 6x över Vertex och 3x över Sagemaker över alla skalfaktorer. På samma sätt för PIT-Correct träningsdata som skapats som fil, visar Hopsworks en genomsnittlig hastighet på 9x över Databricks, 8x över Vertex och 6x över Sagemaker över alla skalfaktorer. Genom att analysera dessa experimentella resultat och den underliggande infrastrukturen identifierar vi orsakerna till denna prestandaklyfta och undersöker styrkorna och begränsningarna i designen.

The purpose of this thesis paper was to develop new features in the cloud-native and open-source machine learning platform STACKn, aiming to strengthen the platform's support for conducting reproducible machine learning experiments through provenance, transparency and reusability. Adhering to the definition of reproducibility as the ability of independent researchers to exactly duplicate scientific results with the same material as in the original experiment, two concepts were explored as alternatives for this specific goal: 1) Increased support for standardized textual documentation of machine learning models and their corresponding datasets; and 2) Increased support for provenance to track the lineage of machine learning models by making code, data and metadata readily available and stored for future reference. We set out to investigate to what degree these features could increase reproducibility in STACKn, both when used in isolation and when combined.  When these features had been implemented through an exhaustive software engineering process, an evaluation of the implemented features was conducted to quantify the degree of reproducibility that STACKn supports. The evaluation showed that the implemented features, especially provenance features, substantially increase the possibilities to conduct reproducible experiments in STACKn, as opposed to when none of the developed features are used. While the employed evaluation method was not entirely objective, these features are clearly a good first initiative in meeting current recommendations and guidelines on how computational science can be made reproducible.