Reinforcement learning (RL) has been at the core of recent advances in fulfilling the AI promise towards general intelligence. Unlike other machine learning (ML) paradigms, such as supervised learning (SL) that learn to mimic how humans act, RL tries to mimic how humans learn, and in many tasks, managed to discover new strategies and achieved super-human performance. This is possible mainly because RL algorithms are allowed to interact with the world to collect the data they need for training by themselves. This is not possible in SL, where the ML model is limited to a dataset collected by humans which can be biased towards sub-optimal solutions. The downside of RL is its high cost when trained on real systems. This high cost stems from the fact that the actions taken by an RL model during the initial phase of training are merely random. To overcome this issue, it is common to train RL models using simulators before deploying them in production. However, designing a realistic simulator that faithfully resembles the real environment is not easy at all. Furthermore, simulator-based approaches don’t utilize the sheer amount of field-data available at their disposal. This work investigates new ways to bridge the gap between SL and RL through an offline pre-training phase. The idea is to utilize the field-data to pre-train RL models in an offline setting (similar to SL), and then allow them to safely explore and improve their performance beyond human-level. The proposed training pipeline includes: (i) a process to convert static datasets into RL-environment, (ii) an MDP-aware data augmentation process of offline-dataset, and (iii) a pre-training step that improves RL exploration phase. We show how to apply this approach to design an action recommendation engine (ARE) that automates network operation centers (NOC); a task that is still tackled by teams of network professionals using hand-crafted rules. Our RL algorithm learns to maximize the Quality of Experience (QoE) of NOC users and minimize the operational costs (OPEX) compared to traditional algorithms. Furthermore, our algorithm is scalable, and can be used to control large-scale networks of arbitrary size.

The master's thesis is focused on the design and implementation of a tool for generating configuration named Ansible. The result of using this tool is generated configuration, which contains random values chosen according to specified parameters and it was deployed on a virtual testing infrastructure. The theoretical part describes approaches of network automation in the process of deploying and configuration of network devices called Infrastructure as code. It also describes programme Ansible, which will be using the output of the implemented tool. The practical part of the thesis is focused on designing the functionality and internal structure of the tool, implementation of the tool and testing implemented tool as well as generated configuration.

Automating the configuration of network devices contributes to faster configuration and helps increase consistency compared with manual configuration. In this thesis, we compare the automation tools Ansible, Puppet, and SaltStack and evaluate their performance when configuring network devices. To evaluate the automation tools this thesis consists of a comprehensive overview and a set of performance experiments. The comprehensive overview focuses on comparing the automation tools and their capabilities whilst the set of performance experiments focuses on comparing the performance of the automation tools. The comprehensive overview reveals that all automation tools we investigate can achieve automated network configuration. However, the approach to configure network devices varies between the automation tools. For the configuration of network devices Ansible is more capable than Puppet and SaltStack. The lack of documentation and support for Puppet and SaltStack leads to an increase in usage complexity. The performance experiments reveal that Ansible and SaltStack have a better performance than Puppet for a lower number of configuration changes. However, with an increasing number of configuration changes, Ansible shows a noticeable difference whilst Puppet and SaltStack do not.

Computer networks are continuously evolving and growing in size and complexity. New technologies are being introduced which further increases the complexity. Net- work Service Orchestration is all about pushing configuration out into the network devices automatically without human intervention. There can be issues that causes the orchestration to fail. In many cases manual operations must be done to recover from the error which is very contradicting since the goal of orchestration is that it should be fully automated. There is some indication that the errors that are being solved manually could be de- tected and handled by a feedback mechanism. This thesis work aimed to build on current insight and if possible, verify that the feedback mechanism is a viable method. After consideration on different ways to solve the research question, the choice fell on creating a test environment where the approach was tested. The test environment was used to investigate if a network orchestration system could be integrated with a feedback mechanism. The result of this project presents a way to automatically de- tect a network failure and send feedback to a Network Service Orchestrator. The or- chestrator is then able to identify and correct the error. / Datornätverk utvecklas kontinuerligt och växer i storlek och komplexitet. Nyteknik införs som ytterligare ökar komplexiteten. Nätverksservice orkestrering handlar om att skicka ut konfiguration automatiskt till enheter i nätverket utan mänsklig in- blandning. Det kan finnas problem som gör att orkestreringen misslyckas. I många fall måste manuella åtgärder utföras för att lösa problemet, vilket är mycket motsä- gelsefullt, eftersom målet med orkestrering är att det ska vara fullt automatiserat. Det finns indikationer på att fel kan detekteras och hanteras av en återkopplings- mekanismen. Detta examensarbete syftar till att bygga på aktuell insikt, och om möj- ligt, verifiera att återkopplingsmekanismen är en möjlig metod. Efter överväganden på vilka olika sätt som projektmålet kunde uppnås föll valet på att skapa en testmiljö där ansatsen kunde testas. Testmiljön användes för att utreda om ett nätverksorkestreringssystem kan integreras med en återkopplings mekanism. Resultat av projektet presenterar ett sätt att automatiskt upptäcka ett nätverksfel och skicka återkoppling till ett nätverksorkestreringssystem. Nätverksorkestreraren kan sedan detektera och åtgärda felet.

Computer networks are continuously evolving and growing in size and complexity. New technologies are being introduced which further increases the complexity. Net- work Service Orchestration is all about pushing configuration out into the network devices automatically without human intervention. There can be issues that causes the orchestration to fail. In many cases manual operations must be done to recover from the error which is very contradicting since the goal of orchestration is that it should be fully automated. There is some indication that the errors that are being solved manually could be de- tected and handled by a feedback mechanism. This thesis work aimed to build on current insight and if possible, verify that the feedback mechanism is a viable method. After consideration on different ways to solve the research question, the choice fell on creating a test environment where the approach was tested. The test environment was used to investigate if a network orchestration system could be integrated with a feedback mechanism. The result of this project presents a way to automatically de- tect a network failure and send feedback to a Network Service Orchestrator. The or- chestrator is then able to identify and correct the error. / Datornätverk utvecklas kontinuerligt och växer i storlek och komplexitet. Nyteknik införs som ytterligare ökar komplexiteten. Nätverksservice orkestrering handlar om att skicka ut konfiguration automatiskt till enheter i nätverket utan mänsklig in- blandning. Det kan finnas problem som gör att orkestreringen misslyckas. I många fall måste manuella åtgärder utföras för att lösa problemet, vilket är mycket motsä- gelsefullt, eftersom målet med orkestrering är att det ska vara fullt automatiserat. Det finns indikationer på att fel kan detekteras och hanteras av en återkopplings- mekanismen. Detta examensarbete syftar till att bygga på aktuell insikt, och om möj- ligt, verifiera att återkopplingsmekanismen är en möjlig metod. Efter överväganden på vilka olika sätt som projektmålet kunde uppnås föll valet på att skapa en testmiljö där ansatsen kunde testas. Testmiljön användes för att utreda om ett nätverksorkestreringssystem kan integreras med en återkopplings mekanism. Resultat av projektet presenterar ett sätt att automatiskt upptäcka ett nätverksfel och skicka återkoppling till ett nätverksorkestreringssystem. Nätverksorkestreraren kan sedan detektera och åtgärda felet.

<p>U disertaciji je predstavljen pristup zasnovan na algoritmu mešovitog celobrojnog linearnog programiranja (MILP) za određivanje optimalnog broja, tipa i lokacije uređaja za automatizaciju distributivne mreže. Ugradnja različitih tipova nove opreme (daljinski kontrolisani reklozeri, sekcioneri i indikatori prolaska struje kvara) kao i relokacija postojeće opreme su istovremeno razmatrani. Prilikom određivanja optimalnog scenarija za automatizaciju, predloženi pristup uvažava troškove ispada potrošača/proizvođača usled trenutnih, kratkotrajnih i dugotrajnih ispada, najčešće korišćene pokazatelje pouzdanosti (SAIFI, SAIDI, MAIFI, i ASIDI) kao i troškove distributivnog preduzeća, asocirane uređajima za automatizaciju i ekipama koje su uključene u proces rešavanja prekida napajanja.<br />Dakle, osnovni cilj ove disertacije je, da se napravi model zasnovan na mešovitom celobrojnom linearnom programiranju koji će omogućiti određivanje najboljeg scenarija za automatizaciju distributivne mreže u slučajevima kada se ocena kvaliteta isporuke definiše preko pokazatelja pouzdanosti, preko troškova usled prekida napajanja ili kombinacijom ova dva pokazatelja.</p> / <p>The dissertation presents a mixed integer linear programming (MILP) based approach for determining the optimal number, type and location of automation devices to be installed in the network by considering different types of devices simultaneously (remotely controlled circuit breakers/reclosers, sectionalizing switches, remotely supervised fault passage indicators). Simultaneously, it determines the new (optimal) locations of the automation devices that already exist in the network. In determining the most effective network automation scenario, the proposed approach takes into account the outage cost of consumers/producers due to momentary, short-term, and long-term interruptions, the commonly used network reliability indices (SAIFI, SAIDI, MAIFI, and ASIDI) as well as the cost of automation devices and the cost of crews. It provides the best network automation scenario in distribution systems if the network reliability indices are used for measuring the distribution system reliability, if cost of interruptions is defined to all consumers/producers, and if both approaches (criteria) are used.</p>

This report discusses network automation primarily with Ansible. Ansible is a software from Red Hat that can be used for network automation. The report also goes through YAML which is a standardized way of exchanging data, Jinja2 that is a templating language, Python as well as the security with Ansible. The report also goes through why network automation is needed as well as how much time might be saved with Ansible. Ansible ships with modules for Cisco IOS such as ios_config and ios_command and for Cisco ASA asa_config, asa_command and asa_acl as well as many other modules for Arista, Juniper and for other vendors. Ansible can use new APIs by creating new modules for handling that particular API, which means that the only change needed in the playbooks is to change the module name. Ansible can handle NETCONF API using the netconf_config module or various Juniper modules. Ansible is used in this report to perform certain tasks such as to adding VLAN's, close ports on ASA's, audit network devices configuration as well as to create network diagram using the information from CDP. Ansible can be made as secure as manually doing the tasks except that Ansible can do it faster and more consistently.  For connecting to normal Linux servers Ansible uses OpenSSH which is a default SSH client on most Linux systems and for connecting to network devices it uses Paramiko. The security in Ansible depends on SSH and may or may not have passwords stored locally, Ansible can be as secure as the administrator wants it to be such as using RSA key-pair to authenticate, using vault encrypted credentials or asking the administrator about which username and password to use. Using Ansible network automation can save time, the amount saved depends on what is being done, how many devices it is doing it on as well as how the playbook is written. / Rapporten behandlar nätverksautomation primärt i Ansible. Ansible är en mjukvara från Red Hat som kan användas för nätverksautomering. Rapporten går igenom YAML som är ett sätt att standardisera överförning av data, Jinja2 som är ett mallspråk, Python samt säkerheten i Ansible. Rapporten går dessutom igenom varför vi ens vill ha nätverksautomation och hur mycket tid som möjligtvis kan sparas. Ansible kommer med moduler för Cisco IOS som exempelvis ios_config och ios_command och för Cisco ASA finns moduler så som asa_config, asa_command och asa_acl. För andra tillverkare så finns det moduler för Arista, Juniper och för andra leverantörer. Om en ny API kommer ut för en ny enhet så kan en ny Ansible modul skapas som använder denna, vilket betyder att Ansible playbooks kan då använda sig av de nya modulerna med samma struktur som tidigare. Ansible kan hantera NETCONF API med hjälp av netconf_config modulen och av flertalet Juniper moduler. Ansible kan användas på ett lika säkert sätt som manuellt arbete, med undantag på att Ansible gör det snabbare och mer konsekvent. För uppkoppling till vanliga Linux-servrar så använder Ansible OpenSSH klienten som standard och mot nätverksenheter utan Python installerat så används Python biblioteket Paramiko. Ansible använder sig av SSH och kan ha lösenord sparat i playbooken, utanför i annan fil, i ett krypterat vault, fråga användaren som användarnamn och lösenord samt autentisering med hjälp av RSA nycklar. Ansible används för att skapa olika VLAN, stänga portar på en ASA, granska nätverksenhetens konfiguration gentemot vad den borde ha för konfiguration samt för att skapa nätverksdiagram baserat på informationen från CDP. Genom att använda sig av Ansible nätverksautomation så kan tid sparas, hur mycket beror helt på vad som ska göras, hur många enheter det ska göras på samt hur playbooken faktiskt är skapad.

The Network Service Ochestrator (NSO) is a network automation system provided by Cisco that is used to automate large network changes with the ability to roll back in case of errors. It provides a rich northbound interface to communicate with the user and a southbound interface to orchestrate network devices securely. On these northbound and southbound interfaces, NSO supports NETCONF and RESTCONF, which is an IETF standard for network automation. NSO native implementation of NETCONF and RESTCONF lacks support for Public-Key Infrastructure (X.509) (PKIX) infrastructure and SSH and SSL/TLS as transport. Instead, Cisco suggests that customers use external relay agents such as PKIX-SSH for SSH and GNUTLS for TLS for NETCONF. The certificates and keys are saved on the hard drive and loaded for every connection via RESTCONF. This workaround solution provides authentication and authorization without audit logging within NSO. In this work, a native implementation of the X509 certification with PKIX infrastructure on SSH and SSL/TLS for NETCONF and RESTCONF is investigated. The project evaluates design alternatives with respect to security, computational complexity, maintainability, and user-friendliness, and concludes by highlighting the pros and cons of both native and workaround implementation. / Ciscos NSO är en nätverksorkestreringsplatform som används för att automatisera stora ändringar i nätverk med egenheten att ändringarna kan backas tillbaka om inte samtliga kan kan utföras. NSO tillhandahåller användare gränssnitt (northbound) för att säkert kommunicera (southbound) med nätverksenheterna. Gränssnitten stödjer de standardiserade protokollen Netconf och Restconf. Båda dessa protokoll saknar inbyggts stöd för PKIX över SSH, SSL och TSL. När detta önskas rekommenderar Cisco sina kunder att externa klienter som PKIX-SSH eller GNUTLS. När detta görs sparas certifikat och nyklar lokalt för varje Restconf koppel och ingen läggning av flödet kommer att ske i NSO. I detta arbete presenteras ett inbyggt stöd för X509 certifiering med PKIX för SSH, SSL, och TLS. Stödet kan användas för Netconf och Restconf. Olikheter mellan dagens tillgängliga stöd och det inbyggda stödet med avseende på säkerhet, komplexitet, underhållbarhet, och användarvänlighet jämförs. Avslutningsvis belyses för- respektive nackdelar med de olika implementateringarna.