Network operators must ensure their networks meet intended traversal policies (e.g., host A can talk to host B, or inbound traffic to host C goes through a firewall and then a NAT). Violations of the policies may result in revenue loss, reputation damage, and security breaches. Today checking whether the intended policies are enforced correctly is stymied by two fundamental sources of complexity: the diversity and stateful nature of the behaviors of real networks. First, we need to account for vast diversity in both the control plane (e.g., different routing protocols and their interactions) and the data plane (e.g., routers, firewalls, and proxies) of the network. Second, we need to reason about a very large space of stateful behaviors in both the control plane (e.g., the current state being characterized by the route advertisements the routers have seen so far) and the data plane (e.g., a firewall’s current state with respect to a TCP session). Prior work on checking network policies is limited to a particular state of the network. Any attempt to reason about the behavior of the network across its state space is hindered by two fundamental challenges: (i) capturing the diversity of the control and data planes, and (ii) exploring the state space of the control and data planes in a scalable manner. This thesis argues for the feasibility of checking the correctness of realistic network policies by addressing the above challenges via two key insights. First, to combat the challenge of diversity, we design unifying abstractions that glue together different routing protocols in the control plane and diverse network appliances (e.g., firewalls, proxies) in the data plane. Second, to explore the state space of the network in a scalable manner, we build tractable models of the control and data planes (e.g., by decomposing logically independent tasks) and design domain-specific optimizations (e.g., by narrowing down the scope of search given the intended policies). Taken together, these two ideas enable systematic reasoning about the correctness of stateful data and control planes. We show the utility and performance of these techniques across a range of realistic settings.

Redes de computadores orientadas à sustentabilidade ou eficientes energeticamente têm a capacidade de adaptar dinamicamente os modos de consumo de energia dos seus dispositivos de acordo com a demanda do tráfego da rede. Por exemplo, colocar no estado dormente os dispositivos que estão abaixo de um nível de utilização predeterminado, considerado de baixa carga. Neste cenário, existem novos desafios no que diz respeito ao modo como confiabilidade e disponibilidade da rede são avaliadas. O cálculo de confiabilidade e disponibilidade é comumente realizado através das técnicas de cadeia de Markov, ou Conjuntos-Conexos e Conjuntos-Desconexos. Porém, tradicionalmente, estas técnicas são baseadas em valores estáticos e não levam em consideração as mudanças dinâmicas que são inseridas no contexto de redes sustentáveis. Desta forma, este trabalho tem como principais objetivos prover um método capaz de avaliar o impacto na confiabilidade ou disponibilidade da rede, quando alguns dispositivos são colocados e tirados de modos de economia de energia, e apresentar a relação de compromisso entre economia de energia, confiabilidade e disponibilidade da rede. O método proposto, chamado REASoN, é uma composição dos dois métodos supracitados, que foram estendidos de forma a considerar no cálculo a dinamicidade dos ajustes dos níveis de energia. Para fins de avaliação, o trabalho realiza um cálculo numérico empregando o método REASoN, em que foi avaliada a confiabilidade dos dispositivos quando colocados no estado dormente. Os impactos de operações de eficiência energética nas métricas de confiabilidade são expressos como mudanças na quinta casa decimal da confiabilidade da rede como 52 minutos de inatividade de componentes da rede e na quarta casa decimal da disponibilidade com 8h de inatividade. Para uma empresa de transações bancárias, 8h de inatividade pode significar R$ 1 bilhão de perda. O trabalho analisa, também, a implementação do REASoN dentro do contexto de um sistema de gerenciamento de rede orientado a sustentabilidade. Os resultados mostram que, quando o sistema não prioriza disponibilidade e confiabilidade, a economia de energia é de 43%. Já quando a disponibilidade e confiabilidade são priorizadas, a economia é de 27%, um valor representativo. Concluímos que o REASoN é uma ferramenta de grande utilidade para a tomada de decisão em redes de computadores sustentáveis, servindo, como base, para uma análise mais acurada sobre o impacto de mecanismos de economia de energia. / Sustainability-oriented computer networks or energy efficient networks have the ability to dynamically adapt the network device power modes in accordance with the demand of network traffic. For example, by putting to sleep devices that handle traffic below a threshold. In this scenario, new challenges appear in the way that the reliability and availability of the network are evaluated. The calculation of reliability and availability is commonly accomplished through techniques such as Markov chain, or Cut-Set and Tie-Set. However, traditionally these techniques are based on static values and do not take into account the dynamic changes that are in the context of sustainable networks. Thus, this study first aims at providing a method to assess the impact on reliability or availability of the network when some devices are put to sleep. Second, it aims at presenting the trade-off between saving energy and changing the reliability and availability of the network. The proposed method, called REASoN, is a composition of two known methods: Markov chain and Cut-Set and Tie-Set, which were extended in order to consider in calculation the dynamic adjustments of the energy levels. To evaluate the proposed method, the work performs a numerical evaluation. In addition, we evaluated the reliability of the network. The results show that the impact of saving energy might change the reliability in six decimal. The work also analyzes REASoN implemented in a network management system oriented to sustainability, called SustNMS. The results show that when the system prioritizes energy efficiency and accepts decreases on availability, the energy savings reach 43%. However, if the system prioritizes availability the energy savings reach 27%. Hence, REASoN is a powerful tool that can be used for decision making in sustainability-oriented computer networks, achieving a more accurate analysis of the impacts of saving energy.

Redes de computadores orientadas à sustentabilidade ou eficientes energeticamente têm a capacidade de adaptar dinamicamente os modos de consumo de energia dos seus dispositivos de acordo com a demanda do tráfego da rede. Por exemplo, colocar no estado dormente os dispositivos que estão abaixo de um nível de utilização predeterminado, considerado de baixa carga. Neste cenário, existem novos desafios no que diz respeito ao modo como confiabilidade e disponibilidade da rede são avaliadas. O cálculo de confiabilidade e disponibilidade é comumente realizado através das técnicas de cadeia de Markov, ou Conjuntos-Conexos e Conjuntos-Desconexos. Porém, tradicionalmente, estas técnicas são baseadas em valores estáticos e não levam em consideração as mudanças dinâmicas que são inseridas no contexto de redes sustentáveis. Desta forma, este trabalho tem como principais objetivos prover um método capaz de avaliar o impacto na confiabilidade ou disponibilidade da rede, quando alguns dispositivos são colocados e tirados de modos de economia de energia, e apresentar a relação de compromisso entre economia de energia, confiabilidade e disponibilidade da rede. O método proposto, chamado REASoN, é uma composição dos dois métodos supracitados, que foram estendidos de forma a considerar no cálculo a dinamicidade dos ajustes dos níveis de energia. Para fins de avaliação, o trabalho realiza um cálculo numérico empregando o método REASoN, em que foi avaliada a confiabilidade dos dispositivos quando colocados no estado dormente. Os impactos de operações de eficiência energética nas métricas de confiabilidade são expressos como mudanças na quinta casa decimal da confiabilidade da rede como 52 minutos de inatividade de componentes da rede e na quarta casa decimal da disponibilidade com 8h de inatividade. Para uma empresa de transações bancárias, 8h de inatividade pode significar R$ 1 bilhão de perda. O trabalho analisa, também, a implementação do REASoN dentro do contexto de um sistema de gerenciamento de rede orientado a sustentabilidade. Os resultados mostram que, quando o sistema não prioriza disponibilidade e confiabilidade, a economia de energia é de 43%. Já quando a disponibilidade e confiabilidade são priorizadas, a economia é de 27%, um valor representativo. Concluímos que o REASoN é uma ferramenta de grande utilidade para a tomada de decisão em redes de computadores sustentáveis, servindo, como base, para uma análise mais acurada sobre o impacto de mecanismos de economia de energia. / Sustainability-oriented computer networks or energy efficient networks have the ability to dynamically adapt the network device power modes in accordance with the demand of network traffic. For example, by putting to sleep devices that handle traffic below a threshold. In this scenario, new challenges appear in the way that the reliability and availability of the network are evaluated. The calculation of reliability and availability is commonly accomplished through techniques such as Markov chain, or Cut-Set and Tie-Set. However, traditionally these techniques are based on static values and do not take into account the dynamic changes that are in the context of sustainable networks. Thus, this study first aims at providing a method to assess the impact on reliability or availability of the network when some devices are put to sleep. Second, it aims at presenting the trade-off between saving energy and changing the reliability and availability of the network. The proposed method, called REASoN, is a composition of two known methods: Markov chain and Cut-Set and Tie-Set, which were extended in order to consider in calculation the dynamic adjustments of the energy levels. To evaluate the proposed method, the work performs a numerical evaluation. In addition, we evaluated the reliability of the network. The results show that the impact of saving energy might change the reliability in six decimal. The work also analyzes REASoN implemented in a network management system oriented to sustainability, called SustNMS. The results show that when the system prioritizes energy efficiency and accepts decreases on availability, the energy savings reach 43%. However, if the system prioritizes availability the energy savings reach 27%. Hence, REASoN is a powerful tool that can be used for decision making in sustainability-oriented computer networks, achieving a more accurate analysis of the impacts of saving energy.

Modelling dialogue management as a reinforcement learning task enables a system to learn to act optimally by maximising a reward function. This reward function is designed to induce the system behaviour required for goal-oriented applications, which usually means fulfilling the user’s goal as efficiently as possible. However, in real-world spoken dialogue systems, the reward is hard to measure, because the goal of the conversation is often known only to the user. Certainly, the system can ask the user if the goal has been satisfied, but this can be intrusive. Furthermore, in practice, the reliability of the user’s response has been found to be highly variable. In addition, due to the sparsity of the reward signal and the large search space, reinforcement learning-based dialogue policy optimisation is often slow. This thesis presents several approaches to address these problems. To better evaluate a dialogue for policy optimisation, two methods are proposed. First, a recurrent neural network-based predictor pre-trained from off-line data is proposed to estimate task success during subsequent on-line dialogue policy learning to avoid noisy user ratings and problems related to not knowing the user’s goal. Second, an on-line learning framework is described where a dialogue policy is jointly trained alongside a reward function modelled as a Gaussian process with active learning. This mitigates the noisiness of user ratings and minimises user intrusion. It is shown that both off-line and on-line methods achieve practical policy learning in real-world applications, while the latter provides a more general joint learning system directly from users. To enhance the policy learning speed, the use of reward shaping is explored and shown to be effective and complementary to the core policy learning algorithm. Furthermore, as deep reinforcement learning methods have the potential to scale to very large tasks, this thesis also investigates the application to dialogue systems. Two sample-efficient algorithms, trust region actor-critic with experience replay (TRACER) and episodic natural actor-critic with experience replay (eNACER), are introduced. In addition, a corpus of demonstration data is utilised to pre-train the models prior to on-line reinforcement learning to handle the cold start problem. Combining these two methods, a practical approach is demonstrated to effectively learn deep reinforcement learning-based dialogue policies in a task-oriented information seeking domain. Overall, this thesis provides solutions which allow truly on-line and continuous policy learning in spoken dialogue systems.