Cette thèse se compose de deux parties. Dans la première partie, nous étudions les stratégies de temps minimum pour le traitement de la pollution dans de grandes ressources en eau, par exemple des lacs ou réservoirs naturels, à l'aide d'un bioréacteur continu qui fonctionne à un état quasi stationnaire. On contrôle le débit d'entrée d'eau au bioréacteur, dont la sortie revient à la ressource avec le même débit. Nous disposons de l'hypothèse d'homogénéité de la concentration de polluant dans la ressource en proposant trois modèles spatialement structurés. Le premier modèle considère deux zones connectées l'une à l'autre par diffusion et seulement une d'entre elles connectée au bioréacteur. Avec l'aide du Principe du Maximum de Pontryagin, nous montrons que le contrôle optimal en boucle fermée dépend seulement des mesures de pollution dans la zone traitée, sans influence des paramètres de volume, diffusion, ou la concentration dans la zone non traitée. Nous montrons que l'effet d'une pompe de recirculation qui aide à homogénéiser les deux zones est avantageux si opérée à vitesse maximale. Nous prouvons que la famille de fonctions de temps minimal en fonction du paramètre de diffusion est décroissante. Le deuxième modèle consiste en deux zones connectées l'une à l'autre par diffusion et les deux connectées au bioréacteur. Ceci est un problème dont l'ensemble des vitesses est non convexe, pour lequel il n'est pas possible de prouver directement l'existence des solutions. Nous surmontons cette difficulté et résolvons entièrement le problème étudié en appliquant le principe de Pontryagin au problème de contrôle relaxé associé, obtenant un contrôle en boucle fermée qui traite la zone la plus polluée jusqu'au l'homogénéisation des deux concentrations. Nous obtenons des limites explicites sur la fonction valeur via des techniques de Hamilton-Jacobi-Bellman. Nous prouvons que la fonction de temps minimal est non monotone par rapport au paramètre de diffusion. Le troisième modèle consiste en deux zones connectées au bioréacteur en série et une pompe de recirculation entre elles. L'ensemble des contrôles dépend de l'état, et nous montrons que la contrainte est active à partir d'un temps jusqu'à la fin du processus. Nous montrons que le contrôle optimal consiste à l'atteinte d'un temps à partir duquel il est optimal de recirculer à vitesse maximale et ensuite ré-polluer la deuxième zone avec la concentration de la première. Ce résultat est non intuitif. Des simulations numériques illustrent les résultats théoriques, et les stratégies optimales obtenues sont testées sur des modèles hydrodynamiques, en montrant qu'elles sont de bonnes approximations de la solution du problème inhomogène. La deuxième partie consiste au développement et l'étude d'un modèle stochastique de réacteur biologique séquentiel. Le modèle est obtenu comme une limite des processus de naissance et de mort. Nous établissons l'existence et l'unicité des solutions de l'équation contrôlée qui ne satisfait pas les hypothèses habituelles. Nous prouvons que pour n'importe quelle loi de contrôle la probabilité d'extinction de la biomasse est positive. Nous étudions le problème de la maximisation de la probabilité d'atteindre un niveau de pollution cible, avec le réacteur à sa capacité maximale, avant l'extinction. Ce problème ne satisfait aucune des suppositions habituelles (la dynamique n'est pas lipschitzienne, diffusion dégénérée localement hölderienne, contraintes d'état, ensembles cible et absorbant s'intersectent), donc le problème doit être étudié dans deux étapes: en premier lieu, nous prouvons la continuité de la fonction de coût non contrôlée pour les conditions initiales avec le volume maximal et ensuite nous développons un principe de programmation dynamique pour une modification du problème original comme un problème de contrôle optimal avec coût final sans contrainte sur l'état. / This thesis consists of two parts. In the first part we study minimal time strategies for the treatment of pollution in large water volumes, such as lakes or natural reservoirs, using a single continuous bioreactor that operates in a quasi-steady state. The control consists of feeding the bioreactor from the resource, with clean output returning to the resource with the same flow rate. We drop the hypothesis of homogeneity of the pollutant concentration in the water resource by proposing three spatially structured models. The first model considers two zones connected to each other by diffusion and only one of them treated by the bioreactor. With the help of the Pontryagin Maximum Principle, we show that the optimal state feedback depends only on the measurements of pollution in the treated zone, with no influence of volume, diffusion parameter, or pollutant concentration in the untreated zone. We show that the effect of a recirculation pump that helps to mix the two zones is beneficial if operated at full speed. We prove that the family of minimal time functions depending on the diffusion parameter is decreasing. The second model consists of two zones connected to each other by diffusion and each of them connected to the bioreactor. This is a problem with a non convex velocity set for which it is not possible to directly prove the existence of its solutions. We overcome this difficulty and fully solve the studied problem applying Pontryagin's principle to the associated problem with relaxed controls, obtaining a feedback control that treats the most polluted zone up to the homogenization of the two concentrations. We also obtain explicit bounds on its value function via Hamilton-Jacobi-Bellman techniques. We prove that the minimal time function is nonmonotone as a function of the diffusion parameter. The third model consists of a system of two zones connected to the bioreactor in series, and a recirculation pump between them. The control set depends on the state variable; we show that this constraint is active from some time up to the final time. We show that the optimal control consists of waiting up to a time from which it is optimal the mixing at maximum speed, and then to repollute the second zone with the concentration of the first zone. This is a non intuitive result. Numerical simulations illustrate the theoretical results, and the obtained optimal strategies are tested in hydrodynamic models, showing to be good approximations of the solution of the inhomogeneous problem. The second part consists of the development and study of a stochastic model of sequencing batch reactor. We obtain the model as a limit of birth and death processes. We establish the existence and uniqueness of solutions of the controlled equation that does not satisfy the usual assumptions. We prove that with any control law the probability of extinction is positive, which is a non classical result. We study the problem of the maximization of the probability of attaining a target pollution level, with the reactor at maximum capacity, prior to extinction. This problem does not satisfy any of the usual assumptions (non Lipschitz dynamics, degenerate locally H"older diffusion parameter, restricted state space, intersecting reach and avoid sets), so the problem must be studied in two stages: first, we prove the continuity of the uncontrolled cost function for initial conditions with maximum volume, and then we develop a dynamic programming principle for a modification of the problem as an optimal control problem with final cost and without state constraint.

Submitted by Renata Lopes (renatasil82@gmail.com) on 2016-01-13T12:47:04Z No. of bitstreams: 1 elizamariaferreira.pdf: 2774756 bytes, checksum: 61a94ee557cce6b0fb63b18f2d57e8e4 (MD5) / Approved for entry into archive by Adriana Oliveira (adriana.oliveira@ufjf.edu.br) on 2016-01-25T17:29:48Z (GMT) No. of bitstreams: 1 elizamariaferreira.pdf: 2774756 bytes, checksum: 61a94ee557cce6b0fb63b18f2d57e8e4 (MD5) / Made available in DSpace on 2016-01-25T17:29:48Z (GMT). No. of bitstreams: 1 elizamariaferreira.pdf: 2774756 bytes, checksum: 61a94ee557cce6b0fb63b18f2d57e8e4 (MD5) Previous issue date: 2015-02-27 / CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior / O objetivo deste trabalho é estudar algumas aplicações da teoria de controle ótimo para problemas biológicos. Assim, apresentamos inicialmente o estudo de dois modelos diferentes: “Optimal Control of Biological Invasions in Lake Network”, proposto por Potapov et al. [13], e “Simulating Optimal Vaccination Times during Cholera Outbreaks” proposto por Modnak et al. [9]. Os modelos têm suas dinâmicas baseadas em equações diferenciais ordinárias e neles foi minimizado um funcional, com uma única e com várias restrições, respectivamente. No primeiro modelo a teoria de controle ótimo é usada para minimizar os custos com a prevenção juntamente com os custos gerados pelos danos da invasão biológica em estudo, e no segundo modelo aplica-se o controle ótimo para minimizar os custos da vacinação e tratamento dos indivíduos infectados durante um surto de cólera. Com base nos modelos propostos por Vieira e Takahashi em “A Sobrevivência do Vírus varicelazoster”, [16], e por Shulgin et al. em “Pulse vaccination strategy in the SIR epidemic model”, [14], propomos um modelo matemático que considera a vacinação da população como uma estratégia de controle da varicela. Nós usamos a teoria de controle ótimo para definir as condições necessárias para minimizar os custos da vacinação e tratamento dos indivíduos infectados com catapora ou com herpes zoster. A dinâmica é baseada em equações diferenciais ordinárias, que são as restrições sob as quais queremos minimizar o funcional utilizando a teoria de controle ótimo. / The goal of this work is to study some applications of the theory of optimal control for biological problems. Thus, initially we present the study of two different models: “Optimal Control of Biological Invasions in Lake Network” proposed by Potapov et al. [13], and “Simulating optimal Vaccination Times During Cholera Outbreaks” proposed by Modnak et al. [9]. The models have their dynamics based on ordinary differential equations and are minimizing the functional with a single and with several restrictions, respectively. The first model uses optimal control theory to minimize costs with prevention and together with the costs generated by the damage of the invasion, the second model applies optimal control to minimize costs in the vaccination and treatment of infected individuals during cholera outbreak. Based on models proposed by Vieira and Takahashi on “The Virus Survival varicella-zoster”, [16], and by Shulgin et al. in “Pulse vaccination strategy in the SIR epidemic model”, [14], we propose a mathematical model that considers a vaccination of the population as a varicella control strategy. We use the optimal control theory to define necessary conditions to minimize the costs of vaccination and treatment of infected individuals with chickenpox or with herpes zoster. The dynamics is based on ordinary differential equations which are the constraints under which we want to minimize the functional in the optimal control theory.

Esta dissertação aborda o problema de redução do consumo de combustível para veículos. Com esse objetivo, realiza-se o levantamento de um modelo estocástico e de seus parâmetros, o desenvolvimento de um controlador para o veículo, e análise dos resultados. O problema considera a interação com o trânsito de outros veículos, que limita a aplicação de resultados antes disponíveis. Para isto, propõe-se modelar a dinâmica do problema de maneira aproximada, usando sistemas com saltos markovianos, e levantar as probabilidades de transição dos estados da cadeia através de um modelo mais completo para o trânsito no percurso. / This dissertation deals with control of vehicles aiming at the fuel consumption optimization, taking into account the interference of traffic. Stochastic interferences like this and other real world phenomena prevents us from directly applying available results. We propose to employ a relatively simple system with Markov jumping parameters as a model for the vehicle subject to traffic interference, and to obtain the transition probabilities from a separate model for the traffic. This dissertation presents the model identification, the solution of the new problem using dynamic programming, and simulation of the obtained control.

Buffer tanks are widely used within the process industry to prevent flow variations from being directly propagated throughout a plant. The capacity of the tank is used to smoothly transfer inlet flow upsets to the outlet. Ideally, the tank thus works as a low pass filter where the available tank capacity limits the achievable flow smoothing. For infrequently occurring upsets, where the system has time to reach steady state between flow changes, the averaging level control problem has been extensively studied. After an inlet flow change, flow filtering has traditionally been obtained by letting the tank level deviate from its nominal value while slowly adapting the outlet to cancel out the flow imbalance and eventually bringing back the level to its set-point. The system is then again in steady state and ready to surge the next upset. By ensuring that the single largest upset can be handled without violating the level constraints, satisfactory flow smoothing is obtained. In this thesis, the smoothing of frequently changing inlet flows is addressed. In this case, standard level controllers struggle to obtain acceptable flow smoothing since the system rarely is in steady state and flow upsets can thus not be treated as separate events. To obtain a control law that achieves optimal filtering while directly accounting for future upsets, the averaging level control problem was approached using robust model predictive control (MPC). The robust MPC differs in the way it obtains flow smoothing by not returning the tank level to a fixed set-point. Instead, it lets the steady state tank level depend on the current value of the inlet flow. This insight was then used to propose a linear control structure, designed to filter frequent upsets optimally. Analyses and simulation results indicate that the proposed linear and robust MPC controller obtain flow smoothing comparable to the standard optimal averaging level controllers for infrequent upsets while handling frequent upsets considerably better.

Motion planning is defined as the problem of computing a feasible trajectory for an agent to follow. It is a well-studied problem with applications in fields such as robotics, control theory and artificial intelligence. In the last decade there has been an increased interest in algorithms for motion planning under uncertainty where the agent does not know the state of the environment due to, e.g. motion and sensing uncertainties. One approach is to generate an initial feasible trajectory using for example an algorithm such as RRT* and then improve that initial trajectory using local optimization. This thesis proposes a new modification of the RRT* algorithm that can be used to generate initial paths from which initial trajectories for the local optimization step can be generated. Unlike standard RRT*, the modified RRT* generates multiple paths at the same time, all belonging to different families of solutions (homotopy classes). Algorithms for motion planning under uncertainty that rely on local optimization of trajectories can use trajectories generated from these paths as initial solutions. The modified RRT* is implemented and its performance with respect to computation time and number of paths found is evaluated on simple scenarios. The evaluations show that the modified RRT* successfully computes solutions in multiple homotopy classes. Two methods for motion planning under uncertainty, Trajectory-optimized LQG (T-LQG), and a belief space variant of iterative LQG (iLQG) are implemented and combined with the modified RRT*. The performance with respect to cost function improvement, computation time and success rate when following the optimized trajectories for the two methods are evaluated in a simulation study. The results from the simulation studies show that it is advantageous to generate multiple initial trajectories. Some initial trajectories, due to for example passing through narrow passages or through areas with high uncertainties, can only be slightly improved by trajectory optimization or results in trajectories that are hard to follow or with a high collision risk. If multiple initial trajectories are generated the probability is higher that at least one of them will result in an optimized trajectory that is easy to follow, with lower uncertainty and lower collision risk than the initial trajectory. The results also show that iLQG is much more computationally expensive than T-LQG, but that it is better at computing control policies to follow the optimized trajectories.

abstract: Robotic lower limb prostheses provide new opportunities to help transfemoral amputees regain mobility. However, their application is impeded by that the impedance control parameters need to be tuned and optimized manually by prosthetists for each individual user in different task environments. Reinforcement learning (RL) is capable of automatically learning from interacting with the environment. It becomes a natural candidate to replace human prosthetists to customize the control parameters. However, neither traditional RL approaches nor the popular deep RL approaches are readily suitable for learning with limited number of samples and samples with large variations. This dissertation aims to explore new RL based adaptive solutions that are data-efficient for controlling robotic prostheses. This dissertation begins by proposing a new flexible policy iteration (FPI) framework. To improve sample efficiency, FPI can utilize either on-policy or off-policy learning strategy, can learn from either online or offline data, and can even adopt exiting knowledge of an external critic. Approximate convergence to Bellman optimal solutions are guaranteed under mild conditions. Simulation studies validated that FPI was data efficient compared to several established RL methods. Furthermore, a simplified version of FPI was implemented to learn from offline data, and then the learned policy was successfully tested for tuning the control parameters online on a human subject. Next, the dissertation discusses RL control with information transfer (RL-IT), or knowledge-guided RL (KG-RL), which is motivated to benefit from transferring knowledge acquired from one subject to another. To explore its feasibility, knowledge was extracted from data measurements of able-bodied (AB) subjects, and transferred to guide Q-learning control for an amputee in OpenSim simulations. This result again demonstrated that data and time efficiency were improved using previous knowledge. While the present study is new and promising, there are still many open questions to be addressed in future research. To account for human adaption, the learning control objective function may be designed to incorporate human-prosthesis performance feedback such as symmetry, user comfort level and satisfaction, and user energy consumption. To make the RL based control parameter tuning practical in real life, it should be further developed and tested in different use environments, such as from level ground walking to stair ascending or descending, and from walking to running. / Dissertation/Thesis / Doctoral Dissertation Electrical Engineering 2020

Cette thèse porte sur un problème couplé des lanceurs, à savoir une manœuvre de l'attitude couplée avec la trajectoire minimisant le temps de manœuvre. La difficulté de ce problème vient essentiellement du phénomène de chattering et du couplage des dynamiques n'ayant pas la même échelle de temps. Avec une analyse géométrique des extrémales venant de l'application du principe du maximum de pontryagin, nous donnons des conditions suffisantes sous lesquelles le phénomène de chattering se produit, pour des systèmes affines bi-entrée. Nons appliquons ensuite ce résultat à notre problème, et montrons que le phénomène de chattering arrive pour les trajectoires optimales, pour certaines données terminales. A l'aide de cette analyse théorique préliminaire, nous mettons en œuvre une méthode de résolution indirecte efficace, combinée à une méthode de continuation prédicteur-correcteur. En cas de chattering, deux stratégies sous-optimales sont proposées: soit une méthode directe dont le contrôle est approché par un contrôle constant par morceaux, soit en stoppant la continuation avant l'échec dû au chattering. Avec le tir multiple et plusieurs paramètres de continuations supplémentaires, cette méthode de résolution est appliquée à chercher une manœuvre de pull-up avec des contraintes sur l'état en minimisant le temps-énergie pour des lanceurs aéroportés. Les résultats numériques permettent de mettre en évidence l'efficacité et la robustesse de notre méthode de résolution. / In this thesis, we investigate the minimum time control problem for the control and guidance of a launch vehicle, whose motion is described by its attitude kinematics and dynamics but also by its trajectory dynamics. The difficulty of this problem is essentially due to the chattering phenomenon and to the coupling of dynamics of different time scales. With a refined geometric study of the extremals coming from the application of the pontryagin maximum principle, we establish a general result for bi-input control-affine systems, providing sufficient conditions under which the chattering phenomenon occurs. We show how this result can be applied to our problem. Based on this preliminary theoretical analysis, we implement an efficient indirect numerical method, combined with numerical predictor-corrector continuation, in order to compute numerically the optimal solutions of the problem. In case of chattering, two sub-optimal strategies are designed: one is a direct method in which the control is approximated by a piecewise constant control, and the other consists of stopping the continuation procedure before its failure due to chattering. With several additional numerical continuation steps, we apply finally the developed indirect approach to the minimum time-energy pull-up maneuver problem, in which state constraints are also considered, for airborne launchers. Numerical simulations illustrate the efficiency and robustness of our method.

Orientador: Daniela Renata Cantane / Resumo: Este trabalho utiliza um modelo epidemiológico para analisar o comportamento de crises bancárias que possuem origem em um determinado país e são propagadas para outros países atingindo proporções mundiais. O modelo matemático epidemiológico Suscetíveis, Infectados e Recuperados (SIR) empregado permite simular a dinâmica da crise separando os países em três estados: suscetíveis, infectados e recuperados, em cada instante de tempo, além de prever a extensão da crise. Os parâmetros do modelo são obtidos da literatura para cada país envolvido e a crise segue uma dinâmica diferente dependendo do país de origem. Uma breve descrição da importância dos bancos em nível macroeconônico e suas funções básicas são apresentadas. Também são apresentadas algumas definições desta crise, denominada crise sistêmica, bem como os canais de transmissão de como um banco com problemas financeiros, denominado infectado, transmite esta condição para outro. Considerada a possibilidade de uma crise sistêmica, o Banco Central deve intervir nos bancos com problemas. Esta tarefa pode ser modelada como um problema de controle ótimo inserindo uma variável de controle no modelo SIR, que representa a intervenção do Banco Central, e uma função objetivo, em que o custo dessa intervenção deve ser minimizado. O objetivo deste trabalho é investigar um modelo de otimização aplicado ao risco bancário e propor o método heurístico \textit{Variable Neighbourhood Search} (VNS) para resolução do problema de controle ótimo... (Resumo completo, clicar acesso eletrônico abaixo) / Abstract: This work uses an epidemiological model to analyze the behavior of bank crises that originate in a given country and are propagated to other countries reaching worldwide proportions. The epidemiological mathematical model Susceptible, Infected and Recovered (SIR) used allows to simulate the dynamics of the crisis separating the countries in three states: susceptible, infected and recovered, in each instant of time, in addition to predicting the extent of the crisis. The model parameters are obtained from the literature for each country involved and the crisis follows a different dynamic depending on the country of origin. A brief description of the importance of banks at the macroeconomic level and their basic functions is presented. Some definitions of this crisis, called systemic crisis, are also presented, as well as the transmission channels of how a bank with financial problems, called infected, transmits this condition to another. Considering the possibility of a systemic crisis, the Central Bank must intervene in troubled banks. This task can be xiv modeled as an optimal control problem by inserting a control variable in the SIR model, which represents Central Bank intervention, and an objective function involving the cost of this intervention and must be minimized. The objective of this work is to investigate an optimization model applied to banking risk and propose the Variable Neighborhood Search (VNS) heuristic method to solve the proposed optimal control problem. ... (Complete abstract click electronic access below) / Mestre

This thesis develops a calibrated model of the Cal Poly Central Chilled Water Plant with Thermal Energy Storage for use in determining an optimal operating control strategy. The model was developed using a transient systems simulation program (TRNSYS) that includes plant performance and manufacturer data for the primary system components, which are comprised of pumps, chillers, cooling towers, and a thermal energy storage tank. The model is calibrated to the actual measured performance of the plant using the current control strategy as a baseline. By observing and quantifying areas for potential improvement in plant performance under conditions of high campus cooling load demands, alternative control strategies for the plant are proposed. Operation of the plant under each of these control strategies is simulated in the model and evaluated for overall energy and demand-usage cost savings. These results are used to recommend improvements in the plant’s current control strategy, as well as to propose an optimal control strategy that may be applied to reduce plant operating costs. The results of the model identify that the plant can perform more economically by employing more chiller power to charge the Thermal Energy Storage tank to higher capacities during overnight periods when the utility rates are lower. Staging the operation of the different chillers to more precisely follow the tank charges during these off-peak periods can ensure faster tank charging when its capacity may not be sufficient to meet the peak and part-peak cooling load demands. A proposed control strategy to accomplish this breaks the overnight Off-Peak rate period into three periods with separate control setpoints, which are designed to maintain the tank charge capacity at the minimum levels to be able to accommodate the daily campus cooling demands during peak and part-peak hours.

In Canada, more than 1.11 TWh of energy per year is generated by diesel generators in off-grid remote areas. Delivering energy to these territories always has a high cost for the local and federal governments both financially and environmentally. Substituting fossil fuels with clean energies is the solution. However, the unreliability and intermittency of renewable energies (RE) are always challenging issues that need to be solved. Zinc air flow battery (ZAFB) with decoupled power and energy capacity can bring sustainability and reliability for microgrids. In this study, an efficient model was developed for ZAFB, which is applicable for large-scale modeling, and incorporated in microgrid modeling. A bilevel optimization approach was implemented in the microgrid model to find the optimal size and control of the microgrid simultaneously over the project lifetime. Using model predictive control (MPC) and based on user-defined foresight horizon and known information like energy demand and RE resources, the control model decides the future changes in microgrid components. This tool is used to propose the best microgrid design for these communities to reduce or eliminate their dependency on fossil fuels. The functionality of this tool was evaluated by three case studies in British Columbia: Blind Channel, Hot Springs Cove and Moresby Island. Zero CO2 emission and zero fuel consumption were achieved by a 100% RE microgrid consisting of wind and tidal turbines and large ZAFB. The net present cost (NPC) of this system and cost of energy are 39 – 46 % and 55 – 60 % less than the base case costs in which diesel is the main energy source. ZAFB with a longer storage duration (50 – 60 hours) satisfies 17 – 23% of annual energy demand in these case studies. / Graduate