Advanced process control solutions are oftentimes inadequate in their handling of uncertainty and disturbances. The main contribution of this work is to address this issue by providing solutions of immediate relevance to process control practitioners. To meet increasing performance demands, this work considers a Hidden Markov Model-based framework for describing non-stationary disturbance signals of practical interest such as intermittent drifts and abrupt jumps. The result is a more sophisticated model used by the state estimator for jump systems. At the expense of slightly higher computational costs (due to the state estimator), the proposed HMM disturbance model provides better tracking compared to a state estimator based on the commonly employed (in process control) integrated white noise disturbance model. Better tracking performance translates to superior closed loop performance without any redesign of the controller, through the typical assumption of separation and certainty equivalence. As a result, this provides a tool that can be readily adopted by process control practitioners. In line with this, the second aim is to develop approximate dynamic programming techniques for the rigorous control of nonlinear stochastic jump systems. The contribution is the creation of a framework that treats uncertainty in a systematic manner whilst leveraging existing off-the-shelf optimization solvers commonly employed by control practitioners.

A series hybrid electric vehicle is a vehicle that is powered by both an engine and a battery pack. An electric motor provides all of the mechanical motive power to the transmission. Engine power is decoupled from the transmission by converting engine power into electricity which powers the electric motor. The mechanical decoupling of the engine from the transmission allows the engine to be run at any operating point (including off) during vehicle operation while the battery back supplies or consumes the remaining power. Therefore, the engine can be operated at its most efficient operating point or in a high-efficiency operating region. The first objective of this research is to develop a dynamic model of a series hybrid diesel-electric powertrain for implementation in Simulink. The vehicle of interest is a John Deere M-Gator utility vehicle. This model serves primarily to test energy management strategies, but it can also be used for component sizing given known load profiles for a vehicle. The second objective of this research is to develop and implement multiple energy management strategies of varying complexity from simple thermostat control to an optimal control law derived using dynamic programming. These energy management strategies are then tested and compared over the criteria of overall fuel efficiency, power availability, battery life, and complexity of implementation. Complexity of implementation is a critical metric for control designers and project managers. The results show that simple point-based control logic can improve upon thermostat control if engine efficiency maps are known. All control method results depend on the load profile being used for a specific application.

This thesis covers a utility optimizing model designed and calibrated for agents of the Swedish economy. The main ingredient providing for this specific country is the modeling of the pension accumulation and pension benefits, which closely mimics the Swedish system. This characteristic is important since it measures one of the only two diversities between genders, that is, the income. The second characteristic is the survival probability. Except for these differences in national statistics, men and women are equal. The reminding model parameters are realistically set estimates from the surrounding economy. When using the model, firstly a baseline agent representing the entire labor force is under the microscope for evaluating the model itself. Next, one representative woman and one representative man from the private and public sectors respectively, composes a set of four samples for investigation of heterogeneity in optimality. The optimum level of consumption and risk-proportion of liquid wealth are solved by maximizing an Epstein-Zin utility function using the method of dynamic programming. The results suggests that both genders benefit from adapting the customized solutions to the problem.

In this work, hybrid systems with regional dynamics are considered. These are systems where transitions between different dynamical regimes occur as the continuous state of the system reaches given switching surfaces. In particular, the attention is focused on the optimal control problem associated with such systems. More precisely, given a specific cost function, the goal is to determine the optimal path of going from a given starting point to a fixed final state during an a priori specified time horizon. The key characteristic of the approach presented in this thesis is a hierarchical decomposition of the hybrid optimal control problem, yielding to a framework which allows a solution on different levels of control. On the highest level of abstraction, the regional structure of the state space is taken into account and a discrete representation of the connections between the different regions provides global accessibility relations between regions. These are used on a lower level of control to formulate the main theorem of this work, namely, the Hybrid Bellman Equation for multimodal systems, which, in fact, provides a characterization of global optimality, given an upper bound on the number of transitions along a hybrid trajectory. Not surprisingly, the optimal solution is hybrid in nature, in that it depends on not only the continuous control signals, but also on discrete decisions as to what domains the system's continuous state should go through in the first place. The main benefit with the proposed approach lies in the fact that a hierarchical Dynamic Programming algorithm can be used to representing both a theoretical characterization of the hybrid solution's structural composition and, from a more application-driven point of view, a numerically implementable calculation rule yielding to globally optimal solutions in a regional dynamics framework. The operation of the recursive algorithm is highlighted by the consideration of numerous examples, among them, a heterogeneous multi-agent problem.

The first part of this research program concerns the development of customized and easily implementable Probably Approximately Correct (PAC)-learning algorithms for episodic tasks over acyclic state spaces. The defining characteristic of our algorithms is that they take explicitly into consideration the acyclic structure of the underlying state space and the episodic nature of the considered learning task. The first of the above two attributes enables a very straightforward and efficient resolution of the ``exploration vs exploitation' dilemma, while the second provides a natural regenerating mechanism that is instrumental in the dynamics of our algorithms. Some additional characteristics that distinguish our algorithms from those developed in the past literature are (i) their direct nature, that eliminates the need of a complete specification of the underlying MDP model and reduces their execution to a very simple computation, and (ii) the unique emphasis that they place in the efficient implementation of the sampling process that is defined by their PAC property. More specifically, the aforementioned PAC-learning algorithms complete their learning task by implementing a systematic episodic sampling schedule on the underlying acyclic state space. This sampling schedule combined with the stochastic nature of the transitions taking place, define the need for efficient routing policies that will help the algorithms complete their exploration program while minimizing, in expectation, the number of executed episodes. The design of an optimal policy that will satisfy a specified pattern of arc visitation requirements in an acyclic stochastic graph, while minimizing the expected number of required episodes, is a challenging problem, even under the assumption that all the branching probabilities involved are known a priori. Hence, the sampling process that takes place in the proposed PAC-learning algorithms gives rise to a novel, very interesting stochastic control/scheduling problem, that is characterized as the problem of the Optimal Node Visitation (ONV) in acyclic stochastic digraphs. The second part of the work presented herein seeks the systematic modelling and analysis of the ONV problem. The last part of this research program explores the computational merits obtained by heuristical implementations that result from the integration of the ONV problem developments into the PAC-algorithms developed in the first part of this work. We study, through numerical experimentation, the relative performance of these resulting heuristical implementations in comparison to (i) the initial version of the PAC-learning algorithms, presented in the first part of the research program, and (ii) standard Q-learning algorithm variations provided in the RL literature. The work presented in this last part reinforces and confirms the driving assumption of this research, i.e., that one can design customized RL algorithms of enhanced performance if the underlying problem structure is taken into account.

[Truncated abstract. Please see the pdf version for the complete text. Also, formulae and special characters can only be approximated here. Please see the pdf version of this abstract for an accurate reproduction.] This thesis treats two aspects of intelligent control: The first part is about long-term optimization by approximating dynamic programming and in the second part a specific class of a fast neural network, related to support vector machines (SVMs), is considered. The first part relates to approximate dynamic programming, especially in the framework of adaptive critic designs (ACDs). Dynamic programming can be used to find an optimal decision or control policy over a long-term period. However, in practice it is difficult, and often impossible, to calculate a dynamic programming solution, due to the 'curse of dimensionality'. The adaptive critic design framework addresses this issue and tries to find a good solution by approximating the dynamic programming process for a stationary environment. In an adaptive critic design there are three modules, the plant or environment to be controlled, a critic to estimate the long-term cost and an action or controller module to produce the decision or control strategy. Even though there have been many publications on the subject over the past two decades, there are some points that have had less attention. While most of the publications address the training of the critic, one of the points that has not received systematic attention is training of the action module.¹ Normally, training starts with an arbitrary, hopefully stable, decision policy and its long-term cost is then estimated by the critic. Often the critic is a neural network that has to be trained, using a temporal difference and Bellman's principle of optimality. Once the critic network has converged, a policy improvement step is carried out by gradient descent to adjust the parameters of the controller network. Then the critic is retrained again to give the new long-term cost estimate. However, it would be preferable to focus more on extremal policies earlier in the training. Therefore, the Calculus of Variations is investigated to discard the idea of using the Euler equations to train the actor. However, an adaptive critic formulation for a continuous plant with a short-term cost as an integral cost density is made and the chain rule is applied to calculate the total derivative of the short-term cost with respect to the actor weights. This is different from the discrete systems, usually used in adaptive critics, which are used in conjunction with total ordered derivatives. This idea is then extended to second order derivatives such that Newton's method can be applied to speed up convergence. Based on this, an almost concurrent actor and critic training was proposed. The equations are developed for any non-linear system and short-term cost density function and these were tested on a linear quadratic regulator (LQR) setup. With this approach the solution to the actor and critic weights can be achieved in only a few actor-critic training cycles. Some other, more minor issues, in the adaptive critic framework are investigated, such as the influence of the discounting factor in the Bellman equation on total ordered derivatives, the target interpretation in backpropagation through time as moving and fixed targets, the relation between simultaneous recurrent networks and dynamic programming is stated and a reinterpretation of the recurrent generalized multilayer perceptron (GMLP) as a recurrent generalized finite impulse MLP (GFIR-MLP) is made. Another subject in this area that is investigated, is that of a hybrid dynamical system, characterized as a continuous plant and a set of basic feedback controllers, which are used to control the plant by finding a switching sequence to select one basic controller at a time. The special but important case is considered when the plant is linear but with some uncertainty in the state space and in the observation vector, and a quadratic cost function. This is a form of robust control, where a dynamic programming solution has to be calculated. &sup1Werbos comments that most treatment of action nets or policies either assume enumerative maximization, which is good only for small problems, except for the games of Backgammon or Go [1], or, gradient-based training. The latter is prone to difficulties with local minima due to the non-convex nature of the cost-to-go function. With incremental methods, such as backpropagation through time, calculus of variations and model-predictive control, the dangers of non-convexity of the cost-to-go function with respect to the control is much less than the with respect to the critic parameters, when the sampling times are small. Therefore, getting the critic right has priority. But with larger sampling times, when the control represents a more complex plan, non-convexity becomes more serious.

DC to DC converters stabilize the voltage obtained from voltage sources such as solar power system, wind energy sources, wave energy sources, rectified voltage from alternators, and so forth. Hence, the need for improving its control algorithm is inevitable. Many algorithms are applied to DC to DC converters. This thesis designs fuzzy adaptive dynamic programming (Fuzzy ADP) algorithm. Also, this thesis implements both adaptive dynamic programming (ADP) and Fuzzy ADP on DC to DC converters to observe the performance of the output voltage trajectories.

Dans l’industrie électrique, le progrès technologique apporté par les réseaux intelligents vient défier l’idée selon laquelle les consommateurs ne pourraient pas réagir aux prix des marchés de gros. L’intégration des Effacements de Demande (ED) dans le système électrique se heurte néanmoins à la question de leur efficacité économique. Cette thèse évalue la valeur économique des ED en s’appuyant sur un modèle de marché de l’énergie sous incertitude permettant de calculer les profits d’un agrégateur, par classe de consommateur et d’usage final. Le modèle appartient à la classe des problèmes linéaires stochastiques à plusieurs périodes. Sa résolution s’appuie sur Stochastic Dual Dynamic Programming. Il apparaît qu’en France, les secteurs rentables sont le load-shedding industriel et le load-shifting du ciment et du papier. Le load-shifting du chauffage électrique n’est pas profitable pour le tertiaire et le résidentiel. De plus, la valeur capacitaire des ED est déterminante. Dans l’ensemble, les ED deviennent viables mais le développement de leur potentiel semble conditionné à une baisse des coûts fixes dans les technologies de réseau intelligent. / In liberalised power markets the inability of consumers to adapt their demand in accordance to wholesale prices is increasingly challenged. Nowadays technical progress within the smart grid industry constitutes promising changes for the integration of end-users into the power system, but the deployment of Demand Response (DR) still faces the challenge of its economic viability. This thesis aims to assess the economic value of DR. We rely on an energy-only market model under uncertainty in order to quantify the revenues of DR aggregators, classified by category of consumers and end-uses of electricity. The model is formulated as a multi-stage stochastic linear problem and solved by Stochastic Dual Dynamic Programming. It appears that in France, industrial load-shedding and load-shifting of cement, paper, and pulp are profitable. For residential and tertiary consumers, load-shifting of electric heating is not profitable. We also show that the capacity value of DR is crucial. Overall, results show that DR is beginning to become economically attractive, but that fixed costs of smart grid technologies still need to come down further to fully develop its potential.

Background: Dynamic programming algorithms provide exact solutions to many problems in computational biology, such as sequence alignment, RNA folding, hidden Markov models (HMMs), and scoring of phylogenetic trees. Structurally analogous algorithms compute optimal solutions, evaluate score distributions, and perform stochastic sampling. This is explained in the theory of Algebraic Dynamic Programming (ADP) by a strict separation of state space traversal (usually represented by a context free grammar), scoring (encoded as an algebra), and choice rule. A key ingredient in this theory is the use of yield parsers that operate on the ordered input data structure, usually strings or ordered trees. The computation of ensemble properties, such as a posteriori probabilities of HMMs or partition functions in RNA folding, requires the combination of two distinct, but intimately related algorithms, known as the inside and the outside recursion. Only the inside recursions are covered by the classical ADP theory. Results: The ideas of ADP are generalized to a much wider scope of data structures by relaxing the concept of parsing. This allows us to formalize the conceptual complementarity of inside and outside variables in a natural way. We demonstrate that outside recursions are generically derivable from inside decomposition schemes. In addition to rephrasing the well-known algorithms for HMMs, pairwise sequence alignment, and RNA folding we show how the TSP and the shortest Hamiltonian path problem can be implemented efficiently in the extended ADP framework. As a showcase application we investigate the ancient evolution of HOX gene clusters in terms of shortest Hamiltonian paths. Conclusions: The generalized ADP framework presented here greatly facilitates the development and implementation of dynamic programming algorithms for a wide spectrum of applications.

Dometic är ett stort industribolag som tillverkar lösningar främst för mobila hem. Fem procent av bolagets omsättning utgörs av en typ av kylbox, så kallad TE-box. TE-boxar består av kropp och ett lock där kylteknologin finns. Långa genomloppstider har resulterat i svårigheter med höga kostnader, långa ledtider från beställning till kund samt svårigheter att hantera fluktuationer i efterfrågan och planera lager. Syftet med detta arbete är att analysera Försörjningskedja för TE-boxarna och föreslå eventuelle förändrings- och förbättringsåtgärder gällande, material-, information- och penningflöde (direkta kostnader). Arbetet är utformat som en fallstudie hos Dometics huvudkontor i Solna strand. Metodvalet för att skapa en förståelse kring operationerna blev möten med intressepersoner. För att skapa en djupare förståelse kring hur problematiken och vilka delar av Supply Chain som är intressanta att undersöka användes teorier inom verksamhetsledning. För att lösa problem användes metoder från optimeringslära och systemteknik. All data som används i studien har tillhandahållits av olika representanter för Dometic. Resultatet visar att informationsflödet bör effektiviseras och att ledtiderna kan sänkas genom att ställa krav på underleverantörer att hålla råvarulager. Det visar även att den nuvarande placeringen av produktionsenheter är den optimala om man väger in olika kostnader och jämför med andra alternativ. Om resultatet skall användas som ett verkligt beslutsunderlag bör fler intressepersoner vara inblandade då man viktar kostnaderna. / Dometic group is a company that produces and sells products for mobile lifestyles. One of many products is a type of coolingbox, called TE-box. The TE-box stands for approximately five percent (5%) of Dometics revenue. Long lead times and presumably avoidable high costs are some problems connected to the production of TE-bxes. There are also imposing difficulties of meeting changes in demand and planning stock. The main purpose of this paper is to examine the supply chain of Dometic, with the TE-box in focus. Recommendations including improvement strategies with respect to material, information and money flows. These will be based on the results of our mathematical analysis and approach to the presented problem. The framework of this paper is the application of relevant mathematical theory to this real-life industrial problem. The models are derived from optimization and systems theory. Data was received directly from the source.