During the past two decades desperate need for energy-efficient vehicles which has less emission have led to a great attention to and development of electrified vehicles like pure electric, Hybrid Electric Vehicle (HEV) and Plug-in Hybrid Electric Vehicles (PHEVs). Resultantly, a great amount of research efforts have been dedicated to development of control strategies for this type of vehicles including PHEV which is the case study in this thesis. This thesis presents a real-time control scheme to improve the fuel economy of plug-in hybrid electric vehicles (PHEVs) by accounting for the instantaneous states of the system as well as the future trip information. To design the mentioned parametric real-time power management policies, we use dynamic programming (DP). First, a representative power-split PHEV powertrain model is introduced, followed by a DP formulation for obtaining the optimal powertrain trajectories from the energy cost point of view for a given drive cycle. The state and decision variables in the DP algorithm are selected in a way that provides the best tradeoff between the computational time and accuracy which is the first contribution of this research effort. These trajectories are then used to train a set of linear maps for the powertrain control variables such as the engine and electric motor/generator torque inputs, through a least-squares optimization process. The DP results indicate that the trip length (distance from the start of the trip to the next charging station) is a key factor in determining the optimal control decisions. To account for this factor, an additional input variable pertaining to the remaining length of the trip is considered during the training of the real-time control policies. The proposed controller receives the demanded propulsion force and the powertrain variables as inputs, and generates the torque commands for the engine and the electric drivetrain system. Numerical simulations indicate that the proposed control policy is able to approximate the optimal trajectories with a good accuracy using the real-time information for the same drive cycles as trained and drive cycle out of training set. To maintain the battery state-of-charge (SOC) above a certain lower bound, two logics have been introduced a switching logic is implemented to transition to a conservative control policy when the battery SOC drops below a certain threshold. Simulation results indicate the effectiveness of the proposed approach in achieving near-optimal performance while maintaining the SOC within the desired range. / MS

"Multi-Agent systems" is a topic for a lot of research, especially research involving strategy, evolution and cooperation among various agents. Various learning algorithm schemes have been proposed such as reinforcement learning and evolutionary computing. In this thesis two solutions to a multi-agent herding problem are presented. One solution is based on Q-learning algorithm, while the other is based on modeling of artificial immune system. Q-learning solution for the herding problem is developed, using region-based local learning for each individual agent. Individual and batch processing reinforcement algorithms are implemented for non-cooperative agents. Agents in this formulation do not share any information or knowledge. Issues such as computational requirements, and convergence are discussed. An idiotopic artificial immune network is proposed that includes individual B-cell model for agents and T-cell model for controlling the interaction among these agents. Two network models are proposed--one for evolving group behavior/strategy arbitration and the other for individual action selection. A comparative study of the Q-learning solution and the immune network solution is done on important aspects such as computation requirements, predictability, and convergence. / Master of Science

We present a resource allocation perspective to Quality of Service in Distributed Virtual Environments. The user of a DVE system will have improved Quality of Service if he/she is allocated the right amount of resources at the right time. Instead of allocating resources on a static basis, we adopt a dynamic need based resource allocation scheme that provides real-time resource allocation. Optimal resource assignments are calculated offline and a neural network is trained with the knowledge of optimal solutions from the offline Operations Research Techniques and it is then used to deliver near-optimal resource allocation decisions in real-time. We also present a case study of network bandwidth allocation and prove the usefulness of the technique. / Master of Science

La régulation adaptative des feux de signalisation est un problème très important. Beaucoup de chercheurs travaillent continuellement afin de résoudre les problémes liés à l’embouteillage dans les intersections urbaines. Il devient par conséquent très utile d’employer des algorithmes intelligents afin d’améliorer les performances de régulation et la qualité du service. Dans cette thèse, nous essayons d'étudier ce problème d’une part à travers une modèlisation microscopique et dynamique en temps discret, et d’autre part en explorant plusieurs approches de résoltion pour une intersection isolée ainsi que pour un réseau distribué d'intersections.La première partie se concentre sur la modélisation dynamique des problèmes des feux de signalisation ainsi que de la charge du réseau d’intersections. Le mode de la “séquence de phase adaptative” (APS) dans un plan de feux est d'abord considéré. Quant à la modélisation du contrôle des feux aux intersections, elle est formulée grâce à un processus décisionnel de markov (MDP). En particulier, la notion de “l'état du système accordable” est alors proposée pour la coordination du réseau de trafic. En outre, un nouveau modèle de “véhicule-suiveur” est proposé pour l'environnement de trafic. En se basant sur la modélisation proposée, les méthodes de contrôle des feux dans cette thèse comportent des algorithmes optimaux et quasi-optimaux. Deux algorithmes exacts de résolution basées sur la programmation dynamique (DP) sont alors étudiés et les résultats montrent certaines limites de cette solution DP surtout dans quelques cas complexes où l'espace d'états est assez important. En raison de l’importance du temps d’execution de l'algorithme DP et du manque d'information du modèle (notamment l’information exacte relative à l’arrivée des véhicules à l’intersection), nous avons opté pour un algorithme de programmation dynamique approximative (ADP). Enfin, un algorithme quasi-optimal utilisant l'ADP combinée à la méthode d’amélioration RLS-TD (λ) est choisi. Dans les simulations, en particulier avec l'intégration du mode de phase APS, l'algorithme proposé montre de bons résultats notamment en terme de performance et d'efficacité de calcul. / Adaptive traffic signal control is a decision making optimization problem. People address this crucial problem constantly in order to solve the traffic congestion at urban intersections. It is very popular to use intelligent algorithms to improve control performances, such as traffic delay. In the thesis, we try to study this problem comprehensively with a microscopic and dynamic model in discrete-time, and investigate the related algorithms both for isolated intersection and distributed network control. At first, we focus on dynamic modeling for adaptive traffic signal control and network loading problems. The proposed adaptive phase sequence (APS) mode is highlighted as one of the signal phase control mechanisms. As for the modeling of signal control at intersections, problems are fundamentally formulated by Markov decision process (MDP), especially the concept of tunable system state is proposed for the traffic network coordination. Moreover, a new vehicle-following model supports for the network loading environment.Based on the model, signal control methods in the thesis are studied by optimal and near-optimal algorithms in turn. Two exact DP algorithms are investigated and results show some limitations of DP solution when large state space appears in complex cases. Because of the computational burden and unknown model information in dynamic programming (DP), it is suggested to use an approximate dynamic programming (ADP). Finally, the online near-optimal algorithm using ADP with RLS-TD(λ) is confirmed. In simulation experiments, especially with the integration of APS, the proposed algorithm indicates a great advantage in performance measures and computation efficiency.

Submitted by Renata Lopes (renatasil82@gmail.com) on 2016-07-28T12:04:17Z No. of bitstreams: 1 rafaelbrunodasilvabrandi.pdf: 13228407 bytes, checksum: 1e92e8c2fa686ddcaea1c9ed0d33b278 (MD5) / Approved for entry into archive by Adriana Oliveira (adriana.oliveira@ufjf.edu.br) on 2016-07-28T12:16:14Z (GMT) No. of bitstreams: 1 rafaelbrunodasilvabrandi.pdf: 13228407 bytes, checksum: 1e92e8c2fa686ddcaea1c9ed0d33b278 (MD5) / Made available in DSpace on 2016-07-28T12:16:14Z (GMT). No. of bitstreams: 1 rafaelbrunodasilvabrandi.pdf: 13228407 bytes, checksum: 1e92e8c2fa686ddcaea1c9ed0d33b278 (MD5) Previous issue date: 2016-02-29 / CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior / CNPq - Conselho Nacional de Desenvolvimento Científico e Tecnológico / O Sistema Elétrico Brasileiro (SEB) apresenta características peculiares devido às grandes dimensões do país e pelo fato da geração elétrica ser proveniente predominantemente de usinas hidráulicas. Como as afluências a estas usinas possuem comportamento estocástico e grandes reservatórios proporcionam ao sistema a capacidade de uma regularização plurianual, a utilização dos recursos hidráulicos deve ser planejada de forma minuciosa em um horizonte de tamanho considerável. Assim, o planejamento da operação de médio prazo compreende um período de 5 a 10 anos com discretização mensal e é realizado por uma cadeia de modelos computacionais tal que o principal modelo desta cadeia é baseado na técnica da Programação Dinâmica Dual Estocástica (PDDE). O objetivo deste trabalho é obter avanços nas metodologias de programação dinâmica atualmente utilizadas. Partindo-se da utilização da inserção iterativa de cortes, implementa-se um modelo computacional para o planejamento da operação de médio prazo baseado na metodologia de Programação Dinâmica Estocástica (PDE) utilizando uma discretização mais eficiente do espaço de estados (PDEE). Além disso, a metodologia proposta de PDE possui um critério de convergência bem definido para o problema, de forma que a inclusão da medida de risco CVaR não altera o processo de avaliação da convergência de forma significante. Dado que a inclusão desta medida de risco à PDDE convencional dificulta a avaliação da convergência do processo pela dificuldade da estimação de um limite superior válido, o critério de convergência proposto na PDEE é, então, base para um novo critério de convergência para a PDDE tal que pode ser aplicado mesmo na consideração do CVaR e não aumenta o custo computacional envolvido. Adicionalmente, obtém-se um critério de convergência mais detalhado em que as séries utilizadas para amostras de afluência podem ser avaliadas individualmente tais que aquelas que, em certo momento, não contribuam de forma determinante para a convergência podem ser descartadas do processo, diminuindo o tempo computacional, ou ainda serem substituídas por novas séries dentro de uma reamostragem mais seletiva dos cenários utilizados na PDDE. As metodologias propostas foram aplicadas para o cálculo do planejamento de médio prazo do SIN baseando-se em subsistemas equivalentes de energia. Observa-se uma melhoria no algoritmo base utilizado para a PDE e que o critério proposto para convergência da PDDE possui validade mesmo quando CVaR é considerado na modelagem. / The Brazilian National Grid (BNG) presents peculiar characteristics due to its huge territory dimensions and hydro-generation predominancy. As the water inflows to these plants are stochastic and a pluriannual regularization for system storage capacity is provided, the use of hydro-generation must be planned in an accurate manner such that it considersalongplanningperiod. So, thelong-termoperationplanning(LTOP)problemis generallysolvedbyachainofcomputationalmodelsthatconsideraperiodof5to10years ahead such that the primary model of this chain is based on Stochastic Dual Dynamic Programming (SDDP) technique. The main contribution of this thesis is to propose some improvements in Stochastic Dynamic Programming techniques usually settled on solving LTOP problems. In the fashion of an iterative cut selection, it is firstly proposed a LTOP problem solution model that uses an ecient state space discretization for Stochastic Dynamic Programming (SDP), called ESDP. The proposed model of SDP has a welldefined convergence criterion such that including CVaR does not hinder convergence analysis. Due to the lack of good upper bound estimators in SDDP when including CVaR, additional issues are encountered on defining a convergence criterion. So, based on ESDP convergence analysis, a new criterion for SDDP convergence is proposed such that it can be used regardless of CVaR representation with no extra computational burden. Moreover, the proposed convergence criterion for SDDP has a more detailed description such that forward paths can be individually assessed and then be accordingly discarded for computational time reduction, or even define paths to be replaced in a more particular resampling scheme in SDDP. Based on aggregate reservoir representation, the proposed methodsofconvergenceofSDDPandtheESDPwereappliedonLTOPproblemsrelatedto BNG. Results show improvements in SDDP based technique and eectiveness of proposed convergence criterion for SDDP when CVaR is used.

[ES] La presente Tesis emplea técnicas de programación dinámica y aprendizaje por refuerzo para el control de sistemas no lineales en espacios discretos y continuos. Inicialmente se realiza una revisión de los conceptos básicos de programación dinámica y aprendizaje por refuerzo para sistemas con un número finito de estados. Se analiza la extensión de estas técnicas mediante el uso de funciones de aproximación que permiten ampliar su aplicabilidad a sistemas con un gran número de estados o sistemas continuos. Las contribuciones de la Tesis son: -Se presenta una metodología que combina identificación y ajuste de la función Q, que incluye la identificación de un modelo Takagi-Sugeno, el cálculo de controladores subóptimos a partir de desigualdades matriciales lineales y el consiguiente ajuste basado en datos de la función Q a través de una optimización monotónica. -Se propone una metodología para el aprendizaje de controladores utilizando programación dinámica aproximada a través de programación lineal. La metodología hace que ADP-LP funcione en aplicaciones prácticas de control con estados y acciones continuos. La metodología propuesta estima una cota inferior y superior de la función de valor óptima a través de aproximadores funcionales. Se establecen pautas para los datos y la regularización de regresores con el fin de obtener resultados satisfactorios evitando soluciones no acotadas o mal condicionadas. -Se plantea una metodología bajo el enfoque de programación lineal aplicada a programación dinámica aproximada para obtener una mejor aproximación de la función de valor óptima en una determinada región del espacio de estados. La metodología propone aprender gradualmente una política utilizando datos disponibles sólo en la región de exploración. La exploración incrementa progresivamente la región de aprendizaje hasta obtener una política convergida. / [CA] La present Tesi empra tècniques de programació dinàmica i aprenentatge per reforç per al control de sistemes no lineals en espais discrets i continus. Inicialment es realitza una revisió dels conceptes bàsics de programació dinàmica i aprenentatge per reforç per a sistemes amb un nombre finit d'estats. S'analitza l'extensió d'aquestes tècniques mitjançant l'ús de funcions d'aproximació que permeten ampliar la seua aplicabilitat a sistemes amb un gran nombre d'estats o sistemes continus. Les contribucions de la Tesi són: -Es presenta una metodologia que combina identificació i ajust de la funció Q, que inclou la identificació d'un model Takagi-Sugeno, el càlcul de controladors subòptims a partir de desigualtats matricials lineals i el consegüent ajust basat en dades de la funció Q a través d'una optimització monotónica. -Es proposa una metodologia per a l'aprenentatge de controladors utilitzant programació dinàmica aproximada a través de programació lineal. La metodologia fa que ADP-LP funcione en aplicacions pràctiques de control amb estats i accions continus. La metodologia proposada estima una cota inferior i superior de la funció de valor òptima a través de aproximadores funcionals. S'estableixen pautes per a les dades i la regularització de regresores amb la finalitat d'obtenir resultats satisfactoris evitant solucions no fitades o mal condicionades. -Es planteja una metodologia sota l'enfocament de programació lineal aplicada a programació dinàmica aproximada per a obtenir una millor aproximació de la funció de valor òptima en una determinada regió de l'espai d'estats. La metodologia proposa aprendre gradualment una política utilitzant dades disponibles només a la regió d'exploració. L'exploració incrementa progressivament la regió d'aprenentatge fins a obtenir una política convergida. / [EN] The present Thesis employs dynamic programming and reinforcement learning techniques in order to obtain optimal policies for controlling nonlinear systems with discrete and continuous states and actions. Initially, a review of the basic concepts of dynamic programming and reinforcement learning is carried out for systems with a finite number of states. After that, the extension of these techniques to systems with a large number of states or continuous state systems is analysed using approximation functions. The contributions of the Thesis are: -A combined identification/Q-function fitting methodology, which involves identification of a Takagi-Sugeno model, computation of (sub)optimal controllers from Linear Matrix Inequalities, and the subsequent data-based fitting of Q-function via monotonic optimisation. -A methodology for learning controllers using approximate dynamic programming via linear programming is presented. The methodology makes that ADP-LP approach can work in practical control applications with continuous state and input spaces. The proposed methodology estimates a lower bound and upper bound of the optimal value function through functional approximators. Guidelines are provided for data and regressor regularisation in order to obtain satisfactory results avoiding unbounded or ill-conditioned solutions. -A methodology of approximate dynamic programming via linear programming in order to obtain a better approximation of the optimal value function in a specific region of state space. The methodology proposes to gradually learn a policy using data available only in the exploration region. The exploration progressively increases the learning region until a converged policy is obtained. / This work was supported by the National Department of Higher Education, Science, Technology and Innovation of Ecuador (SENESCYT), and the Spanish ministry of Economy and European Union, grant DPI2016-81002-R (AEI/FEDER,UE). The author also received the grant for a predoctoral stay, Programa de Becas Iberoamérica- Santander Investigación 2018, of the Santander Bank. / Díaz Iza, HP. (2020). Value Function Estimation in Optimal Control via Takagi-Sugeno Models and Linear Programming [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/139135 / TESIS

No description available.

Master of Science / Department of Electrical and Computer Engineering / Caterina M. Scoglio / Disruption Tolerant Networks or opportunistic networks represent a class of networks where there is no contemporaneous path from source to destination. In other words, these are networks with intermittent connections. These networks are generally sparse or highly mobile wireless networks. Each node has a limited radio range and the connections between nodes may be disrupted due to node movement, hostile environments or power sleep schedules, etc. A common example of such networks is a sensor network monitoring nature or military field or a herd of animals under study. Epidemic routing is a widely proposed routing mechanism for data propagation in these type of networks. According to this mechanism, the source copies its packets to all the nodes it meets in its radio range. These nodes in turn copy the received packets to the other nodes they meet and so on. The data to be transmitted travels in a way analogous to the spread of an infection in a biological network. The destination finally receives the packet and measures are taken to eradicate the packet from the network. The task of routing in epidemic networks faces certain difficulties involving minimizing the delivery delay with a reduced consumption of resources. Every node has severe power constraints and the network is also susceptible to temporary but random failure of nodes. In the previous work, the parameter of mobility has been considered a constant for a certain setting. In our setting, we consider a varying parameter of mobility. In this framework, we determine the optimal mobility pattern and a forwarding policy that a network should follow in order to meet the trade-off between delivery delay and power consumption. In addition, the mobility pattern should be such that it can be practically incorporated. In our work, we formulate an optimization problem which is solved by using the principles of dynamic programming. We have tested the optimal algorithm through extensive simulations and they show that this optimization problem has a global solution.

The primary aim of this thesis is to develop effective solution techniques for large-scale maritime inventory routing problems that possess a core substructure common in many real-world applications. We use the term “large-scale” to refer to problems whose standard mixed-integer linear programming (MIP) formulations involve tens of thousands of binary decision variables and tens of thousands of constraints and require days to solve on a personal computer. Although a large body of literature already exists for problems combining vehicle routing and inventory control for road-based applications, relatively little work has been published in the realm of maritime logistics. A major contribution of this research is in the advancement of novel methods for tackling problems orders of magnitude larger than most of those considered in the literature. Coordinating the movement of massive vessels all around the globe to deliver large quantities of high value products is a challenging and important problem within the maritime transportation industry. After introducing a core maritime inventory routing model to aid decision-makers with their coordination efforts, we make three main contributions. First, we present a two-stage algorithm that exploits aggregation and decomposition to produce provably good solutions to complex instances with a 60-period (two-month) planning horizon. Not only is our solution approach different from previous methods discussed in the maritime transportation literature, but computational experience shows that our approach is promising. Second, building on the recent successes of approximate dynamic programming (ADP) for road-based applications, we present an ADP procedure to quickly generate good solutions to maritime inventory routing problems with a long planning horizon of up to 365 periods. For instances with many ports (customers) and many vessels, leading MIP solvers often require hours to produce good solutions even when the planning horizon is limited to 90 periods. Our approach requires minutes. Our algorithm operates by solving many small subproblems and, in so doing, collecting and learning information about how to produce better solutions. Our final research contribution is a polyhedral study of an optimization problem that was motivated by maritime inventory routing, but is applicable to a more general class of problems. Numerous planning models within the chemical, petroleum, and process industries involve coordinating the movement of raw materials in a distribution network so that they can be blended into final products. The uncapacitated fixed-charge transportation problem with blending (FCTPwB) that we study captures a core structure encountered in many of these environments. We model the FCTPwB as a mixed-integer linear program and derive two classes of facets, both exponential in size, for the convex hull of solutions for the problem with a single consumer and show that they can be separated in polynomial time. Finally, a computational study demonstrates that these classes of facets are effective in reducing the integrality gap and solution time for more general instances of the FCTPwB.

Our aim is to examine the problem of optimal asset allocation for investors exhibiting a behaviour in the face of uncertainty which is not consistent with the usual axioms of Expected Utility Theory. This thesis is divided into two main parts. In the first one, comprising Chapter II, we consider an arbitrage-free discrete-time financial model and an investor whose risk preferences are represented by a possibly nonconcave utility function (defined on the non-negative half-line only). Under straightforward conditions, we establish the existence of an optimal portfolio. As for Chapter III, it consists of the study of the optimal investment problem within a continuous-time and (essentially) complete market framework, where asset prices are modelled by semi-martingales. We deal with an investor who behaves in accordance with Kahneman and Tversky's Cumulative Prospect Theory, and we begin by analysing the well-posedness of the optimisation problem. In the case where the investor's utility function is not bounded above, we derive necessary conditions for well-posedness, which are related only to the behaviour of the distortion functions near the origin and to that of the utility function as wealth becomes arbitrarily large (both positive and negative). Next, we focus on an investor whose utility is bounded above. The problem's wellposedness is trivial, and a necessary condition for the existence of an optimal trading strategy is obtained. This condition requires that the investor's probability distortion function on losses does not tend to zero faster than a given rate, which is determined by the utility function. Provided that certain additional assumptions are satisfied, we show that this condition is indeed the borderline for attainability, in the sense that, for slower convergence of the distortion function, there does exist an optimal portfolio. Finally, we turn to the case of an investor with a piecewise power-like utility function and with power-like distortion functions. Easily verifiable necessary conditions for wellposedness are found to be sufficient as well, and the existence of an optimal strategy is demonstrated.