Stochastic dynamic programming is a recursive method for solving sequential or multistage decision problems. It helps economists and mathematicians construct and solve a huge variety of sequential decision making problems in stochastic cases. Research on stochastic dynamic programming is important and meaningful because stochastic dynamic programming reflects the behavior of the decision maker without risk aversion; i.e., decision making under uncertainty. In the solution process, it is extremely difficult to represent the existing or future state precisely since uncertainty is a state of having limited knowledge. Indeed, compared to the deterministic case, which is decision making under certainty, the stochastic case is more realistic and gives more accurate results because the majority of problems in reality inevitably have many unknown parameters. In addition, time scale calculus theory is applicable to any field in which a dynamic process can be described with discrete or continuous models. Many stochastic dynamic models are discrete or continuous, so the results of time scale calculus are directly applicable to them as well. The aim of this thesis is to introduce a general form of a stochastic dynamic sequence problem on complex discrete time domains and to find the optimal sequence which maximizes the sequence problem.

Thesis (M.S.)--University of North Carolina Wilmington, 2009. / Title from PDF title page (February 16, 2010) Includes bibliographical references (p. 30)

Thesis (M.S.)--University of North Carolina Wilmington, 2009. / Title from PDF title page (February 16, 2010) Includes bibliographical references (p. 34-35)

In this dissertation, certain problems of stochastic optimal control and relevant analysis of random vibrations are considered. Dynamic Programming approach is used to find an optimal control law for a linear single-degree-of-freedom system subjected to Gaussian white-noise excitation. To minimize a system's mean response energy, a bounded in magnitude control force is applied. This approach reduces the problem of finding the optimal control law to a problem of finding a solution to the Hamilton-Jacobi-Bellman (HJB) partial differential equation. A solution to this partial differential equation (PDE) is obtained by developed 'hybrid' solution method. The application of bounded in magnitude control law will always introduce a certain type of nonlinearity into the system's stochastic equation of motion. These systems may be analyzed by the Energy Balance method, which introduced and developed in this dissertation. Comparison of analytical results obtained by the Energy Balance method and by stochastic averaging method with numerical results is provided. The comparison of results indicates that the Energy Balance method is more accurate than the well-known stochastic averaging method.

The first part of this research program concerns the development of customized and easily implementable Probably Approximately Correct (PAC)-learning algorithms for episodic tasks over acyclic state spaces. The defining characteristic of our algorithms is that they take explicitly into consideration the acyclic structure of the underlying state space and the episodic nature of the considered learning task. The first of the above two attributes enables a very straightforward and efficient resolution of the ``exploration vs exploitation' dilemma, while the second provides a natural regenerating mechanism that is instrumental in the dynamics of our algorithms. Some additional characteristics that distinguish our algorithms from those developed in the past literature are (i) their direct nature, that eliminates the need of a complete specification of the underlying MDP model and reduces their execution to a very simple computation, and (ii) the unique emphasis that they place in the efficient implementation of the sampling process that is defined by their PAC property. More specifically, the aforementioned PAC-learning algorithms complete their learning task by implementing a systematic episodic sampling schedule on the underlying acyclic state space. This sampling schedule combined with the stochastic nature of the transitions taking place, define the need for efficient routing policies that will help the algorithms complete their exploration program while minimizing, in expectation, the number of executed episodes. The design of an optimal policy that will satisfy a specified pattern of arc visitation requirements in an acyclic stochastic graph, while minimizing the expected number of required episodes, is a challenging problem, even under the assumption that all the branching probabilities involved are known a priori. Hence, the sampling process that takes place in the proposed PAC-learning algorithms gives rise to a novel, very interesting stochastic control/scheduling problem, that is characterized as the problem of the Optimal Node Visitation (ONV) in acyclic stochastic digraphs. The second part of the work presented herein seeks the systematic modelling and analysis of the ONV problem. The last part of this research program explores the computational merits obtained by heuristical implementations that result from the integration of the ONV problem developments into the PAC-algorithms developed in the first part of this work. We study, through numerical experimentation, the relative performance of these resulting heuristical implementations in comparison to (i) the initial version of the PAC-learning algorithms, presented in the first part of the research program, and (ii) standard Q-learning algorithm variations provided in the RL literature. The work presented in this last part reinforces and confirms the driving assumption of this research, i.e., that one can design customized RL algorithms of enhanced performance if the underlying problem structure is taken into account.

Diss. (sammanfattning) Uppsala : Univ., 2003. / Härtill 6 uppsatser.

Thesis (Ph.D.)--University of Adelaide, Dept. of Electrical and Electronic Engineering, 2003. / "January 2003" Includes bibliographical references (leaves 120-129).

Orientador: João Bosco Ribeiro do Val / Dissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Eletrica e de Computação / Made available in DSpace on 2018-08-14T11:15:46Z (GMT). No. of bitstreams: 1 Pinheiro_NathalieCarvalho_M.pdf: 2872807 bytes, checksum: 929137a3cc76534c1832b041f3175ff3 (MD5) Previous issue date: 2009 / Resumo: Os elevados custos de produção na extração de petróleo marítimo são reduzidos com um sistema inovador que a Petrobras está desenvolvendo, denominado VASPS (do inglês, Vertical Annular Separation and Pumping System), capaz de separar gás e líquido ainda no assoalho oceânico. Esta dissertação trata do ajuste do nível de l'iquido no reservatório do VASPS, via controle impulsional estocástico, que se distancia bastante de um regulador convencional. A vazão de entrada flutua e o controle consiste em alterar a velocidade da bomba de saída do tanque, mantendo o nível numa faixa de operação, na qual ele varia livremente sem prejuízo. Todavia, é necessário de um lado observar o risco de operar próximo aos extremos de nível e de outro minimizar o número de intervenções na velocidade da bomba para prolongar sua vida útil. Optou-se por modelar o sistema por meio de um processo de difusão e formular por controle impulsional. Para a sua solução, o controle impulsional é convertido em uma sequência de problemas de parada ótima iterados, resolvidos utilizando-se a Discretização do Valor Médio, MVS (do inglês, Mean Value Scheme). Esta dissertação introduz o uso do controle impulsional nesta aplicação além da técnica citada para resolver problemas de parada ótima. / Abstract: The high costs in offshore oil production are reduced with the use of an innovative system which has been developed by Petrobras, the Brazilian oil company. Called VASPS (Vertical Annular Separation and Pumping System), it consists of an undersea gas/liquid separator. This work presents a strategy for the liquid level adjustment in the VASPS tank, which is subject to uncertain liquid inflow. This is far from a strict control regulator problem, since the liquid level may drifts freely inside an operation range. Although, in one hand, it is necessary to account for a risky operation near the limits, on the other hand, acting freely and continuously in the controlled pump may drastically shorten the lifetime of the equipment. To prevent premature worn with halt in the oil production, the control input variations should be meager. We propose a stochastic impulse control for varying the outflow pump speed. This formulation is transformed in a sequence of iterated optimal stopping problems, which results in a sequence of variational inequalities. We employ the numerical method called Mean Value Scheme (MVS) to solve this type of problem. This monograph introduces impulse control to the resevoir level adjustment of VASPS, together with the application of the MVS to its solution. / Mestrado / Automação / Mestre em Engenharia Elétrica

The world bond market is nearly twice as large as the equity market. The goal of this dissertation is to study the dynamics of bond price. Among the liquidity risk, interest rate risk and default risk, this dissertation will focus on the liquidity risk and trading strategy. Under the mathematical frame of stochastic control, we model price setting in U.S. bond markets where dealers have multiple instruments to smooth inventory imbalances. The difficulty in obtaining the optimal trading strategy is that the optimal strategy and value function depend on each other, and the corresponding HJB equation is nonlinear. To solve this problem, we derived an approximate optimal explicit trading strategy. The result shows that this trading strategy is better than the benchmark central symmetric trading strategy.

<p dir="ltr">The future of the global economy lies in space. As the economic and scientific benefits from space become more accessible and apparent to the public, the demand for more spacecrafts will only increase. However, simply using the current space architecture to sustain any major activities past low Earth orbit is infeasible. The limiting factor of relying on ground operators via the Deep Space Network will blunt future growth in cislunar space traffic as the bandwidth is insufficient to satisfy the needs of every spacecraft in this domain. For this reason, spacecrafts must begin to operate autonomously or semi-autonomously for operators to be able to manage more missions at a given time. This thesis focuses on the guidance and navigation policies that could help vehicles such as logistical or resupply spacecrafts perform their rendezvous autonomously. It is found that using GNSS signals and Moon-based optical navigation has the potential to help spacecrafts perform autonomous orbit determination in near-Moon trajectories. The estimations are high enough quality such that a stochastic controller can use this navigation solution to confidently guide the spacecraft to a target within a tolerance before proximity operations commence. As the reliance on the ground is shifted away, spacecrafts would be able to operate in greater numbers outside of Earth's lower orbits, greatly assisting humanity's presence in space. </p>