Recent developments in the area of reinforcement learning have yielded a number of new algorithms for the prediction and control of Markovian environments. These algorithms, including the TD(lambda) algorithm of Sutton (1988) and the Q-learning algorithm of Watkins (1989), can be motivated heuristically as approximations to dynamic programming (DP). In this paper we provide a rigorous proof of convergence of these DP-based learning algorithms by relating them to the powerful techniques of stochastic approximation theory via a new convergence theorem. The theorem establishes a general class of convergent algorithms to which both TD(lambda) and Q-learning belong.

Anders Lindquist, Sanjoy Mitter, Giorgio Picci. / Bibliography: leaves 14-15. / "October, 1981" "Feedback and Synthesis of Linear and Nonlinear Systems -Proceedings of the Workshop in Bielefeld, West-Germany, June 22-26, 1981, and Rome Italy, June 29-July 3, 1981." / "National Science Foundation Grant ECS-7903731" "Air Force Office of Scientific Research grant AFOSR 78-3519"

Thesis (Ph.D.)--University of Delaware, 2007. / Principal faculty advisor: Jeremy S. Edwards, Dept. of Chemical Engineering. Includes bibliographical references.

The current Thesis is devoted to comprehensive studies of comparison, or stochastic domination, theorems. It presents a combination of theoretical research and practical ideas formulated in several specific examples. Previously known results and their place it the theory of stochastic processes and stochastic differential equations is reviewed. This part of the work yielded three new theoretical results, formulated as theorems. Two of them are extensions of commonly used methods to more sophisticated processes and conditions. The third theorem is proven using previously not exploited technique. The place of all three results in the global theory is demonstrated by examining interconnections and possible distinctions between old and new theorems. Second and equally important part of the work focuses on more practical issues. Its main goal is to demonstrate where and how various theoretical findings can be applied to typical financial problems, such as option pricing, hedging, risk management and others. The example chapter summarizes the best of the obtained results in this direction. / Mathematical Finance

This research has three parts. Part 1: The Policy on Geometric Design of Highways and Street provides 5 methods of superelevation (e) distribution. Many states use methods 2 and 5 for low speed, urban and rural high-speed facilities. Method 5 aims to address speed variations; but is complicated, computationally intractable and may violate design consistency. Design recommendation by NCHRP439 accounts for speed variation, tractable; but is cumbersome along with irregular/step-wise design curves. New reliability based e distribution method is developed that addresses the speed variation; which is simple in determining and evaluating acceptable required e rates. At 95% level of reliability, the e rate obtained is lower than that from current practice resulting in cost savings. Part 2: Current practice/research does not address safety issue of the left-turn-bay at high degree of saturation (x). Left-Turn-Bay distance has three components: clearance, breaking to a stop and queue. The variation in the queue length reduces clearance and breaking distance resulting in unsafe breaking. Failure = clearance plus breaking distance < demand. The reliability of the left-turn-bay defined as the availability of the three components for left-turning vehicles to complete clearance and breaking maneuver safely; measured as increase in the deceleration rate over limit of 11.2ft/s2, safety index and probability of failure. Results show that at 95% reliability, current design practice fails when x exceeds 50%. Part 3: Current practice uses mean traffic volumes (Vd) as input for traffic signal control at roadway intersections. Variations in traffic flows affect the performance of intersection measured by the delay per vehicle traversing the intersection in seconds. Peak hour factor (PHF), the hourly volume divided by the peak 15-min flow rate within the peak hour is adopted by Highway Capacity Manual (HCM) to control surge. HCM suggests PHF design value of 0.92 for urban and 0.88 for rural areas. Fixed PHF may lead to increase in delay. Effects of variation of peak hour volumes on intersection signal delays are examined with large data. A new model is developed for PHF and Vd and used in signal timing to minimize intersection delay. The results show that the assumption of Poisson distribution for Vd is not reliable; delay reduction of 6.2 seconds per vehicle is achieved. Annual savings in travel time, fuel consumption and emissions cost is estimated in billions of dollars.

Thesis (Ph.D.)--University of Delaware, 2006. / Principal faculty advisor: Lian-Ping Wang, Dept. of Mechanical Engineering. Includes bibliographical references.

Abstract Micro-electromechanical systems (MEMS) are subject to inevitable and inherent uncertainties in their dimensional and material parameters. Those lead to variability in their performance and reliability. Manufacturing processes leave substantial variability in the shape and geometry of the device due to its small dimensions and high feature complexity, while the material properties of a component are inherently subject to scattering. The effects of these variations have to be considered and a modeling methodology is needed in order to ensure required MEMS performance under uncertainties. Furthermore, in the design of high-Q micro-resonators, dissipation mechanisms may have detrimental effects on the quality factor (Q). One of the major dissipation phenomena to consider is thermoelastic damping, so that performances are directly related to the thermoelastic quality factor, which has to be predicted accurately. The purpose of this research is to develop a numerical method to analyze the effects of geometric and material property random variations on the thermoelastic quality factor of micro-resonators. The extension of the Perturbation Stochastic Finite Element Method (PSFEM) to the analysis of strongly coupled multiphysic phenomena allows the quantification of the influence of uncertainties, making available a new efficient numerical tool to MEMS designers. Résumé Dans le domaine des microsystèmes électromécaniques (MEMS), les micro-résonateurs jouent un rôle important pour le développement de micro-capteurs de plus en plus précis (ex : micro-accéléromètres). Dans cette optique daugmentation de la précision, les pertes dénergie qui limitent les performances des micro-résonateurs doivent être identifiées et quantifiées. Le facteur limitant des micro-résonateurs actuels est leur facteur de qualité thermo-élastique, qui doit donc être prédit de manière précise. De plus, suite à la tendance actuelle de miniaturisation et complexification accrues des MEMS, les sources de dispersions sont très nombreuses, à la fois sur les constantes physiques des matériaux utilisés et sur les paramètres géométriques. La mise au point doutils numériques permettant de prendre en compte les incertitudes de manière efficace est donc primordiale afin daméliorer les prestations densemble du microsystème et dassurer un certain niveau de robustesse et de fiabilité. Le but de cette recherche est de développer une méthode numérique pour analyser les effets des variations aléatoires des propriétés matérielles et géométriques sur le facteur de qualité thermo-élastique de micro-résonateurs. Pour ce faire, lapproche dite perturbative de la méthode des éléments finis stochastiques (PSFEM) est étendue à lanalyse de phénomènes multiphysiques fortement couplés, fournissant ainsi aux acteurs de lindustrie des MEMS un nouvel outil de conception efficace.

In this paper we investigate pricing of variance swaps contracts. The literature is mostly dedicated to the pricing using replication with portfolio of vanilla options. In some papers the valuation with stochastic volatility models is discussed as well. Stochastic volatility is becoming more and more interesting to the investors. Therefore we decided to perform valuation with the Heston stochastic volatility model, as well as by using replication strategy. The thesis was done at SunGard Front Arena, so for testing the replica- tion strategy Front Arena software was used. For calibration and testing of the Heston model we used MatLab.

Adaptive modulation has been actively researched as a means to increase spectral efficiency of wireless communications systems. In general, analytic closed form models have been derived for the performance of the communications system as a function of the control parameters. However, in systems where general error correction coding is employed, it may be difficult to derive closed form performance functions of the communications systems. In addition, in closed form optimization, real time adaptation is not possible. Systems designed with deterministic state optimization are developed offline for a certain set of parameters and hardwired into mobile devices. In this thesis we present stochastic learning algorithms for adaptive modulation design. The algorithms presented allow for adaptive modulation system design in-dependent of error correction coding and modulation constellation requirements. In real time, the performance of the system is measured and stochastic approximation techniques are used to learn the optimal transmission parameters of the system. The technique is applied to Software Defined Radio (SDR) platforms, an emerging wireless technology which is currently being researched as a means of designing intelligent communications devices. The fundamental property that sets SDR apart from traditional radios is that the communications parameters are controlled in software, allowing for real-time control of physical layer communications. Our treatment begins by modeling the time evolution of the adaptive modulation process as a general state space Markov chain. We show the existence and uniqueness of the invariant measure and model performance functions as expectations with respect to the invariant measure. We consider constrained and unconstrained throughput optimization. We show that the cost functions considered are convex. Next we present stochastic approximation algorithms that are used to estimate the gradient of the cost function given only noisy estimates. We conclude by presenting simulation results obtained by the presented method. The learning based method is able to achieve the maximum throughput as dictated by exhaustive Monte Carlo simulation of the communications system, which provide an upper bound on performance. In addition, the learning algorithm is able to optimize communications under various error correction schemes. The tracking abilities of the algorithm are also demonstrated. We see that the proposed method is able to track optimal throughput settings as constraints are changed in time.

Integrated robust planning systems that cover all levels of SC hierarchy have become increasingly important. Strategic, tactical, and operational SC plans should not be generated in isolation to avoid infeasible and conflicting decisions. On the other hand, enterprise planning systems contain over millions of records that are processed in each planning iteration. In such enterprises, the ability to generate robust plans is vital to their success because such plans can save the enterprise resources that may otherwise have to be reserved for likely SC plan changes. A robust SC plan is valid in all circumstances and does not need many corrections in the case of interruption, error, or disturbance. Such a reliable plan is proactive as well as reactive. Proactivity can be achieved by forecasting the future events and taking them into account in planning. Reactivity is a matter of agility, the capability of keeping track of system behaviour and capturing alarming signals from its environment, and the ability to respond quickly to the occurrence of an unforeseen event. Modeling such a system behaviour and providing solutions after such an event is extremely important for a SC. This study focuses on integrated supply chain planning with stochastic dynamic considerations. An integrated tactical-operational model is developed and then segregated into two sub-models which are solved iteratively. A SC is a stochastic dynamic system whose state changes over time often in an unpredictable manner. As a result, the customer demand is treated as an uncertain parameter and is handled by exploiting scenario-based stochastic programming. The increase in the number of scenarios makes it difficult to obtain quick and good solutions. As such, a Branch and Fix algorithm is developed to segregate the stochastic model into isolated islands so as to make the computationally intractable problem solvable. However not all the practitioners, planners, and managers are risk neutral. Some of them may be concerned about the risky extreme scenarios. In view of this, the robust optimization approach is also adopted in this thesis. Both the solution robustness and model robustness are taken into account in the tactical model. Futhermore, the dynamic behaviour of a SC system is handled with the concept of Model Predictive Control (MPC).