In this thesis we investigate single and multi-player stochastic dynamic optimization prob-lems. We consider both discrete and continuous time processes. In the multi-player setup we investigate zero-sum games with both complete and partial information. We study partially observable stochastic games with average cost criterion and the state process be-ing discrete time controlled Markov chain. The idea involved in studying this problem is to replace the original unobservable state variable with a suitable completely observable state variable. We establish the existence of the value of the game and also obtain optimal strategies for both players. We also study a continuous time zero-sum stochastic game with complete observation. In this case the state is a pure jump Markov process. We investigate the nite horizon total cost criterion. We characterise the value function via appropriate Isaacs equations. This also yields optimal Markov strategies for both players. In the single player setup we investigate risk-sensitive control of continuous time Markov chains. We consider both nite and in nite horizon problems. For the nite horizon total cost problem and the in nite horizon discounted cost problem we characterise the value function as the unique solution of appropriate Hamilton Jacobi Bellman equations. We also derive optimal Markov controls in both the cases. For the in nite horizon average cost case we shown the existence of an optimal stationary control. we also give a value iteration scheme for computing the optimal control in the case of nite state and action spaces. Further we introduce a new class of stochastic processes which we call stochastic processes with \age-dependent transition rates". We give a rigorous construction of the process. We prove that under certain assunptions the process is Feller. We also compute the limiting probabilities for our process. We then study the controlled version of the above process. In this case we take the risk-neutral cost criterion. We solve the in nite horizon discounted cost problem and the average cost problem for this process. The crucial step in analysing these problems is to prove that the original control problem is equivalent to an appropriate semi-Markov decision problem. Then the value functions and optimal controls are characterised using this equivalence and the theory of semi-Markov decision processes (SMDP). The analysis of nite horizon problems becomes di erent from that of in nite horizon problems because of the fact that in this case the idea of converting into an equivalent SMDP does not seem to work. So we deal with the nite horizon total cost problem by showing that our problem is equivalent to another appropriately de ned discrete time Markov decision problem. This allows us to characterise the value function and to nd an optimal Markov control.

Several manufacturers of road vehicles today are working on developing autonomous vehicles. One subject that is often up for discussion when it comes to integrating autonomous road vehicles into the infrastructure is the safety aspect. There is in the context no common view of how safety should be quantified. As a contribution to this discussion we propose describing each potential hazardous event of a vehicle as a Semi-Markov Process (SMP). A reliability-based method for using the semi-Markov representation to calculate the probability of a hazardous event to occur is presented. The method simplifies the expression for the reliability using the Laplace-Stieltjes transform and calculates the transform of the reliability exactly. Numerical inversion algorithms are then applied to approximate the reliability up to a desired error tolerance. The method is validated using alternative techniques and is thereafter applied to a system for automated steering based on a real example from the industry. A desired evolution of the method is to involve a framework for how to represent each hazardous event as a SMP. / Flertalet tillverkare av vägfordon jobbar idag på att utveckla autonoma fordon. Ett ämne ofta på agendan i diskussionen om att integrera autonoma fordon på vägarna är säkerhet. Det finns i sammanhanget ingen klar bild över hur säkerhet ska kvantifieras. Som ett bidrag till denna diskussion föreslås här att beskriva varje potentiellt farlig situation av ett fordon som en Semi-Markov process (SMP). En metod presenteras för att via beräkning av funktionssäkerheten nyttja semi-Markov representationen för att beräkna sannolikheten för att en farlig situation ska uppstå. Metoden nyttjar Laplace-Stieltjes transformen för att förenkla uttrycket för funktionssäkerheten och beräknar transformen av funktionssäkerheten exakt. Numeriska algoritmer för den inversa transformen appliceras sedan för att beräkna funktionssäkerheten upp till en viss feltolerans. Metoden valideras genom alternativa tekniker och appliceras sedan på ett system för autonom styrning baserat på ett riktigt exempel från industrin. En fördelaktig utveckling av metoden som presenteras här skulle vara att involvera ett ramverk för hur varje potentiellt farlig situation ska representeras som en SMP.

在此篇論文中，我們提出一個慢性Ｂ型肝炎病毒感染病程之數學模型。因為在病症間的轉移機率(Transition probability)是隨著患者的年齡變動，所以在過去的文獻中，已經有學者提出，在疾病轉移機率模型中，應加入國民生命表(Life table)，藉此讓機率模型更符合Ｂ型肝炎病患的生命歷程。但是過去的文獻中，學者並沒有利用加入國民生命表之後疾病模型做進一步的病程分析。在這篇論文當中，我們假設原始的疾病轉移模型是符合馬可夫鏈的性質，並且提出一種加入國民生命表的方法，賦予疾病有年齡相關特性之模型。根據文獻數據和類馬可夫機率性質，我們使用著名的Chapman-Kolmogorov公式計算Ｂ型肝炎的自然病程機率，並畫出病人的生存機率曲線(Survival curve)。文章最後將會藉由兩個例子來介紹此篇論文提出的模型。實驗數據結果證實，此模型不僅提供了一個更精確的方法去分析在病症與死亡間的轉移機率、平均餘命(Life expectancy)、以及在不同年齡的存活機率(Survival probability)，並且可以更進一步的分析且瞭解病情狀態之間的轉移狀況。 / In this thesis, we propose a new mathematical model extending the natural history of hepatitis B virus (HBV) prognosis progression on chronic HBV infection. Since the actual transition probabilities between symptoms are dependent of ages, it has been proposed that the life table should be accommodated to the HBV prognosis progression model so that it can more properly explain the disease progression of the HBV patients. But in the literature, no further disease analysis and applications of it with the life table are discussed. In this thesis, we assume that the original disease progression is described by a Markov model, and propose a new method to combine the HBV progression with the life table so that the proposed model integrates data from the life table and allows the accommodation of age-dependent properties of the target disease. With clinical data based on annual incidence rates, the entire model is Semi-Markov based in nature. Computation methods similar to the celebrated Chapman-Kolmogorov equation can be applied to study the associated probability of each likely trajectory with desired initial ages and health states under the scenarios of natural history and various treatment policies. This method provides a more accurate way to analyze the transitions between symptoms, such as the mean life expectancy or the survival probabilities at different ages. We will give examples to demonstrate the proposed method in this thesis. Numerical results show the proposed model not only provides a more accurate method to analyze the mean life expectancy, the survival probabilities at different ages, and the transition probabilities from symptoms to death but also helps us to understand the transitions between symptoms.

The motion planning problem can be formulated as a Markov decision process (MDP), if the uncertainties in the robot motion and environments can be modeled probabilistically. The complexity of solving these MDPs grow exponentially as the dimension of the problem increases and hence, it is nearly impossible to solve the problem even without constraints. Using hierarchical methods, these MDPs can be transformed into a semi-Markov decision process (SMDP) which only needs to be solved at certain landmark states. In the deterministic robotics motion planning community, sampling based algorithms like probabilistic roadmaps (PRM) and rapidly exploring random trees (RRTs) have been successful in solving very high dimensional deterministic problem. However they are not robust to system with uncertainties in the system dynamics and hence, one of the primary objective of this work is to generalize PRM/RRT to solve motion planning with uncertainty. We first present generalizations of randomized sampling based algorithms PRM and RRT, to incorporate the process uncertainty, and obstacle location uncertainty, termed as "generalized PRM" (GPRM) and "generalized RRT" (GRRT). The controllers used at the lower level of these planners are feedback controllers which ensure convergence of trajectories while mitigating the effects of process uncertainty. The results indicate that the algorithms solve the motion planning problem for a single agent in continuous state/control spaces in the presence of process uncertainty, and constraints such as obstacles and other state/input constraints. Secondly, a novel adaptive sampling technique, termed as "adaptive GPRM" (AGPRM), is proposed for these generalized planners to increase the efficiency and overall success probability of these planners. It was implemented on high-dimensional robot n-link manipulators, with up to 8 links, i.e. in a 16-dimensional state-space. The results demonstrate the ability of the proposed algorithm to handle the motion planning problem for highly non-linear systems in very high-dimensional state space. Finally, a solution methodology, termed the "multi-agent AGPRM" (MAGPRM), is proposed to solve the multi-agent motion planning problem under uncertainty. The technique uses a existing solution technique to the multiple traveling salesman problem (MTSP) in conjunction with GPRM. For real-time implementation, an ?inter-agent collision detection and avoidance? module was designed which ensures that no two agents collide at any time-step. Algorithm was tested on teams of homogeneous and heterogeneous agents in cluttered obstacle space and the algorithm demonstrate the ability to handle such problems in continuous state/control spaces in presence of process uncertainty.

La maintenance prédictive joue un rôle important dans le maintien des systèmes de production continue car elle peut aider à réduire les interventions inutiles ainsi qu'à éviter des pannes imprévues. En effet, par rapport à la maintenance conditionnelle, la maintenance prédictive met en œuvre une étape supplémentaire, appelée le pronostic. Les opérations de maintenance sont planifiées sur la base de la prédiction des états de détérioration futurs et sur l'estimation de la vie résiduelle du système. Dans le cadre du projet européen FP7 SUPREME (Sustainable PREdictive Maintenance for manufacturing Equipment en Anglais), cette thèse se concentre sur le développement des modèles de détérioration stochastiques et sur des méthodes d'estimation de la vie résiduelle (Remaining Useful Life – RUL en anglais) associées pour les adapter aux cas d'application du projet. Plus précisément, les travaux présentés dans ce manuscrit sont divisés en deux parties principales. La première donne une étude détaillée des modèles de détérioration et des méthodes d'estimation de la RUL existant dans la littérature. En analysant leurs avantages et leurs inconvénients, une adaptation d’une approche de l'état de l'art est mise en œuvre sur des cas d'études issus du projet SUPREME et avec les données acquises à partir d’un banc d'essai développé pour le projet. Certains aspects pratiques de l’implémentation, à savoir la question de l'échange d'informations entre les partenaires du projet, sont également détaillées dans cette première partie. La deuxième partie est consacrée au développement de nouveaux modèles de détérioration et les méthodes d'estimation de la RUL qui permettent d'apporter des éléments de solutions aux problèmes de modélisation de détérioration et de prédiction de RUL soulevés dans le projet SUPREME. Plus précisément, pour surmonter le problème de la coexistence de plusieurs modes de détérioration, le concept des modèles « multi-branche » est proposé. Dans le cadre de cette thèse, deux catégories des modèles de type multi-branche sont présentées correspondant aux deux grands types de modélisation de l'état de santé des système, discret ou continu. Dans le cas discret, en se basant sur des modèles markoviens, deux modèles nommés Mb-HMM and Mb-HsMM (Multi-branch Hidden (semi-)Markov Model en anglais) sont présentés. Alors que dans le cas des états continus, les systèmes linéaires à sauts markoviens (JMLS) sont mis en œuvre. Pour chaque modèle, un cadre à deux phases est implémenté pour accomplir à la fois les tâches de diagnostic et de pronostic. A travers des simulations numériques, nous montrons que les modèles de type multi-branche peuvent donner des meilleures performances pour l'estimation de la RUL par rapport à celles obtenues par des modèles standards mais « mono-branche ». / Predictive maintenance plays a crucial role in maintaining continuous production systems since it can help to reduce unnecessary intervention actions and avoid unplanned breakdowns. Indeed, compared to the widely used condition-based maintenance (CBM), the predictive maintenance implements an additional prognostics stage. The maintenance actions are then planned based on the prediction of future deterioration states and residual life of the system. In the framework of the European FP7 project SUPREME (Sustainable PREdictive Maintenance for manufacturing Equipment), this thesis concentrates on the development of stochastic deterioration models and the associated remaining useful life (RUL) estimation methods in order to be adapted in the project application cases. Specifically, the thesis research work is divided in two main parts. The first one gives a comprehensive review of the deterioration models and RUL estimation methods existing in the literature. By analyzing their advantages and disadvantages, an adaption of the state of the art approaches is then implemented for the problem considered in the SUPREME project and for the data acquired from a project's test bench. Some practical implementation aspects, such as the issue of delivering the proper RUL information to the maintenance decision module are also detailed in this part. The second part is dedicated to the development of innovative contributions beyond the state-of-the-are in order to develop enhanced deterioration models and RUL estimation methods to solve original prognostics issues raised in the SUPREME project. Specifically, to overcome the co-existence problem of several deterioration modes, the concept of the "multi-branch" models is introduced. It refers to the deterioration models consisting of different branches in which each one represent a deterioration mode. In the framework of this thesis, two multi-branch model types are presented corresponding to the discrete and continuous cases of the systems' health state. In the discrete case, the so-called Multi-branch Hidden Markov Model (Mb-HMM) and the Multi-branch Hidden semi-Markov model (Mb-HsMM) are constructed based on the Markov and semi-Markov models. Concerning the continuous health state case, the Jump Markov Linear System (JMLS) is implemented. For each model, a two-phase framework is carried out for both the diagnostics and prognostics purposes. Through numerical simulations and a case study, we show that the multi-branch models can help to take into account the co-existence problem of multiple deterioration modes, and hence give better performances in RUL estimation compared to the ones obtained by standard "single branch" models.

In this thesis, nonlinearly perturbed stochastic models in discrete time are considered. We give algorithms for construction of asymptotic expansions with respect to the perturbation parameter for various quantities of interest. In particular, asymptotic expansions are given for solutions of renewal equations, quasi-stationary distributions for semi-Markov processes, and ruin probabilities for risk processes. / <p>At the time of the doctoral defense, the following papers were unpublished and had a status as follows: Paper 4: Manuscript. Paper 5: Manuscript. Paper 6: Manuscript.</p>

Cette thèse porte sur inférence statistique, les méthodes bootstrap et l’analyse multivariée dans le cadre des processus semi-markoviens. Les applications principales concernent un problème de la mécanique de la rupture. Ce travail a une contribution double. La première partie concerne la modélisation stochastique du phénomène de la propagation de fissure de fatigue. Une équation différentielle stochastique décrit le mécanisme de la dégradation et le caractère aléatoire inné du phénomène est traité par un processus de perturbation. Sous l'hypothèse que ce processus soit un processus markovien (ou semi-markovien) de saut, la fiabilité du modèle est étudiée en faisant usage de la théorie du renouvellement markovien et une nouvelle méthode, plus rapide, de calcul de fiabilité est proposée avec l'algorithme correspondant. La méthode et le modèle pour le processus markovien de perturbation sont validés sur des données expérimentales. Ensuite, la consistance forte des estimateurs des moindres carrés des paramètres du modèle est obtenue en supposant que les résidus du modèle stochastique de régression, dans lequel le modèle initial est transformé, soient des différences de martingales. Dans la deuxième partie de la thèse, nous avons abordé le problème difficile de l'approximation de la distribution limite de certains estimateurs non paramétriques des noyaux semi-markoviens ou certaines fonctionnelles via la méthode bootstrap pondérée dans un cadre général. Des applications de ces résultats sur des problèmes statistiques sont données pour la construction de bandes de confiance, les tests statistiques, le calcul de la valeur p du test et pour l’estimation des inverses généralisés. / The present dissertation is devoted to the statistical inference, bootstrap methods and multivariate analysis in the framework of semi-Markov processes. The main applications concern a mechanical problem from fracture mechanics. This work has a two-fold contribution. The first part concerns in general the stochastic modeling of the fatigue crack propagation phenomenon. A stochastic differential equation describes the degradation mechanism and the innate randomness of the phenomenon is handled by a perturbation process. Under the assumption that this process is a jump Markov (or semi-Markov) process, the reliability of the model is studied by means of Markov renewal theory and a new, faster, reliability calculus method is proposed with the respective algorithm. The method and the model for the Markov perturbation process are validated on experimental fatigue data. Next, the strong consistency of the least squares estimates of the model parameters is obtained by assuming that the residuals of the stochastic regression model are martingale differences into which the initial model function is transformed. In the second part of the manuscript, we have tackled the difficult problem of approximating the limiting distribution of certain non-parametric estimators of semi-Markov kernels or some functionals of them via the weighted bootstrap methodology in a general framework. Applications of these results on statistical problems such as the construction of confidence bands, the statistical tests, the computation of the p-value of the test are provided and the estimation of the generalized inverses.

Esta tesis reune algunas contribuciones al reconocimiento de formas estadístico y, más especícamente, a varias tareas del procesamiento del lenguaje natural. Varias técnicas estadísticas bien conocidas se revisan en esta tesis, a saber: estimación paramétrica, diseño de la función de pérdida y modelado estadístico. Estas técnicas se aplican a varias tareas del procesamiento del lenguajes natural tales como clasicación de documentos, modelado del lenguaje natural y traducción automática estadística. En relación con la estimación paramétrica, abordamos el problema del suavizado proponiendo una nueva técnica de estimación por máxima verosimilitud con dominio restringido (CDMLEa ). La técnica CDMLE evita la necesidad de la etapa de suavizado que propicia la pérdida de las propiedades del estimador máximo verosímil. Esta técnica se aplica a clasicación de documentos mediante el clasificador Naive Bayes. Más tarde, la técnica CDMLE se extiende a la estimación por máxima verosimilitud por leaving-one-out aplicandola al suavizado de modelos de lenguaje. Los resultados obtenidos en varias tareas de modelado del lenguaje natural, muestran una mejora en términos de perplejidad. En a la función de pérdida, se estudia cuidadosamente el diseño de funciones de pérdida diferentes a la 0-1. El estudio se centra en aquellas funciones de pérdida que reteniendo una complejidad de decodificación similar a la función 0-1, proporcionan una mayor flexibilidad. Analizamos y presentamos varias funciones de pérdida en varias tareas de traducción automática y con varios modelos de traducción. También, analizamos algunas reglas de traducción que destacan por causas prácticas tales como la regla de traducción directa; y, así mismo, profundizamos en la comprensión de los modelos log-lineares, que son de hecho, casos particulares de funciones de pérdida. Finalmente, se proponen varios modelos de traducción monótonos basados en técnicas de modelado estadístico . / Andrés Ferrer, J. (2010). Statistical approaches for natural language modelling and monotone statistical machine translation [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/7109

Health technology assessment: case study on breast carcinoma treatment in the Czech Republic Bc. Lenka Šlegerová January 4, 2019 Abstract This thesis proposes an original method for assessing total costs of med- ical treatment. It defines the semi-Markov model with four states that are associated with specific costs of the treatment, and not with patients' health statuses. This method is applied to individuals' treatment data drawn from the Czech clinical practice in the treatment of the metastatic HER2+ breast cancer. The aim is to assess the cost-effectiveness of adding medication per- tuzumab to the combination of trastuzumab+docetaxel within first-line therapy and to examine whether using individual data on Czech patients and the economic conditions leads to different results from foreign stud- ies. Furthermore, employing censored data from the clinical practice in the thesis complicates the estimation of patients' overall survival in compari- son to clinical-trials data that form random samples. Therefore, survival functions were not only estimated by the Kaplan-Meier estimator but also using the Cox proportional hazard model and the Accelerated failure time model that both control for the effects of included covariates. The addition of pertuzumab does not result in significantly longer pa- tients'...

本研究以確定給付型退休基金為對象，建構廣義隨機評價模型，以衡量不確定情況下退休基金之財務風險。希望藉著模型建構的過程，適切地描述基金評價過程中所應考量的各項要素。 為了強調基金評價時同時考量內外部精算假設的重要性，本研究將模型分為存活函數、經濟函數和給付函數三部份討論；存活函數利用離散時間非同質性半馬可夫過程（Discrete Time Non-Homogeneous semi-Markov Process）描述成員狀態轉移的機率，把成員工作年資、年齡和及狀態納入評價過程，有別於傳統僅以年齡為假設基礎之精算方法；經濟函數則以隨機過程表達外部環境的不確定性，結合上述假設資訊預估未來給付後，成為半馬可夫隨機精算評價模型，此一般性的模型能推展至基金評價時所需的各項流程。因此，本研究將模型應用於我國公務人員退撫基金，針對公務人員退撫基金的給付特性發展財務評價公式，完整地描述基金精算成本計算、未來人力與現金流量結構模擬以及敏感度分析等過程。 最後，本研究撰寫公務人員精算評價資訊系統，具體化半馬可夫隨機精算評價模型，實證公務人員退撫基金財務評價公式。實證結果也顯示，不論基金的性質或外部經濟環境，都將影響退休基金財務評價結果，為基金評價時不可忽略的精算假設。 / This study focuses on constructing a generalized valuation model for the defined benefit pension schemes. Financial soundness and funding stability are critical issues in pension fund management. In this study, a realistic stochastic model is built to monitor the uncertainty factors in affecting the financial risk and cash flow dynamics along the decision process. In order to evaluate the importance of the interior and exterior actuarial assumptions in pension valuation. Detailed models in describing the turnover patterns, economic uncertainties and benefit structures are explored. Semi-Markov process proposed by Dominicis, Manca and Granata (1991) and Janssen and Manca (1997) is extended in structuring the transition pattern of the plan’s population and the economic based factors are generated through stochastic processes. Modifications according to classification and movements of the plan member and the plan’s turnover pattern are employed to improve its practical usefulness. Then the actuarial valuations, cash flow analyses and workforce projection are performed and investigated. We has explicitly formulated the plan’s realistic phenomenon and implemented the proposed mechanism into a risk management framework for pension finance. By using this realistic approach, the cost factors could be monitored throughout the valuation. Typically these analyses involve substantial assumptions. This article has outlined the procedure of building the proposed model. Finally, Taiwan Public Employees Retirement System is simplified to illustrate the proposed methodology in pension valuation. The results from this study show that the structure of the pension schemes and the assumed economic factors are the significant factors in pension valuation. It also indicated that the fund manager can evaluate these impacts through the proposed model.