Markov processes have a long history of being used to model safety critical systems. However, with the development of autonomous vehicles and their increased complexity, Markov processes have been shown to not be sufficiently precise for reliability calculations. Therefore there has been the need to consider a more general stochastic process, namely the Semi-Markov process (SMP). SMPs allow for transitions with general distributions between different states and can be used to precisely model complex systems. This comes at the cost of increased complexity when calculating the reliability of systems. As such, methods to increase the interpretability of the system and allow for appropriate approximations have been considered and researched. In this thesis, a novel classification approach for transitions in SMP has been defined and complemented with different conjectures and properties. A transition is classified as good or bad by comparing the reliability of the original system with the reliability of any perturbed system, for which the studied transition is more likely to occur. Cases are presented to illustrate the use of this classification technique. Multiple suggestions and conjectures for future work are also presented and discussed. / Markovprocesser har länge använts för att modellera säkerhetskritiska system. Med utvecklingen av autonoma fordon och deras ökade komplexitet, har dock markovprocesser visat sig vara otillräckliga exakta för tillförlitlighetsberäkningar. Därför har det funnits ett behov för en mer allmän stokastisk process, nämligen semi-markovprocessen (SMP). SMP tillåter generella fördelningar mellan tillstånd och kan användas för att modellera komplexa system med hög noggrannhet. Detta innebär dock en ökad komplexitet vid beräkningen av systemens tillförlitlighet. Metoder för att öka systemets tolkningsbarhet och möjliggöra lämpliga approximationer har därför övervägts och undersökts. I den här masteruppsatsen har en ny klassificeringsmetod för övergångar i SMP definierats och kompletteras med olika antaganden och egenskaper. En övergång klassificeras som antingen bra eller dålig genom en jämförelse av tillförlitligheten i det ursprungliga systemets och ett ändrat system, där den studerade övergången har högre sannolikhet att inträffa. Fallstudier presenteras för att exemplifiera användningen av denna klassificeringsteknik. Flera förslag och antaganden för framtida arbete presenteras och diskuteras också.

No description available.

This thesis focuses on the problem of scalable optimization of dialogue behaviour in speech-based conversational systems using reinforcement learning. Most previous investigations in dialogue strategy learning have proposed flat reinforcement learning methods, which are more suitable for small-scale spoken dialogue systems. This research formulates the problem in terms of Semi-Markov Decision Processes (SMDPs), and proposes two hierarchical reinforcement learning methods to optimize sub-dialogues rather than full dialogues. The first method uses a hierarchy of SMDPs, where every SMDP ignores irrelevant state variables and actions in order to optimize a sub-dialogue. The second method extends the first one by constraining every SMDP in the hierarchy with prior expert knowledge. The latter method proposes a learning algorithm called 'HAM+HSMQ-Learning', which combines two existing algorithms in the literature of hierarchical reinforcement learning. Whilst the first method generates fully-learnt behaviour, the second one generates semi-learnt behaviour. In addition, this research proposes a heuristic dialogue simulation environment for automatic dialogue strategy learning. Experiments were performed on simulated and real environments based on a travel planning spoken dialogue system. Experimental results provided evidence to support the following claims: First, both methods scale well at the cost of near-optimal solutions, resulting in slightly longer dialogues than the optimal solutions. Second, dialogue strategies learnt with coherent user behaviour and conservative recognition error rates can outperform a reasonable hand-coded strategy. Third, semi-learnt dialogue behaviours are a better alternative (because of their higher overall performance) than hand-coded or fully-learnt dialogue behaviours. Last, hierarchical reinforcement learning dialogue agents are feasible and promising for the (semi) automatic design of adaptive behaviours in larger-scale spoken dialogue systems. This research makes the following contributions to spoken dialogue systems which learn their dialogue behaviour. First, the Semi-Markov Decision Process (SMDP) model was proposed to learn spoken dialogue strategies in a scalable way. Second, the concept of 'partially specified dialogue strategies' was proposed for integrating simultaneously hand-coded and learnt spoken dialogue behaviours into a single learning framework. Third, an evaluation with real users of hierarchical reinforcement learning dialogue agents was essential to validate their effectiveness in a realistic environment.

Le présent travail porte sur l’estimation d’un système en temps discret dont l’évolution est décrite par une chaîne semi-markovienne (CSM) d’espace d’état fini. Nous présentons le principe d’invariance sous forme multidimensionnelle pour le noyau semi-markovien (NSM), ainsi que diverses mesures du processus. Ensuite, nous étudions l’estimation non-paramétrique de la loi stationnaire de la CSM, en considérant deux estimateurs différents, et nous montrons qu’ils ont le même comportement asymptotique. La probabilité de la première entrée est également introduite. Nous proposons un estimateur et nous étudions ses propriétés asymptotiques : la convergence forte et la normalité asymptotique.D’autre part, nous nous concentrons sur l’étude de la fiabilité des systèmes semi-markoviens. Nous définissons la fiabilité sur intervalle d’un système dont la fiabilité et la disponibilité sont des cas particuliers et nous étudions les propriétés asymptotiques d’un estimateur proposé. De plus, nous présentons une comparaison de l’estimation des différentes mesures de fiabilité fondées sur deux estimateurs du NSM, en réalisant une trajectoire unique et des observations multiples indépendantes. Ce travail fournit aussi des résultats dans le cas semi-markovien à temps discret avec espace d’état général. Nous évaluons l’approximation de moyenne et de diffusion des chaînes de renouvellement markovien. Enfin, nous nous sommes aussi intéressés à une autre classe des processus pour laquelle nous obtenons des résultats dans le cadre des files d’attente. Nous étudions l’approximation de moyenne pour le modèle d’Engset en temps continu et nous appliquons ce résultat aux files d’attente avec ré-essais. / The present work concerns the estimation of a discrete-time system whose evolution is governed by a semi-Markov chain (SMC) with finitely many states. We present the invariance principle in a multidimensional form for the semi-Markov kernel (SMK) and some associated measures of the process. Afterwards, we study the nonparametric estimation of the stationary distribution of the SMC, considering two different estimators, and we prove that they hold the same asymptotic behavior. We introduce also the first hitting probability. We propose an estimator and study its asymptotic properties : the strong consistency and the asymptotic normality. On the other hand, we focus on the study of the dependability of semi-Markovsystems. We introduce the interval reliability whose special cases are the reliability and the availability measures and we study the asymptotic properties of a proposed estimator. Moreover, we present a comparison of nonparametric estimation for various reliability measures based on two estimators of the SMK, realizing a unique trajectory and multiple independent observations.Furthermore, this work provides results on the discrete-time semi-Markov case with general state space. We evaluate the average and diffusion approximation of Markov renewal chains. Finally, we are also interested in another class of processes for which we obtain results in the framework of queueing systems. We establish the average approximationfor the Engset model in continuous time and we apply this result to retrial queues.

In der Dissertation werden die Gesetze zur Reform des Arbeitsmarktes in Deutschland, besser bekannt als Hartz-Reformen, untersucht. Zunächst wird ein Überblick über die wichtigsten Änderungen aus den vier Reform-Paketen gegeben sowie die Effekte, welche man sich davon versprach. Des Weiteren werden zwei grundlegende Reformmaßnahmen, nämlich die Zusammenlegung der Arbeitslosen- und Sozialhilfe (Hartz IV) sowie die Verkürzung der Bezugsdauer der Arbeitslosenversicherungsleistung, analysiert, um deren Auswirkungen auf das individuelle Verhalten und die aggregierte Ökonomie zu evaluieren. Diese Untersuchung geschieht im Rahmen eines Matching-Modells mit optimaler verweildauerabhängiger Suchleistung. Mit Hilfe von Semi-Markov-Methoden, deren Anwendung in der Arbeitsmarkttheorie beschrieben wird, findet schließlich eine Aggregierung statt. Auf diese Weise können die Auswirkungen der Hartz-IV-Reformen auf die Verweildauer in Arbeitslosigkeit, die optimale Suchleistung und die Arbeitslosigkeit quantifiziert werden. / This thesis analyzes the 2001-2006 labor market reforms in Germany. The aim of this work is twofold. First, an overview of the most important reform measures and the intended effects is given. Second, two specific and very fundamental amendments, namely the merging of unemployment assistance and social benefits, as well as changes in the duration of unemployment insurance benefits, are analyzed in detail to evaluate their effects on individuals and the entire economy. Using a matching model with optimal search intensity and Semi-Markov methods, the effects of these two amendments on the duration of unemployment, optimal search intensity and unemployment are analyzed.

In dieser Arbeit werden zwei intrazelluläre Signalwege, betreffend den Tumorsuppressor p53 und das Signalmolekül Ca2+ , diskutiert und modelliert. Einzelzellmessungen des Tumorsuppressors p53 zeigen pulsatile Antwor- ten nach Zufügung von DNA Doppelstrangbrüchen (DSBs). Außer für sehr hohe Schadensdosen, ist das zeitliche auftreten dieser Pulse unregelmäßig. Mithilfe eines Wavelet basierten Pulsdetektors werden die einzelzell Trajek- torien untersucht und die inter-Puls Intervall (IPI) Verteilungen extrahiert. Diese weisen auf nicht-oszillatorische Regime in den Daten hin. Die Theorie der anregbaren Systeme angewendet auf regulatorische Netzwerke ermöglicht dieses komplexe Verhalten mathematisch zu beschreiben. Die Kopplung von Schadens-Sensor-Kinase Dynamik mit dem kanonischen p53 negativen feedback loop, ergibt ein anregbares p53 Modell. Detaillier- te Bifurkationsanalysen zeigen ein robustes anregbares Regime, welches durch ein starkes Schadenssignal auch in Oszillationen überführt werden kann. Treibt man das p53 Modell mit einem stochastischen DNA-Schadens-Prozess, kann sowohl das oszillatorische Verhalten nach hohem Schaden, als auch das unregelmäßige pulsatile Verhalten ohne äußere Stimulation reproduziert werden. Intrazelluläre Ca 2+ Spikes entstehen durch eine hierarchische Kaskade stochastischer prozesse. Die Anwendung einer semi-markovschen Beschreibung führt zu praktischen analytischen Lösungen des erstpassagezeiten Problems. Eine hierbei entdeckte Zeitskalenseparation ermöglicht ein neues allgemeines Ca2+ -Modell. Dieses erklärt auf äußerst prägnante Weise viele wesentliche experimentelle Ergebnisse, insbesondere die Momentenbeziehungen der inter-Spike Intervall Verteilungen. Schließlich erlaubt die hier vorgestellte Theorie Berechnungen der Stimulus-Enkodierung, also die Adaption des Ca 2+ Signals auf veränderliche extrazelluläre Stimuli. Die Vorhersage einer fold change Enkodierung kann durch Experimente gestützt werden. / In this work two signaling pathways, involving the tumor suppressor p53 and the second messenger Ca2+ , are to be discussed and modelled. The tumor suppressor p53 shows a pulsatile response in single cells after induction of DNA double strand breaks (DSBs). Except for very high amounts of damage, these pulses appear at irregular times. The concept of excitable systems is employed as a convenient way to model such observed dynamics. An application to biomolecular reaction networks shows the need for a positive feedback within the p53 regulatory network. Exploiting the reported ultrasensitive dynamics of the upstream damage sensor kinases, leads to a simplified excitable kinase-phosphatase model. Coupling that to the canonical negative feedback p53 regulatory loop, is the core idea behind the construction of the excitable p53 model. A detailed bifurcation analysis of the model establishes a robust excitable regime, which can be switched to oscillatory dynamics via a strong DNA damage signal. Driving the p53 model with a stochastic DSB process yields pulsatile dynamics which reflect different experimental scenarios. Intracellular Ca 2+ concentration spikes arise from a hierarchic cascade of stochastic events. An analytical solution strategy, employing a semi-Markovian description and involving Laplace transformations, is devised and successfully applied to a specific Ca2+ model. The new gained insights are then used, to construct a new generic Ca2+ model, which elegantly captures many known features of Ca2+ signaling. In particular the experimentally observed relations between the average and the standard deviation of the inter spike intervals (ISIs) can be explained in a concise way. Finally, the theoretical considerations allow to calculate the stimulus encoding relation, which governs the adaption of the Ca 2+ signals to varying extracellular stimuli. This is predicted to be a fold change response and new experimental results display a strong support of this idea.

La Dominance Temporelle des Sensations (DTS) est une méthode d’analyse sensorielle qui mesure la perception temporelle d’un produit au cours de sa dégustation. Pour un panéliste, la DTS consiste à choisir parmi une liste de descripteurs lequel est dominant à chaque instant. Ce travail a pour but la modélisation des données DTS à l’aide de processus stochastiques et propose d’utiliser les processus semi-markoviens (PSM), une généralisation des chaînes de Markov qui permet de modéliser librement les durées de dominance. Le modèle obtenu peut être utilisé pour comparer des échantillons DTS en réalisant un rapport de vraisemblance. Étant donné que les probabilités de transition entre les descripteurs peuvent dépendre du temps, nous proposons d’utiliser des modèles différents par période et nous proposons un algorithme pour déterminer le nombre et les frontières de ces périodes de manière optimale. Le modèle est représenté sous forme d’un graphe montrant les transitions entre descripteurs les plus observées. Finalement, ce travail introduit les modèles de mélange de processus semi-markoviens afin de segmenter le panel en fonction des différences de perception interindividuelles.Les méthodes développées sont appliquées à des jeux de données DTS variés : chocolats, fromages frais et Goudas. Les résultats montrent que la modélisation par un PSM apporte de nouvelles informations sur la perception temporelle, en particulier sur la variabilité de perception au sein d’un panel, alors que les méthodes classiques se focalisent sur une vision moyenne de la perception du panel. De plus, à notre connaissance, ce travail est le premier à proposer l’identification d’un modèle de mélange de processus semi-markoviens. / Temporal Dominance of Sensations (TDS) is a technique to measure temporal perception of food product during tasting. For a panelist, it consists in choosing in a list of attributes which one is dominant at any time. This work aims to model TDS data with a stochastic process and proposes to use semi-Markov processes (SMP), a generalization of Markov chains which allows dominance durations to be modeled by any type of distribution. The model can then be used to compare TDS samples based on likelihood ratio. Because probabilities of transition from one attribute to another one can also depend on time, we propose to model TDS by period and we propose a method to select optimally the number of periods and the frontiers between periods. Graphs built upon the stochastic pattern can be plotted to represent main chronological transitions between attributes. Finally, this work introduces new statistical models based on finite mixtures of semi-Markov processes in order to derive consumer segmentation based on individual differences in temporal perception of a product.The methods are applied to various TDS datasets: chocolates, fresh cheeses and Gouda cheeses. Results show that SMP modeling gives new information about temporal perception compared to classical methods. It particularly emphasizes the existence of several perceptions for a same product in a panel, whereas classical methods only provide a mean panel overview. Furthermore, as far as we know, this work is the first one that considers mixtures of semi-Markov processes.

The document is my habilitation thesis, which is a prerequisite for obtaining the "habilitation à diriger des recherche (HDR)" in France (https://fr.wikipedia.org/wiki/Habilitation_universitaire#En_France). The thesis is of cumulative form, thus providing an overview of my published works until summer 2013.

L’objectif de cette thèse est de construire un framework qui représente les incertitudes aléatoires et épistémiques basé sur les approches probabilistes et des théories d’incertain, de comparer les méthodes et de trouver les propres applications sur les grands systèmes avec événement rares. Dans la thèse, une méthode de normalité asymptotique a été proposée avec simulation de Monte Carlo dans les cas binaires ainsi qu'un modèle semi-Markovien dans les cas de systèmes multi-états dynamiques. On a aussi appliqué la théorie d’ensemble aléatoire comme un modèle de base afin d’évaluer la fiabilité et les autres indicateurs de performance dans les systèmes binaires et multi-états avec technique bootstrap. / Our research objective is to build frameworks representing both aleatory and epistemic uncertainties based on probabilistic approach and uncertainty approaches and to compare these methods and find the proper applicatin for these methods in large scale systems with rare event. In this thesis, an asymptotic normality method is proposed with Monte Carlo simulation in case of binary systems as well as semi-Markov model for cases of dynamic multistate system. We also apply random set as a basic model to evaluate system reliability and other performance indices on binary and multistate systems with bootstrap technique.

Classical value estimation reinforcement learning algorithms do not perform very well in dynamic environments. On the other hand, the reinforcement learning of animals is quite flexible: they can adapt to dynamic environments very quickly and deal with noisy inputs very effectively. One feature that may contribute to animals' good performance in dynamic environments is that they learn and perceive the time to reward. In this research, we attempt to learn and perceive the time to reward and explore situations where the learned time information can be used to improve the performance of the learning agent in dynamic environments. The type of dynamic environments that we are interested in is that type of switching environment which stays the same for a long time, then changes abruptly, and then holds for a long time before another change. The type of dynamics that we mainly focus on is the time to reward, though we also extend the ideas to learning and perceiving other criteria of optimality, e.g. the discounted return, so that they can still work even when the amount of reward may also change. Specifically, both the mean and variance of the time to reward are learned and then used to detect changes in the environment and to decide whether the agent should give up a suboptimal action. When a change in the environment is detected, the learning agent responds specifically to the change in order to recover quickly from it. When it is found that the current action is still worse than the optimal one, the agent gives up this time's exploration of the action and then remakes its decision in order to avoid longer than necessary exploration. The results of our experiments using two real-world problems show that they have effectively sped up learning, reduced the time taken to recover from environmental changes, and improved the performance of the agent after the learning converges in most of the test cases compared with classical value estimation reinforcement learning algorithms. In addition, we have successfully used spiking neurons to implement various phenomena of classical conditioning, the simplest form of animal reinforcement learning in dynamic environments, and also pointed out a possible implementation of instrumental conditioning and general reinforcement learning using similar models.