Algorithmically designed reward functions can influence groups of learning agents toward measurable desired sequential joint behaviours. Influencing learning agents toward desirable behaviours is non-trivial due to the difficulties of assigning credit for global success to the deserving agents and of inducing coordination. Quantifying joint behaviours lets us identify global success by ranking some behaviours as more desirable than others. We propose a real-valued metric for turn-taking, demonstrating how to measure one sequential joint behaviour. We describe how to identify the presence of turn-taking in simulation results and we calculate the quantity of turn-taking that could be observed between independent random agents. We demonstrate our turn-taking metric by reinterpreting previous work on turn-taking in emergent communication and by analysing a recorded human conversation. Given a metric, we can explore the space of reward functions and identify those reward functions that result in global success in groups of learning agents. We describe 'medium access games' as a model for human and machine communication and we present simulation results for an extensive range of reward functions for pairs of Q-learning agents. We use the Nash equilibria of medium access games to develop predictors for determining which reward functions result in turn-taking. Having demonstrated the predictive power of Nash equilibria for turn-taking in medium access games, we focus on synthesis of reward functions for stochastic games that result in arbitrary desirable Nash equilibria. Our method constructs a reward function such that a particular joint behaviour is the unique Nash equilibrium of a stochastic game, provided that such a reward function exists. This method builds on techniques for designing rewards for Markov decision processes and for normal form games. We explain our reward design methods in detail and formally prove that they are correct.

The performance modelling of large-scale systems using discrete-state approaches is fundamentally hampered by the well-known problem of state-space explosion, which causes exponential growth of the reachable state space as a function of the number of the components which constitute the model. Because they are mapped onto continuous-time Markov chains (CTMCs), models described in the stochastic process algebra PEPA are no exception. This thesis presents a deterministic continuous-state semantics of PEPA which employs ordinary differential equations (ODEs) as the underlying mathematics for the performance evaluation. This is suitable for models consisting of large numbers of replicated components, as the ODE problem size is insensitive to the actual population levels of the system under study. Furthermore, the ODE is given an interpretation as the fluid limit of a properly defined CTMC model when the initial population levels go to infinity. This framework allows the use of existing results which give error bounds to assess the quality of the differential approximation. The computation of performance indices such as throughput, utilisation, and average response time are interpreted deterministically as functions of the ODE solution and are related to corresponding reward structures in the Markovian setting. The differential interpretation of PEPA provides a framework that is conceptually analogous to established approximation methods in queueing networks based on meanvalue analysis, as both approaches aim at reducing the computational cost of the analysis by providing estimates for the expected values of the performance metrics of interest. The relationship between these two techniques is examined in more detail in a comparison between PEPA and the Layered Queueing Network (LQN) model. General patterns of translation of LQN elements into corresponding PEPA components are applied to a substantial case study of a distributed computer system. This model is analysed using stochastic simulation to gauge the soundness of the translation. Furthermore, it is subjected to a series of numerical tests to compare execution runtimes and accuracy of the PEPA differential analysis against the LQN mean-value approximation method. Finally, this thesis discusses the major elements concerning the development of a software toolkit, the PEPA Eclipse Plug-in, which offers a comprehensive modelling environment for PEPA, including modules for static analysis, explicit state-space exploration, numerical solution of the steady-state equilibrium of the Markov chain, stochastic simulation, the differential analysis approach herein presented, and a graphical framework for model editing and visualisation of performance evaluation results.

Dans un procédé de séchage, l'évacuation de l'eau induit des modifications structurelles et physicochimiques qui engendrent des difficultés d'écoulement dans le séchoir. Cette étude a porté sur la modélisation du séchage continu par contact avec agitation. Nous avons développé un modèle markovien pour décrire les transferts de masse, de chaleur et l'écoulement dans un séchoir continu. Les paramètres d'entrée du modèle ont été déterminés expérimentalement sur un produit pâteux, tandis que d'autres paramètres ont été obtenus à partir de la littérature scientifique. L'étude de sensibilité de la teneur en eau et de la masse sèche retenue dans le séchoir a permis d'analyser et de hiérarchiser les effets des différentes variables opératoires. Enfin, nous avons étudié l'influence des variables opératoires sur le profil de teneur en eau, le temps de passage et les courbes de distribution de temps de séjours simulées. L'évolution structurelle du produit a été mise en évidence dans le modèle. / In a drying process, the evacuation of water induced structural and physico-chemical modifications that generate difficulties of flow in the dryer. This study focused on the modeling of agitated indirect continuous dryer. We developed a Markov model to describe the mass transfer, heat transfer and flow in a continuous dryer. The input parameters of the model were determined experimentally on a pasty product, while other parameters were obtained from the scientific literature. The sensitivity study of the moisture content and dry mass retained in the dryer was used to analyze and treat on a hierarchical basis the effects of different operating variables. Finally, we studied the influence of variables operating on the profile of moisture content, the passage time and the simulated curves of residence time distribution. The structural evolution of the product was considered in the model

Áreas como política, economia e marketing sofrem grandes influências no que diz respeito à difusão de informação. Por este motivo, diversos ramos da ciência tem estudado tais fenômenos a fim de simulá-los e compreendê-los por meio de modelos matemáticos e/ou estocásticos. Em virtude disto, este trabalho de doutorado tem como objetivo generalizar modelos de difusão de inovação já existentes na literatura. O primeiro modelo utiliza o mecanismo de social reinforcement para difusão de inovação e o qual foi construído para o grafo completo. Neste caso, consideramos uma população finita, fechada, totalmente misturada e subdividida em quatro classes de indivíduos denominados ignorantes, conscientes, adotadores e abandonadores da inovação. Assim, será apresentado uma Lei Fraca dos Grandes Números e um Teorema Central do Limite para a proporção final da população que nunca escutou sobre a inovação e aqueles que já conhecem sobre ela mas ainda não adotaram. Ademais, também será apresentado um resultado de convergência para o máximo de adotadores em um intervalo estocástico, assim como o instante de tempo em que o processo atinge esse estado. Para esse estudo, foram utilizados resultados da teoria de cadeias de Markov dependentes da densidade. Ademais, formulamos um modelo estocástico com estrutura de estágios para descrever o fenômeno da difusão de inovação em uma população estruturada. Mais precisamente, propomos uma cadeia de Markov a tempo contínuo definida na rede hipercúbica d-dimensional. Cada indivíduo da população deve estar em algum dos M+1 estados pertencentes ao conjunto {0;1;2; ::;M}. Nesse sentido, 0 representa um ignorante, i para i ∈ {1; :::;M - 1} um consciente no estágio i e M um adotador. Dessa forma, são estudados argumentos que permitem encontrar condições suficientes nas quais a inovação se espalha ou não com probabilidade positiva. / Areas such as politics, economics and marketing are heavily influential in terms of information diffusion. For this reason, several branches of science have studied such phenomena in order to simulate and understand them by mathematical and/or stochastic models. In this context, this phd project aims to generalize innovation diffusion models that there is in the literature. The first model uses the social reinforcement mechanism for diffusion of innovation and which was built for the complete graph. In this case, we consider a finite population, closed, totally mixed and subdivided into four classes of individuals called ignorants, aware, adopters and abandoner of innovation. We prove a Law of Large Numbers and a Central Limit Theorem for the proportion of the population who have never heard about the innovation and those who know about ir but they have not adopted it yet. In addition, we also obtain result for the convergence of the maximum of adopter in a stochastic interval, as well as the instant of time that the process reaches that state. For this study, we used results of the theory of density dependent Markov chains. Furthermore, we formulated a stochastic model with structure stages to describe the phenomenon of innovation diffusion in a structured population. More precisely, we proposed a continuous time Markov chain defined in a population represented by the d-dimensional integer lattice. Each individual of the population must be in some of the M +1 states belonging to the set {0;1;2; :::;M}. In this sense, 0 stands for ignorant, i for i ∈ {1; :::;M - 1} for aware in stage i and M for adopter. The arguments, that allow to obtain sufficient conditions under which the innovation either becomes extinct or survives with positive probability, are studied.

Sistemas Lineares com Saltos Markovianos (SLSMs) são estudados desde a década de 1960 e vêm ganhando visibilidade desde então, com diversas aplicações dentre as quais Finanças, Robótica e Engenharias diversas. Um problema de regulação trata de controlar o SLSM buscando fazer sua trajetória se aproximar de zero. Quando os saltos markovianos são observados, o problema é simples e bem resolvido, muito diferente de quando não se observam os saltos. Neste trabalho é estudado um algoritmo da literatura utilizado para resolver o problema de regulação sem observação dos saltos, chamado Método Variacional (MV). Sendo um dos melhores métodos para o dado problema, enfrenta dificuldades de cunho numérico. Neste trabalho se procura analisar e melhorar o condicionamento dos subproblemas envolvidos, de forma a favorecer a convergência do método. São testadas abordagens diferentes usando precondicionadores e comparados os resultados, permitindo concluir que três das cinco abordagens é que trouxeram os melhores resultados. Por se tratar de sistemas lineares do tipo Ax = b, as abordagens de condicionamento podem ser adaptadas para outros problemas semelhantes. / Markov Jump Linear Systems (MJLSs) have been studied since the decade of 1960 and they are gaining visibility ever since, due to a wide range of applications, such as Finance, Robotics, several Engeneerings among others. The so called regulation problem is to control the MJLS seeking to make its trajectory to approach zero. When markovian jumps are observed, the problem is simple and the solution given as closed formulas, which is quite different from the situation when jumps are not observed. We study an algorithm available in literature called Variational Method (VM). Even though it is one of the best methods to solve the problem, it has some numerical difficulties. We analyse its performance and propose some ideas aiming at the ill-conditioning of the subproblems involved, in order to improve the convergence of the method. Different approaches are tested using preconditioners and the results are compared, indicating that three approaches of the five tested ones are promising for convergence improvement. Because the subproblems are linear systems of type Ax = b, these approaches can be adapted to similar problems.

Este trabalho propõe uma metodologia de identificação para sistemas lineares sujeitos a saltos markovianos. Dada uma sequência de observações ruidosas da variável de estados, busca-se estimá-la juntamente com os parâmetros (desconhecidos) que descrevem o sistema dinâmico no espaço de estados. Como é bem conhecido, a ltragem ótima nesta classe de sistemas tem requisitos computacionais exponencialmente crescentes em função do tamanho da amostra, e torna-se inviável na prática. Recorre-se, portanto, a um algoritmo sub-ótimo de ltragem, cujos resultados são utilizados na identificação por máxima verossimilhança segundo a metodologia apresentada. Simulações realizadas mostram boa convergência. / This paper proposes a methodology for the identification of Markov-jump linear systems. Given a sequence of noisy observations of the state variable, our objective is to estimate it along with the (unknown) parameters that drive the system in the state-space. As it is well known, the optimal ltering in this class of systems requires exponentially increasing computing power, in proportion to the sample size, and is not feasible in practice. We resort, therefore, to a sub-optimal algorithm, whose results are used for a maximum likelihood identification according to the methodology presented here. Simulations show a good convergence.

Nous avons proposé dans cette thèse une nouvelle approche de modélisation des réponses cellulaire et tumorale durant la radiothérapie. Cette modélisation est fondée sur les chaînes de Markov. Elle se situe dans le cadre de la théorie de cible qui suppose qu'il existe dans la cellule des régions sensibles appelées cibles, qui doivent toutes être désactivées pour tuer la cellule. Un premier travail est consisté à proposer un modèle à temps discret en tenant compte non seulement des phases de réparations cellulaires entre les fractions de dose mais également de l'hétérogénéité des dommages cellulaires.Nous avons ensuite proposé un modèle stochastique de la durée de vie cellulaire. Cette modélisation fut également étendue à une population de cellules et a permis d'établir de nouvelles expressions des probabilités d'efficacité et de complication thérapeutique. Nos derniers travaux portent sur le développement d'un modèle de type chaîne de Markov à temps continu qui pourrait être appliqué aux réponses des tumeurs traitées par la thérapie photodynamique / In this thesis, we have proposed a Markov chain modeling of the cell and tumor behaviors during radiotherapy. Our approach is based on the target theory where it is assumed that the cell contains a number of sensitive sites called targets which must be all deactivated to produce the cell death. A first task was to provide a discrete-time model taking cell repair between dose fractions and the heterogeneity of cellular damage into account. Then, we proposed a stochastic model of the cell lifespan. This model was also extended to a cell population and allowed to derive new expressions of the efficiency and complication probabilities of the treatment. Finally, we focused on the development of a model based on Markov chain in continuous time which could be applied to the responses of tumors treated by photodynamic therapy

Les études de sûreté de fonctionnement (SdF) sont en général basées sur l’hypothèse d’indépendance des événements de défaillance et de réparation ainsi que sur l’analyse des coupes qui décrivent les sous-ensembles de composants entraînant la défaillance du système. Dans le cas des systèmes dynamiques pour lesquels l’ordre d’occurrence des événements a une incidence directe sur le comportement dysfonctionnel du système, il est important de privilégier l’utilisation de séquences d’événements permettant une évaluation des indicateurs de SdF plus précise que les coupes. Ainsi, nous avons proposé, dans une première partie de nos travaux, un cadre formel permettant la détermination des séquences d’événements qui décrivent l’évolution du système ainsi que leur évaluation quantitative, en recourant à la théorie de langages probabilistes et à la théorie des processus markoviens/semi-markoviens. L'évaluation quantitative des séquences intègrent le calcul de leur probabilité d'occurrence ainsi que leur criticité (coût et longueur des séquences). Pour l’évaluation des séquences décrivant l’évolution des systèmes complexes présentant plusieurs modes de fonctionnement ou de défaillance, une approche modulaire basée sur les opérateurs de composition (choix et concaténation) a été proposée. Celle-ci consiste à calculer la probabilité d'une séquence d'événements globale à partir d'évaluations réalisées localement, mode par mode. Les différentes contributions sont appliquées sur deux cas d'étude de taille et complexité croissante. / Dependability studies are often based on the assumption of events (failures and repairs) independence but also on the analyse of cut-set which describes the subsets of components causing a system failure. In the case of dynamic systems where the events occurrence order has a direct impact on the dysfunctional behaviour, it is important to promote using event sequences instead of cut-sets for dependability assessment. In the first part, a formal framework is proposed. It helps in determining sequences of events that describe the evolution of the system and their assessment, using the theory of probabilistic languages and the theory of Markov/semi-Markov processes. The assessment integrates the calculation of the probability occurrence of the event sequences and their criticality (cost and length). For the assessment of complex systems with multiple operating/failure modes, a modular approach based on composition operators (choice and concatenation) is proposed. Evaluation of the probability of a global sequence of events is performed from local Markov/semi-Markov models for each mode of the system. The different contributions are applied on two case studies with a growing complexity.

A hipertensão arterial sistêmica (HAS) é um fator de risco para 9,4 milhões de mortes em todo o mundo. No Brasil, a HAS atribui um custo anual de aproximadamente R$ 969,231,436,00 ao Sistema Único de Saúde (SUS). Alguns estudos têm mostrado uma grande efetividade da Atenção Farmacêutica (ATF) para melhorar os parâmetros clínicos de pacientes com HAS. Diante desse contexto, este trabalho teve o objetivo de desenvolver uma ferramenta econômica considerando custos e desfechos para auxiliar os gestores de saúde a implantar a ATF no SUS e reduzir a morbi-mortalidade ocasionada pela HAS. Para isso, foi conduzido um estudo farmacoeconômico de modelagem com análises de custo-efetividade, custo-benefício e custo-utilidade. Em duas unidades básicas do SUS, em Ribeirão Preto, 104 pacientes foram acompanhados por um programa de ATF durante o ano de 2009. Foram coletados dados clínicos, assistenciais e econômicos referentes aos anos de 2006 a 2012. Os dados foram divididos por períodos, 2006-2008 (pré ATF), 2009 (ATF) e 2010-2012 (pós ATF). A análise econômica foi realizada em cinco etapas: análise direta dos custos, análise indireta dos custos com modelagem de Markov, análise de custo-benefício com obtenção do valor presente líquido (VPL), análise de sensibilidade dos custos, e análise do impacto epidemiológico. O controle pressórico foi alcançado em 54,4%, 98,2% e 93,0% dos pacientes nos períodos pré ATF, ATF e pós ATF, respectivamente. Na análise direta dos custos houve diferença dos custos totais de -R$307,23 e R$4.053,32 e a razão de custo efetividade incremental foi R$1.124,24 e R$100,98, nos períodos ATF e pós ATF, respectivamente. Na projeção por modelagem de Markov houve 46 complicações da HAS e 17 mortes; e 15 complicações e 5 mortes ao longo de dez anos para a assistência a saúde sem a ATF e com a ATF, respectivamente. Houve o custo por paciente de R$ 17,09 / anos de vida ajustados por qualidade e R$ 28,98 por anos de vida ganhos após a alta da ATF. O VPL obtido com o investimento nesta prática farmacêutica foi R$ 3.791.111,13 e a razão benefício - custo (RBC) foi 30,03 (26,74 - 34,28), RBC > 1 com taxa de retorno de investimento de 303%. Conclui-se que a ATF foi capaz de melhorar o controle pressórico e os níveis de colesterol total reduzindo o risco cardiovascular, o que ocasionou a redução de gastos com a saúde e o impacto positivo epidemiológico. Essa análise foi suficiente para fundamentar uma ferramenta farmacoeconômica capaz de auxiliar os gestores na tomada de decisão quanto à implementação da ATF para a redução da morbi-mortalidade ocasionada pela HAS. / Systemic Arterial Hypertension (SAH) is a risk factor for 9.4 million deaths worldwide. In Brazil, SAH assigns an annual cost of approximately R$969,231,436,00 the Public Health System (PHS). Some studies have shown a great effectiveness of pharmaceutical care (PC) to improve clinical parameters of patients with SAH. In this context, this study aimed to develop an economic tool considering costs and outcomes to help health managers to deploy the PC in the PHS and reduce the morbidity and mortality caused by hypertension. For this, a pharmacoeconomic modeling study of cost-effectiveness analysis, cost-benefit and cost-utility was conducted. Into two basic units of PHS, in Ribeirão Preto, 104 patients were followed for an PC program during 2009. Clinical, healthcare and economic data were collected for the years 2006 to 2012. The data were divided into periods, 2006-2008 pre PC, 2009 PC and 2010-2012 post-PC. The economic analysis was carried out in five stages: Direct cost analysis; indirect cost analysis with Markov modeling; Cost-benefit analysis to obtain the net present value (NPV); Sensitivity analysis of the costs and analysis of epidemiological impact. The pressure control of patients was 54.4%, 98.2% and 93% in the pre PC, PC and post-PC, respectively. In direct cost analysis the difference between the total cost was -R$ 307.23 and R$ 4,053.32; the Incremental Cost Effectiveness Ratio was R$ 1,124.24 and R$ 100.98 for the periods PC and post-PC, respectively. There were 46 complications of hypertension and 17 deaths; and 15 complications and 5 deaths over ten years for health care without the PC and with PC, respectively. The cost per patient was R$ 17.09 / quality-adjusted life years and R$ 28.98 per year of life saved after discharge from the PC. The NPV obtained with the investment in this pharmaceutical practice was R$ 3,791,111.13 and the Benefit-cost Ratio (BCR) was 30.03 (26.74 - 34.28), BCR > 1 and investment return rate was 303%. It was concluded that the PC was able to improve the blood pressure control and total cholesterol level reducing the cardiovascular risk. That meant reduction spending on health and a positive epidemiological impact. This analysis was enough to develop a pharmacoeconomic tool that can assist managers in making decisions regarding the implementation of PC to reduce morbidity and mortality caused by SAH.

Introdução: A curva de crescimento é um modelo empírico da evolução de uma quantidade ao longo do tempo. As curvas de crescimento são utilizadas em muitas disciplinas , em particular no domínio da estatística, onde há uma grande literatura sobre o assunto relacionado a modelos não lineares. Método:No desenvolvimento dessa dissertação de mestrado, foi realizado um estudo baseado em dados de crescimento nas áreas biológica e médica para comparar os dois tipos de inferência (Clássica e Bayesiana), na busca de melhores estimativas e resultados para modelos de regressão não lineares, especialmente considerando alguns modelos de crescimento introduzidos na literatura. No método Bayesiano para a modelagem não linear assumimos erros normais uma suposição usual e também distribuições estáveis para a variável resposta. Estudamos também alguns aspectos de robustez dos modelos de regressão não linear para a presença de outliers ou observações discordantes considerando o uso de distribuições estáveis para a resposta no lugar da suposição de normalidade habitual. Resultados e Conclusões: Análise dos dois exemplos pode-se observar melhores ajustes quando utilizada o método Bayesiano de ajustes de modelos não lineares de curvas de crescimento. É bem sabido que, em geral, não há nenhuma forma fechada para a função densidade de probabilidade de distribuições estáveis. No entanto, sob uma abordagem Bayesiana, a utilização de uma variável aleatória latente ou auxiliar proporciona uma simplificação para obter qualquer distribuição a posteriori quando relacionado com distribuições estáveis. Esses resultados poderiam ser de grande interesse para pesquisadores e profissionais, ao lidar com dados não Gauss. Para demonstrar a utilidade dos aspectos computacionais, a metodologia é aplicada a um exemplo relacionado com as curvas de crescimento intra-uterino para prematuros. Resumos a posteriori de interesse são obtidos utilizando métodos MCMC (Markov Chain Monte Carlo) e o software OpenBugs. / Introduction: The growth curve is an empirical model of the evolution of a quantity over time. Growth curves are used in many disciplines, particularly in the field of statistics, where there is a large literature on the subject related to nonlinear models. Method: In the development of this dissertation, a study based on data growth in biological areas and medical was conducted to compare two types of inferences (Classical and Bayesian), in search of better estimates and results for nonlinear regression models, especially considering some growth models introduced in the literature. The Bayesian method for nonlinear modeling assume normal errors an usual assumption and also stable distributions for the response variable. We also study some aspects of robustness of nonlinear regression models for the presence of outliers or discordant observations regarding the use of stable distributions to the response in place of the usual assumption of normality. Results and Conclusions: In the analysis of two examples it can be seen best results using Bayesian methodology for non linear models of growth curves. It is well known that, in general, there is no closed form for the probability density function of stable distributions. However, under a Bayesian approach, the use of a latent random variable or auxiliary variable provides a simplification to get every conditional posterior related to stable distributions. These results could be of great interest to researchers and practitioners when dealing with non-Gaussian data. To demonstrate the utility of the computational aspects, the methodology is also applied to an example related to intrauterine growth curves for premature infants. Posterior summaries of interest are obtained using MCMC methods (MCMC) and the OpenBugs software.