Bayesian process control is a statistical process control (SPC) scheme that uses the posterior state probabilities as the control statistic. The key issue is to decide when to restore the process based on real-time observations. Such problems have been extensively studied in the framework of partially observable Markov decision processes (POMDP), with particular emphasis on the structure of optimal control policy. Almost all existing structural results on the optimal policies are limited to the two-state processes, where the class of control-limit policy is optimal. However, the two-state model is a gross simplification, as real production processes almost always involve multiple states. For example, a machine in the production system often has multiple failure modes differing in their effects; the deterioration process can often be divided into multiple stages with different degradation levels; the condition of a complex multi-unit system also requires a multi-state representation. We investigate the optimal control policies for multi-state processes with fixed sampling scheme, in which information about the process is represented by a belief vector within a high dimensional probability simplex. It is well known that obtaining structural results for such high-dimensional POMDP is challenging. Firstly, we prove that for an infinite-horizon process subject to multiple competing assignable causes, a so-called conditional control limit policy is optimal. The optimal policy divides the belief space into two individually connected regions, which have analytical bounds. Next, we address a finite-horizon process with at least one absorbing state and show that a structured optimal policy can be established by transforming the belief space into a polar coordinate system, where a so-called polar control limit policy is optimal. Our model is general enough to include many existing models in the literature as special cases. The structural results also lead to significantly efficient algorithms for computing the optimal policies. In addition, we characterize the condition for some out-of-control state to be more desirable than the in-control state. The existence of such counterintuitive situation indicates that multi-state process control is drastically different from the two-state case.

Markov decision processes (MDPs) have proven to be a useful model for sequential decision- theoretic reasoning under uncertainty, yet they require the specification of a reward function that can require sophisticated human judgement to assess relevant tradeoffs. This dissertation casts the problem of specifying rewards as one of preference elicitation and aims to minimize the degree of precision with which a reward function must be specified while still allowing optimal or near-optimal policies to be produced. We demonstrate how robust policies can be computed for MDPs given only partial reward information using the minimax regret criterion. Minimax regret offers an intuitive bound on loss; however, it is computationally intractable in general. This work develops techniques for exploiting MDP structure to allow for offline precomputation that enables efficient online minimax regret computation. To complement this exact approach we develop several general approximations that offer both upper and lower bounds on minimax regret. We further show how approximations can be improved online during the elicitation procedure to balance accuracy and efficiency. To effectively reduce regret, we investigate a spectrum of elicitation approaches that range from the computationally-demanding optimal selection of complex queries about full MDP policies (which are informative, but, we believe, cognitively difficult) to the heuristic selection of simple queries that focus on a small set of reward parameters. Results are demonstrated on MDPs drawn from the domains of assistive technology and autonomic computing. Finally we demonstrate our framework on a realistic website optimization domain, per- forming elicitation on websites with tens of thousands of webpages. We show that minimax regret can be efficiently computed, and develop informative and cognitively reasonable queries that quickly lower minimax regret, producing policies that offer significant improvement in the design of the underlying websites.

The 4th generation wireless communication systems aim to provide users with the convenience of seamless roaming among heterogeneous wireless access networks. To achieve this goal, the support of vertical handoff is important in mobility management. This thesis focuses on the vertical handoff decision algorithm, which determines the criteria under which vertical handoff should be performed. The problem is formulated as a constrained Markov decision process. The objective is to maximize the expected total reward of a connection subject to the expected total access cost constraint. In our model, a benefit function is used to assess the quality of the connection, and a penalty function is used to model the signaling incurred and call dropping. The user's velocity and location information are also considered when making the handoff decisions. The policy iteration and Q-learning algorithms are employed to determine the optimal policy. Structural results on the optimal vertical handoff policy are derived by using the concept of supermodularity. We show that the optimal policy is a threshold policy in bandwidth, delay, and velocity. Numerical results show that our proposed vertical handoff decision algorithm outperforms other decision schemes in a wide range of conditions such as variations on connection duration, user's velocity, user's budget, traffic type, signaling cost, and monetary access cost.

The main focus of this thesis is Markovian decision processes with an emphasis on incorporating time-dependence into the system dynamics. When considering such decision processes, we provide value equations that apply to a large range of classes of Markovian decision processes, including Markov decision processes (MDPs) and semi-Markov decision processes (SMDPs), time-homogeneous or otherwise. We then formulate a simple decision process with exponential state transitions and solve this decision process using two separate techniques. The first technique solves the value equations directly, and the second utilizes an existing continuous-time MDP solution technique. To incorporate time-dependence into the transition dynamics of the process, we examine a particular decision process with state transitions determined by the Erlang distribution. Although this process is originally classed as a generalized semi-Markov decision process, we re-define it as a time-inhomogeneous SMDP. We show that even for a simply stated process with desirable state-space properties, the complexity of the value equations becomes so substantial that useful analytic expressions for the optimal solutions for all states of the process are unattainable. We develop a new technique, utilizing phase-type (PH) distributions, in an effort to address these complexity issues. By using PH representations, we construct a new state-space for the process, referred to as the phase-space, incorporating the phases of the state transition probability distributions. In performing this step, we effectively model the original process as a continuous-time MDP. The information available in this system is, however, richer than that of the original system. In the interest of maintaining the physical characteristics of the original system, we define a new valuation technique for the phase-space that shields some of this information from the decision maker. Using the process of phase-space construction and our valuation technique, we define an original system of value equations for this phasespace that are equivalent to those for the general Markovian decision processes mentioned earlier. An example of our own phase-space technique is given for the aforementioned Erlang decision process and we identify certain characteristics of the optimal solution such that, when applicable, the implementation of our phase-space technique is greatly simplified. These newly defined value equations for the phase-space are potentially as complex to solve as those defined for the original model. Restricting our focus to systems with acyclic state-spaces though, we describe a top-down approach to solution of the phase-space value equations for more general processes than those considered thus far. Again, we identify characteristics of the optimal solution to look for when implementing this technique and provide simplifications of the value equations where these characteristics are present. We note, however, that it is almost impossible to determine a priori the class of processes for which the simplifications outlined in our phase-space technique will be applicable. Nevertheless, we do no worse in terms of complexity by utilizing our phase-space technique, and leave open the opportunity to simplify the solution process if an appropriate situation arises. The phase-space technique can handle time-dependence in the state transition probabilities, but is insufficient for any process with time-dependent reward structures or discounting. To address such decision processes, we define an approximation technique for the solution of the class of infinite horizon decision processes whose state transitions and reward structures are described with reference to a single global clock. This technique discretizes time into exponentially distributed length intervals and incorporates this absolute time information into the state-space. For processes where the state-transitions are not exponentially distributed, we use the hazard rates of the transition probability distributions evaluated at the discrete time points to model the transition dynamics of the system. We provide a suitable reward structure approximation using our discrete time points and guidelines for sensible truncation, using an MDP approximation to the tail behaviour of the original infinite horizon process. The result is a finite-state time-homogeneous MDP approximation to the original process and this MDP may be solved using standard existing solution techniques. The approximate solution to the original process can then be inferred from the solution to our MDP approximation. / Thesis (Ph.D.) -- University of Adelaide, School of Mathematical Sciences, 2008

abstract: TaxiWorld is a Matlab simulation of a city with a fleet of taxis which operate within it, with the goal of transporting passengers to their destinations. The size of the city, as well as the number of available taxis and the frequency and general locations of fare appearances can all be set on a scenario-by-scenario basis. The taxis must attempt to service the fares as quickly as possible, by picking each one up and carrying it to its drop-off location. The TaxiWorld scenario is formally modeled using both Decentralized Partially-Observable Markov Decision Processes (Dec-POMDPs) and Multi-agent Markov Decision Processes (MMDPs). The purpose of developing formal models is to learn how to build and use formal Markov models, such as can be given to planners to solve for optimal policies in problem domains. However, finding optimal solutions for Dec-POMDPs is NEXP-Complete, so an empirical algorithm was also developed as an improvement to the method already in use on the simulator, and the methods were compared in identical scenarios to determine which is more effective. The empirical method is of course not optimal - rather, it attempts to simply account for some of the most important factors to achieve an acceptable level of effectiveness while still retaining a reasonable level of computational complexity for online solving. / Dissertation/Thesis / M.S. Computer Science 2011

Os Processos de Decisão de Markov (PDMs) são uma importante ferramenta de planejamento e otimização em ambientes que envolvem incertezas. Contudo a especificação e representação computacional das distribuições de probabilidades subjacentes a PDMs é uma das principais dificuldades de utilização desta ferramenta. Este trabalho propõe duas estratégias para representação destas probabilidades de forma compacta e eficiente. Estas estratégias utilizam redes Bayesianas e regularidades entre os estados e as variáveis. As estratégias apresentadas são especialmente úteis em sistemas onde as variáveis têm muitas categorias e possuem forte inter-relação. Além disso, é apresentada a aplicação destes modelos no gerenciamento de grupos de impressoras (um problema real da indústria e que motivou o desenvolvimento do trabalho) permitindo que estas atuem coletiva e não individualmente. O último tópico discutido é uma análise comparativa da mesma aplicação utilizando Lógica Difusa. / Markov Decision Processes (MDPs) are an important tool for planning and optimization in environments under uncertainty. The specification and computational representation of the probability distributions underlying MDPs are central difficulties for their application. This work proposes two strategies for representation of probabilities in a compact and efficient way. These strategies use Bayesian networks and regularities among states and variables. The proposed strategies are particularly useful in systems whose variables have many categories and have strong interrelation. This proposal has been applied to the management of clusters of printers, a real problem that in fact motivated the work. Markov Decision Processes are then used to allow printers to act as a group, and not just individually. The work also presents a comparison between MDPs and Fuzzy Logic in the context of clusters of printers.

Orientador : Celso Pascoli Bottura / Tese (doutorado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica / Made available in DSpace on 2018-08-17T06:32:39Z (GMT). No. of bitstreams: 1 CostaFilho_JoseTarcisio_D.pdf: 11412677 bytes, checksum: cbe4c9ca6c7a461ef41e5bff9934bfbd (MD5) Previous issue date: 1992 / Resumo: Sistemas reais de grande porte constituídos de subsistemas interconectados fisicamente e distribuídos espacialmente, frequentemente envolvem a descentralização do con trole entre vários agentes de decisão com objetivos diferentes e comumente conflitantes. Quando mais de um agente de decisão, AD, está envolvido, as teorias de jogos dinâmicos e de sistemas hierárquicos devem ser consideradas. Neste sentido, o jogo dinâmico, visto como uma tarefa organizacional, pode ser mapeado sobre um rede de computadores onde cada agente de decisão é associado a um computador. Para a computação paralela e distribui da de jogos dinâmicos esta tese trata de alguns aspectos importantes. Primeiramente, a proposta de um jogo distribuído assíncrono onde a distribuição dos agentes de decisão sobre a rede de computadores permite o acoplamento algoritmo/ hal'dware. Para isto, o conceito de jogo de Stackelberg é estendido para incluir jogos multiníveis onde os agentes são divididos em dois grupos: um grupo de agentes líderes (múltiplos coordenadores) e um grupo de agentes seguidores (controladores locais), e cada AD pode ou não estar cooperando dentro de seu grupo. Neste sentido, propusemos uma metodologia de otimização distribui da heterárquica que envolve esquemas de decomposições espacial e temporal com distribuiçâo da coordenaçâo, baseada em estratégias em malha aberta de Stackelberg, de Nash, de Pareto e de Equipe. Em segundo lugar, propomos formas de implementaçâo dos algoritmos de decisâo propostos sobre arquiteturas de computação paralela, vetorial e distribuida, com os objetivos de melhorar o desempenho computacional e de obter decisões on-line através da exploração dos graus de paralelismo e de vetorização nos algoritmos para a obtençâo de decisões individuais e da computação distribuida do sistema de múltiplos agentes. Em terceiro lugar, desenvolvemos uma ferramenta de software de suporte para múltiplas decisões distribui das sobre uma rede de estações de trabalho UNIX / Abstract: Reallarge scale systems consisting of physically interconnected subsystems dis ributed in space, frequently involve a decentralization of control among several decision makers with different and usually conflicting als hen more than one decision maker, D M, is involved, dynamic games theory and hierarchical approaches ould be consid ered. ln this sense, the dynamic game viewed as an organizational task can be mapped onto a computeI' network where each local decision is associated to a computeI'. For the parallel and distributed computation of dynamic games. this thesis studies some important aspects. Firstly, the proposal of an ynchronous distributed game where the distribuition of the decision makers over the computeI' network allows the matching algorithm/hardware. For doing this, the Stackelberg concept is idened to include multilevel games where the D.M sare assumed to be divided into t o groups: a group of leaders (multiple coordinators) and a group of followers (local controlers), and every DM may or may not be cooperating within his own group. ln this sense e propose an heterarchical distributed optimization methodology that inyolves temporal and spatial decomposition schemes with coordination distribution based on open-Ioop Stackelberg, Nash, Pareto and Team strategies. Secondly, we propose implementation forms for the proposed de cision algorithms on the parallel, vectorial and distributed computeI' architectures, with the objectives of computational rformance imprO ement and of on-line decision making through the algorithms parallelism and vectorizati_n degrees exploration for the individual decisions achievement and through the decision makers distributed computation. Thirdly, a distributed multidecision making suport soft are over a workstation network was developed / Doutorado / Doutor em Engenharia Elétrica

Orientador: Luiz Henrique Antunes Rodrigues / Dissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Agricola / Made available in DSpace on 2018-08-13T20:39:32Z (GMT). No. of bitstreams: 1 AndradeFilho_ValdirBarreto_M.pdf: 1448479 bytes, checksum: 5a7006a16f49c5cbbe9b8692fcdd2f3a (MD5) Previous issue date: 2009 / Resumo: A necessidade de eficiência e eficácia, motivada pelo ambiente competitivo, pelas exigências de mercado e pela escassez de recursos, torna a disciplina de Gerência de Projetos cada vez mais importante para as Organizações. Saber, portanto, como o Gerenciamento dos Projetos é desempenhado torna-se um conhecimento importante para que estas entendam suas forças e fraquezas e a partir daí possam definir como evoluir. Modelos de Maturidade em Gerenciamento de Projetos são ferramentas para se obter este conhecimento e, particularmente, o "Organizational Project Management Maturity Model" (OPM3), pela sua abrangência e por ter sido criado pelo organismo internacional mais conceituado atualmente nesta área, o "Project Management Institute" (PMI), se afigura como um importante modelo para se aferir o "status" da Organização em relação a seus processos de Gerenciamento de Projetos. Dada a importância do Agronegócio no país e do papel desempenhado pelas Instituições de Pesquisa do Agronegócio do Estado de S. Paulo, pelo seu volume de Projetos e por sua importância no cenário nacional torna-se importante a determinação de como estas Instituições se posicionam em termos de Maturidade no Gerenciamento de seus Projetos. Tomando como hipóteses que o grau de maturidade das Instituições de Pesquisa do Agronegócio do Estado de S. Paulo é semelhante ao da maioria das organizações do País e de que as Instituições em estágio mais avançado compartilham de práticas comuns, este trabalho tem como objetivo apresentar a Maturidade em Gerenciamento de Projetos de um conjunto destas Instituições, bem como o desenvolvimento de um modelo de apoio à decisão que oriente estas Instituições na definição de sua evolução em Maturidade. Para realizá-lo, aplicou-se as técnicas preconizadas pelo modelo OPM3 - questionários e "software" específico - a um conjunto de pesquisadores de diversas Instituições, adaptando-as ao contexto específico de apenas se investigar Projetos, não considerando Programas ou Portfólios. Os resultados indicam a confirmação das Hipóteses, apresentando-se, inclusive, as Melhores Práticas comuns às Instituições em estágio maior de evolução. O modelo proposto de apoio à decisão, baseado no método "Analytic Hierarchy Process" (AHP), apresenta uma hierarquia de um único nível englobando os Domínios, os Estágios, os Grupos de Processos e as Áreas de Conhecimento. À hierarquia é adicionada uma estrutura desenvolvida originalmente neste trabalho (Matriz de Pertinência) para que a prioridade das ações de evolução seja definida O modelo mostra-se factível do ponto de vista prático e abrange todos os domínios do Gerenciamento de Projetos Organizacional. Palavras-chave: Processo de Análise Hierárquica (AHP); Modelo de Maturidade do Gerenciamento de Projetos Organizacional (OPM3); Melhor Prática; Instituto de Gerenciamento de Projetos (PMI) / Abstract: The necessity of efficiency and effectiveness, motivated by the current competitive environment, market needs and lack of resources, brought Project Management to a top position inside Organizations. To know how Project Management is performed becomes a very important knowledge for these organizations in order to learn their strengths and weaknesses. With this knowledge they will be able to define how to evolve in this aspect. Project Management Maturity Models are tools to obtain this knowledge. The Organizational Project Management Maturity Model (OPM3), by its comprehensiveness e due the fact that it was developed by the most considered international institution in this subject, the Project Management Institute (PMI), may be considered as an important model to measure the Organization "status" related to its Project Management processes. Taking into account the Agribusiness relevance in Brazil and the role performed by Agribusiness Research Institutes in São Paulo State, by their Project numbers and by their relevance in Brazil's research environment, it becomes very relevant to know how these Institutions are positioned concerning about Project Management Maturity in their Projects. Taking as hypothesis that the Agribusiness Research Institutes in São Paulo State maturity level is similar to the most country's Organizations and that Institutions in a more evolved level share a common Best Practices nucleous, this work aims to show the Project Management Maturity of some of these Institutions and, also, the development of a decision support model that help them to define their evolution path. In order to do it, OPM3 techniques were applied - a questionnaire and OPM3 specific software - to a selected researchers group from some Institutions, adapting them to the Project Management specific context and leaving outside Programs and Portfolios. The results confirm both Hypothesis, and the Best Practices nucleous is showed. The decision support model proposed, based in the Analytic Hierarchy Process (AHP), shows an one level hierarchy including the Domains, the Stages, the Processes Groups and the Knowledge Areas. One originally developed structure is added to the hierarchy (Pertinency Matrix) to allow the evolution actions prioritization. The model is feasible to pratical use and includes all of the Organizational Project Management domains. Keywords: Analytic Hierarchy Process (AHP); Organizational Project Management Maturity Model (OPM3); Best Practice; Project Management Institute (PMI) / Mestrado / Planejamento e Desenvolvimento Rural Sustentável / Mestre em Engenharia Agrícola

The 4th generation wireless communication systems aim to provide users with the convenience of seamless roaming among heterogeneous wireless access networks. To achieve this goal, the support of vertical handoff is important in mobility management. This thesis focuses on the vertical handoff decision algorithm, which determines the criteria under which vertical handoff should be performed. The problem is formulated as a constrained Markov decision process. The objective is to maximize the expected total reward of a connection subject to the expected total access cost constraint. In our model, a benefit function is used to assess the quality of the connection, and a penalty function is used to model the signaling incurred and call dropping. The user's velocity and location information are also considered when making the handoff decisions. The policy iteration and Q-learning algorithms are employed to determine the optimal policy. Structural results on the optimal vertical handoff policy are derived by using the concept of supermodularity. We show that the optimal policy is a threshold policy in bandwidth, delay, and velocity. Numerical results show that our proposed vertical handoff decision algorithm outperforms other decision schemes in a wide range of conditions such as variations on connection duration, user's velocity, user's budget, traffic type, signaling cost, and monetary access cost. / Applied Science, Faculty of / Electrical and Computer Engineering, Department of / Graduate

京都大学 / 0048 / 新制・課程博士 / 博士(人間・環境学) / 甲第22535号 / 人博第938号 / 新制||人||223(附属図書館) / 2019||人博||938(吉田南総合図書館) / 京都大学大学院人間・環境学研究科共生人間学専攻 / (主査)准教授 櫻川 貴司, 教授 立木 秀樹, 教授 日置 尋久 / 学位規則第4条第1項該当 / Doctor of Human and Environmental Studies / Kyoto University / DGAM