abstract: Animals learn to choose a proper action among alternatives according to the circumstance. Through trial-and-error, animals improve their odds by making correct association between their behavioral choices and external stimuli. While there has been an extensive literature on the theory of learning, it is still unclear how individual neurons and a neural network adapt as learning progresses. In this dissertation, single units in the medial and lateral agranular (AGm and AGl) cortices were recorded as rats learned a directional choice task. The task required the rat to make a left/right side lever press if a light cue appeared on the left/right side of the interface panel. Behavior analysis showed that rat's movement parameters during performance of directional choices became stereotyped very quickly (2-3 days) while learning to solve the directional choice problem took weeks to occur. The entire learning process was further broken down to 3 stages, each having similar number of recording sessions (days). Single unit based firing rate analysis revealed that 1) directional rate modulation was observed in both cortices; 2) the averaged mean rate between left and right trials in the neural ensemble each day did not change significantly among the three learning stages; 3) the rate difference between left and right trials of the ensemble did not change significantly either. Besides, for either left or right trials, the trial-to-trial firing variability of single neurons did not change significantly over the three stages. To explore the spatiotemporal neural pattern of the recorded ensemble, support vector machines (SVMs) were constructed each day to decode the direction of choice in single trials. Improved classification accuracy indicated enhanced discriminability between neural patterns of left and right choices as learning progressed. When using a restricted Boltzmann machine (RBM) model to extract features from neural activity patterns, results further supported the idea that neural firing patterns adapted during the three learning stages to facilitate the neural codes of directional choices. Put together, these findings suggest a spatiotemporal neural coding scheme in a rat AGl and AGm neural ensemble that may be responsible for and contributing to learning the directional choice task. / Dissertation/Thesis / Ph.D. Electrical Engineering 2014

Open-world computer games present the players with a large degree of freedom to interact with the virtual environment. The increased player freedom makes open-world games a challenging domain for artificial intelligence. In this thesis we present three novel techniques to handle various types of complexity inherent in developing artificial intelligence for open-world games. We developed behavior objects that extend the well-known concept of smart objects and help in structuring codebase for reactive reasoning, we propose and implement constraint satisfaction techniques to specify behavior from a global viewpoint and we have shown how adversarial search techniques can mitigate the need for complex reactive decision mechanisms when a large number of parameters has to be taken into account. The general techniques are implemented and evaluated in the context of a complete open-world game Kingdom Come: Deliverance. Powered by TCPDF (www.tcpdf.org)

In daily activities, the brain regularly assigns different roles to the hands dependingon task and context. Yet, little is known about the underlying neural processes. Thiscertainly applies to how the brain, where each hemisphere primarily controls onehand, manages the between-hand coordination required in bimanual objectmanipulation. By using behavioral, neurophysiological and functional magneticresonance imaging techniques, the present thesis examines neural mechanisms thatsupport hand coordination during tasks where the two hands apply spatiotemporallycoupled but opposing forces for goal attainment, e.g., as when removing the cap froma bottle. Although the two hands seem to operate symmetrically in such tasks, Study Ishowed that one hand primarily acts while the other assists. Moreover, this roledifferentiation was found to be flexible with the brain appointing either hand asprime actor depending on the spatial congruency between hand forces and desiredmovement consequences. Accordingly, when we remove a cap from a bottle, the handthat grasps the cap, be it left or right depending on overall task constraints, isappointed as prime actor because the twist forces it generates are aligned with thegoal to remove the cap, while the other hand, holding the bottle, applies stabilizingforces in the opposite direction. Changes in hand assignments are caused by amidline shift of lateralized activity throughout the motor system, from distal handmuscles to corticospinal pathways and primary sensorimotor and cerebellar corticalareas (Study I). Although the bimanual actions examined involved both within- andbetween-hand coordination, Study II failed to reveal additional brain activity duringbimanual as compared to matching unimanual actions, except for the primarysensorimotor areas where subpopulations of neurons were preferentially engagedduring either bimanual or unimanual actions. Thus, dedicated neurons in the motorcortices might support critical bimanual coordinative operations. While imagingresults indicated that a mainly left-lateralized parietal-premotor network managedthe task irrespective of prime actor, premotor areas presumably established handassignment by allocating the lead either to the left or the right primary sensorimotorareas (Study I and II). Regarding the process of prime actor selection and hence thecontrol of these premotor networks, imaging results indicate a transitory involvementof prefrontal cortical areas (Study III). The detected areas belong to a networkconsidered critical for cognitive operations such as judgment and decision-making,and for evaluation of utility of actions, including conflict detection. The implicitselection of prime actor during bimanual tasks thus seems to be supported by corticalareas traditionally associated primarily with complex cognitive challenges.

Can a robot understand a human's social behavior? Moreover, how should a robot act in response to a human's behavior? If the goals of artificial intelligence are to understand, imitate, and interact with human level intelligence then researchers must also explore the social underpinnings of this intellect. Our endeavor is buttressed by work in biology, neuroscience, social psychology and sociology. Initially developed by Kelley and Thibaut, social psychology's interdependence theory serves as a conceptual skeleton for the study of social situations, a computational process of social deliberation, and relationships (Kelley&Thibaut, 1978). We extend and expand their original work to explore the challenge of interaction with an embodied, situated robot. This dissertation investigates the use of outcome matrices as a means for computationally representing a robot's interactions. We develop algorithms that allow a robot to create these outcome matrices from perceptual information and then to use them to reason about the characteristics of their interactive partner. This work goes on to introduce algorithms that afford a means for reasoning about a robot's relationships and the trustworthiness of a robot's partners. Overall, this dissertation embodies a general, principled approach to human-robot interaction which results in a novel and scientifically meaningful approach to topics such as trust and relationships.

Le travail présenté dans cette thèse porte sur le rôle du cortex prémoteur dorsal (PMd) au sujet de la prise de décision (sélection d’une action parmis nombreux choix) et l'orientation visuelle des mouvements du bras. L’ouvrage décrit des expériences électrophysiologiques chez le singe éveillé (Macaca mulatta) permettant d’adresser une fraction importante des prédictions proposées par l'hypothèse des affordances concurrentes (Cisek, 2006; Cisek, 2007a). Cette hypothèse suggère que le choix de toute action est l’issue d'une concurrence entre les représentations internes des exigences et des atouts de chacune des options présentées (affordances; Gibson, 1979). Un intérêt particulier est donné au traitement de l'information spatiale et la valeur des options (expected value, EV) dans la prise de décisions. La première étude (article 1) explore la façon dont PMd reflète ces deux paramètres dans la période délai ainsi que de leur intéraction. La deuxième étude (article 2) explore le mécanisme de décision de façon plus détaillée et étend les résultats au cortex prémoteur ventral (PMv). Cette étude porte également sur la représentation spatiale et l’EV dans une perspective d'apprentissage. Dans un environnement nouveau les paramètres spatiaux des actions semblent être présents en tout temps dans PMd, malgré que la représentation de l’EV apparaît uniquement lorsque les animaux commencent à prendre des décisions éclairées au sujet de la valeur des options disponibles. La troisième étude (article 3) explore la façon dont PMd est impliqué aux “changements d'esprit“ dans un procès de décision. Cette étude décrit comment la sélection d’une action est mise à jour à la suite d'une instruction de mouvement (GO signal). I II Les résultats principaux des études sont reproduits par un modèle computationnel (Cisek, 2006) suggérant que la prise de décision entre plusieurs actions alternatives peux se faire par voie d’un mécanisme de concurrence (biased competition) qui aurait lieu dans la même région qui spécifie les actions. / This thesis examines the role of the dorsal premotor cortex (PMd) in the process of decision making (action selection) and visual guidance of arm movements. The work describes electrophysiological experiments conducted in awake monkeys (Macaca mulatta) and tests a number of important predictions suggested by the affordance competition hypothesis (Cisek, 2006; Cisek, 2007a). This hypothesis suggests that decisions can be viewed as the result of a competition between internal representations of conflicting demands and opportunities for actions or affordances (Gibson, 1979). Specific interest is given to the interaction between spatial information and expected value (EV) in a proposed affordance competition mechanism for action selection. The first study presented (article 1) explores how EV is represented during the delay period in PMd. This study also describes how this area reflects the spatial metrics of the options and examines the interaction between value and spatial information. The second study (article 2) explores the mechanism of action selection in more detail and extends the results to ventral premotor cortex (PMv). This study also addresses the nature of value and spatial representations from a learning perspective. In a novel environment the spatial metrics of the actions seem to be invariably present in PMd, meanwhile EV representations appear only once the animals make behaviorally informed decisions about the value of the available options. The third study (article 3) explores how PMd is involved in “changes of mind” in which action selection is updated following a movement instruction (GO signal). III IV The major findings in all these studies are reproduced by a computational model (Cisek, 2006) suggesting that decisions between actions can be made through a biased competition process that takes place in the same region that specifies the actions.

Title: Controlling Virtual People Author: Mgr. Jakub Gemrot Department: Department of Software and Computer Science Education Supervisor: Mgr. Cyril Brom, PhD. Abstract: In this thesis, we provide a computational formalization of reactive planning as a paradigm for decision making of intelligent virtual agents and videogame non-player characters. We formalize agent decision-making (ADM) as a process of deciding on which body actions to execute next and differentiate it from agent reasoning as a process of computing facts needed for decision making. We show that imperative programming languages are not suitable for ADM specification and explain why they are not suitable. Thereafter, we create a new computational model that we use as the basis for the definition of Behavior Design Language (BDL). We show that BDL can model decision-making specified by scripting, hierarchical finite-state machines, AgentSpeak(L), GOAL, SPOSH and Behavior trees. Importantly, BDL can model these approaches economically in terms of the number of behavior primitives. The key strengths of the BDL language are: versatility (it can mix decision making patterns of multiple languages together), extensibility (it allows developers to devise new language primitives as they see fit), and generality (it can abstract any computable...

Este trabalho investiga a aplicação de rede de comportamentos estendidas ao domínio de jogos de computador. Redes de comportamentos estendidas (RCE) são uma classe de arquiteturas para seleção de ações capazes de selecionar bons conjuntos de ações para agentes complexos situados em ambientes contínuos e dinâmicos. Foram aplicadas com sucesso na Robocup, mas nunca foram aplicadas a jogos. PHISH-Nets, um modelo de redes de comportamentos capaz de selecionar apenas uma ação por vez, foi aplicado à modelagem de personagens, com bons resultados. Apesar de RCEs serem aplicáveis a um conjunto de domínios maior, nunca foram usadas para modelagem de personagens. Apresenta-se como projetar um agente controlado por uma rede de comportamentos para o domínio do Unreal Tournament e como integrar a rede de comportamentos a sensores nebulosos e comportamentos baseados em máquinas de estado-finito aumentadas. Investiga-se a qualidade da seleção de ações e a correção do mecanismo em uma série de experimentos. A performance é medida através da comparação das pontuações de um agente baseado em redes de comportamentos com outros dois agentes. Um dos agentes foi implementado por outro grupo e usava sensores, efetores e comportamentos diferentes. O outro agente era idêntico ao agente baseado em RCEs, exceto pelo mecanismo de controle empregado. A modelagem de personalidade é investigada através do projeto e análise de cinco estereótipos: Samurai, Veterano, Berserker, Novato e Covarde. Apresenta-se três maneiras de construir personalidades e situa-se este trabalho dentro de outras abordagems de projeto de personalidades. Conclui-se que a rede de comportamentos estendida é um bom mecanismo de seleção de ações para o domínio de jogos de computador e um mecanismo interessante para a construção de agentes com personalidades simples. / This work investigates the application of extended behavior networks to the computer game domain. We use as our test bed the game Unreal Tournament. Extended Behavior Networks (EBNs) are a class of action selection architectures capable of selecting a good set of actions for complex agents situated in continuous and dynamic environments. They have been successfully applied to the Robocup, but never before used in computer games. PHISH-Nets, a behavior network model capable of selecting just single actions, was applied to character modeling with promising results. Although extended behavior networks are applicable to a larger domain, they had not been used to character modeling before. We present how to design an agent with extended behavior networks, fuzzy sensors and finite-state machine based behaviors. We investigate the quality of the action selection mechanism and its correctness in a series of experiments. The performance is assessed comparing the scores of an agent using an extended behavior network against a plain reactive agent with identical sensory-motor apparatus and against a totally different agent built around finite-state machines. We investigate how EBNs fare on agent personality modeling via the design and analysis of five stereotypes in Unreal Tournament. We discuss three ways to build character personas and situate our work within other approaches. We conclude that extended behavior networks are a good action selection architecture for the computer game domain and an interesting mechanism to build agents with simple personalities.

Este trabalho investiga a aplicação de rede de comportamentos estendidas ao domínio de jogos de computador. Redes de comportamentos estendidas (RCE) são uma classe de arquiteturas para seleção de ações capazes de selecionar bons conjuntos de ações para agentes complexos situados em ambientes contínuos e dinâmicos. Foram aplicadas com sucesso na Robocup, mas nunca foram aplicadas a jogos. PHISH-Nets, um modelo de redes de comportamentos capaz de selecionar apenas uma ação por vez, foi aplicado à modelagem de personagens, com bons resultados. Apesar de RCEs serem aplicáveis a um conjunto de domínios maior, nunca foram usadas para modelagem de personagens. Apresenta-se como projetar um agente controlado por uma rede de comportamentos para o domínio do Unreal Tournament e como integrar a rede de comportamentos a sensores nebulosos e comportamentos baseados em máquinas de estado-finito aumentadas. Investiga-se a qualidade da seleção de ações e a correção do mecanismo em uma série de experimentos. A performance é medida através da comparação das pontuações de um agente baseado em redes de comportamentos com outros dois agentes. Um dos agentes foi implementado por outro grupo e usava sensores, efetores e comportamentos diferentes. O outro agente era idêntico ao agente baseado em RCEs, exceto pelo mecanismo de controle empregado. A modelagem de personalidade é investigada através do projeto e análise de cinco estereótipos: Samurai, Veterano, Berserker, Novato e Covarde. Apresenta-se três maneiras de construir personalidades e situa-se este trabalho dentro de outras abordagems de projeto de personalidades. Conclui-se que a rede de comportamentos estendida é um bom mecanismo de seleção de ações para o domínio de jogos de computador e um mecanismo interessante para a construção de agentes com personalidades simples. / This work investigates the application of extended behavior networks to the computer game domain. We use as our test bed the game Unreal Tournament. Extended Behavior Networks (EBNs) are a class of action selection architectures capable of selecting a good set of actions for complex agents situated in continuous and dynamic environments. They have been successfully applied to the Robocup, but never before used in computer games. PHISH-Nets, a behavior network model capable of selecting just single actions, was applied to character modeling with promising results. Although extended behavior networks are applicable to a larger domain, they had not been used to character modeling before. We present how to design an agent with extended behavior networks, fuzzy sensors and finite-state machine based behaviors. We investigate the quality of the action selection mechanism and its correctness in a series of experiments. The performance is assessed comparing the scores of an agent using an extended behavior network against a plain reactive agent with identical sensory-motor apparatus and against a totally different agent built around finite-state machines. We investigate how EBNs fare on agent personality modeling via the design and analysis of five stereotypes in Unreal Tournament. We discuss three ways to build character personas and situate our work within other approaches. We conclude that extended behavior networks are a good action selection architecture for the computer game domain and an interesting mechanism to build agents with simple personalities.

Este trabalho investiga a aplicação de rede de comportamentos estendidas ao domínio de jogos de computador. Redes de comportamentos estendidas (RCE) são uma classe de arquiteturas para seleção de ações capazes de selecionar bons conjuntos de ações para agentes complexos situados em ambientes contínuos e dinâmicos. Foram aplicadas com sucesso na Robocup, mas nunca foram aplicadas a jogos. PHISH-Nets, um modelo de redes de comportamentos capaz de selecionar apenas uma ação por vez, foi aplicado à modelagem de personagens, com bons resultados. Apesar de RCEs serem aplicáveis a um conjunto de domínios maior, nunca foram usadas para modelagem de personagens. Apresenta-se como projetar um agente controlado por uma rede de comportamentos para o domínio do Unreal Tournament e como integrar a rede de comportamentos a sensores nebulosos e comportamentos baseados em máquinas de estado-finito aumentadas. Investiga-se a qualidade da seleção de ações e a correção do mecanismo em uma série de experimentos. A performance é medida através da comparação das pontuações de um agente baseado em redes de comportamentos com outros dois agentes. Um dos agentes foi implementado por outro grupo e usava sensores, efetores e comportamentos diferentes. O outro agente era idêntico ao agente baseado em RCEs, exceto pelo mecanismo de controle empregado. A modelagem de personalidade é investigada através do projeto e análise de cinco estereótipos: Samurai, Veterano, Berserker, Novato e Covarde. Apresenta-se três maneiras de construir personalidades e situa-se este trabalho dentro de outras abordagems de projeto de personalidades. Conclui-se que a rede de comportamentos estendida é um bom mecanismo de seleção de ações para o domínio de jogos de computador e um mecanismo interessante para a construção de agentes com personalidades simples. / This work investigates the application of extended behavior networks to the computer game domain. We use as our test bed the game Unreal Tournament. Extended Behavior Networks (EBNs) are a class of action selection architectures capable of selecting a good set of actions for complex agents situated in continuous and dynamic environments. They have been successfully applied to the Robocup, but never before used in computer games. PHISH-Nets, a behavior network model capable of selecting just single actions, was applied to character modeling with promising results. Although extended behavior networks are applicable to a larger domain, they had not been used to character modeling before. We present how to design an agent with extended behavior networks, fuzzy sensors and finite-state machine based behaviors. We investigate the quality of the action selection mechanism and its correctness in a series of experiments. The performance is assessed comparing the scores of an agent using an extended behavior network against a plain reactive agent with identical sensory-motor apparatus and against a totally different agent built around finite-state machines. We investigate how EBNs fare on agent personality modeling via the design and analysis of five stereotypes in Unreal Tournament. We discuss three ways to build character personas and situate our work within other approaches. We conclude that extended behavior networks are a good action selection architecture for the computer game domain and an interesting mechanism to build agents with simple personalities.

Le travail présenté dans cette thèse porte sur le rôle du cortex prémoteur dorsal (PMd) au sujet de la prise de décision (sélection d’une action parmis nombreux choix) et l'orientation visuelle des mouvements du bras. L’ouvrage décrit des expériences électrophysiologiques chez le singe éveillé (Macaca mulatta) permettant d’adresser une fraction importante des prédictions proposées par l'hypothèse des affordances concurrentes (Cisek, 2006; Cisek, 2007a). Cette hypothèse suggère que le choix de toute action est l’issue d'une concurrence entre les représentations internes des exigences et des atouts de chacune des options présentées (affordances; Gibson, 1979). Un intérêt particulier est donné au traitement de l'information spatiale et la valeur des options (expected value, EV) dans la prise de décisions. La première étude (article 1) explore la façon dont PMd reflète ces deux paramètres dans la période délai ainsi que de leur intéraction. La deuxième étude (article 2) explore le mécanisme de décision de façon plus détaillée et étend les résultats au cortex prémoteur ventral (PMv). Cette étude porte également sur la représentation spatiale et l’EV dans une perspective d'apprentissage. Dans un environnement nouveau les paramètres spatiaux des actions semblent être présents en tout temps dans PMd, malgré que la représentation de l’EV apparaît uniquement lorsque les animaux commencent à prendre des décisions éclairées au sujet de la valeur des options disponibles. La troisième étude (article 3) explore la façon dont PMd est impliqué aux “changements d'esprit“ dans un procès de décision. Cette étude décrit comment la sélection d’une action est mise à jour à la suite d'une instruction de mouvement (GO signal). I II Les résultats principaux des études sont reproduits par un modèle computationnel (Cisek, 2006) suggérant que la prise de décision entre plusieurs actions alternatives peux se faire par voie d’un mécanisme de concurrence (biased competition) qui aurait lieu dans la même région qui spécifie les actions. / This thesis examines the role of the dorsal premotor cortex (PMd) in the process of decision making (action selection) and visual guidance of arm movements. The work describes electrophysiological experiments conducted in awake monkeys (Macaca mulatta) and tests a number of important predictions suggested by the affordance competition hypothesis (Cisek, 2006; Cisek, 2007a). This hypothesis suggests that decisions can be viewed as the result of a competition between internal representations of conflicting demands and opportunities for actions or affordances (Gibson, 1979). Specific interest is given to the interaction between spatial information and expected value (EV) in a proposed affordance competition mechanism for action selection. The first study presented (article 1) explores how EV is represented during the delay period in PMd. This study also describes how this area reflects the spatial metrics of the options and examines the interaction between value and spatial information. The second study (article 2) explores the mechanism of action selection in more detail and extends the results to ventral premotor cortex (PMv). This study also addresses the nature of value and spatial representations from a learning perspective. In a novel environment the spatial metrics of the actions seem to be invariably present in PMd, meanwhile EV representations appear only once the animals make behaviorally informed decisions about the value of the available options. The third study (article 3) explores how PMd is involved in “changes of mind” in which action selection is updated following a movement instruction (GO signal). III IV The major findings in all these studies are reproduced by a computational model (Cisek, 2006) suggesting that decisions between actions can be made through a biased competition process that takes place in the same region that specifies the actions.