Modular reinforcement learning is an approach to resolve the curse of dimensionality problem in traditional reinforcement learning. We design and implement a modular reinforcement learning algorithm, which is based on three major components: Markov decision process decomposition, module training, and global action selection. We define and formalize module class and module instance concepts in decomposition step. Under our framework of decomposition, we train each modules efficiently using SARSA($\lambda$) algorithm. Then we design, implement, test, and compare three action selection algorithms based on different heuristics: Module Combination, Module Selection, and Module Voting. For last two algorithms, we propose a method to calculate module weights efficiently, by using standard deviation of Q-values of each module. We show that Module Combination and Module Voting algorithms produce satisfactory performance in our test domain. / text

Agents, by definition, (1) are situated in an environment upon which their actions affect changes and (2) have some level of autonomy from the control of humans or other agents. Being situated requires that the agent have a mechanism for sensing the environment as well as actuators for changing the environment. Autonomy implies that each agent has the freedom to make their own decisions. Rational agents are those agents that decide to execute actions that are in their “best interests” according to their desires, using a model of those desires on which they make those decisions. Action selection is complicated due to uncertainty when operating in a dynamic environment or where other actors (agents) can also influence the environment. This dissertation presents an action selection framework and algorithms that are (1) rational with respect to multiple desires and (2) responsive with respect to changing desires. Agents can use the concept of commitments, and the subsequent communication of those commitments, to coordinate their actions and reduce their uncertainty. Coordination is layered on top of this framework by describing and analyzing how commitments affect the agents’ desires in their action selection models. This research uses the domain of UAV surveillance to experimentally explore the balance between under-commitment and over-commitment. Where previous approaches concentrate on the semantics of commitment, this research concentrates on the pragmatics of commitment, describing how to use utility calculations to enable an agent to decide when making a commitment is in its best interests. / text

The basal ganglia (BG) are a group of nuclei in the basal forebrain critical in movement, goal directed behaviour and action selection. Cortical projections to the largest BG nucleus, the striatum, are highly important in theories of BG function. Therefore, we have investigated the role of striatal neurons in modulating the activity of corticostriatal synapses. In an in-vitro preparation of rodent brain slices, we conducted whole-cell patch clamp recordings of single and pairs of striatal neurons and recorded responses of medium spiny neurons (MSNs) to stimulation of corticostriatal fibres. In the presence of opioid, GABAA, NK1 and cholinergic receptor antagonists, antidromic stimulation of a population of MSNs (5 stims, 50 Hz) caused suppression of subsequently evoked EPSPs in MSNs. This suppression was dependent upon the interval between antidromic MSN stimulation and the stimulation of evoked EPSPs; suppression was larger at 500 ms intervals than at 1 or 2 s intervals. These effects were completely blocked by the GABAB antagonist CGP 52432. Bursts of evoked action potentials (5 APs, 50 Hz) in a single MSN were insufficient to cause these effects in a nearby MSN. Similar spikes in single fast spiking interneurons and low threshold spiking interneurons (LTSIs) were also insufficient. Conversely, single neurogliaform interneurons (NGFIs) could suppress evoked EPSPs in nearby MSNs in a GABAB-dependent manner. This suppression was more likely in NGFI-MSN pairs that exhibited direct GABAergic interactions. We also tested long depolarisations in LTSIs, a protocol that preferentially releases NO, which was shown to suppress evoked EPSPs through a non-GABAergic mechanism. Finally, we tested the application of exogenous NPY to slices, which also inhibited corticostriatal transmission. These results provide the first demonstration of how GABAB receptors at corticostriatal synapses are activated by endogenous GABA released by striatal neurons. They also reveal novel mechanisms through which striatal factors influence these synapses.

The Pogamut toolkit designed for rapid prototyping of computer game agents has been so far used for prototyping the agents based on 3D FPS Unreal Tournament 2004 and its sequels. After the environment of RTS Defcon was connected to Pogamut a question arose how difficult it would be to connect some other significantly different environments and action selection mechanisms. In order to test this flexibility of Pogamut we have interconnected it with more complex RTS video game StarCraft:Brood War and large neural network simulator Emergent, together with Jason and POSH action selection mechanisms. The work analyzes created connections to detail and demonstrates their functionality on examples. An integral part of the work is also web with video tutorials and guides. In this work we also analyze Pogamut's readiness to be connected to other environments.

No description available.

Schizophrenia remains an enigmatic disorder with unclear neuropathology. Recent advances in neuroimaging and genetic research suggest alterations in glutamate-dopamine interactions adversely affecting synaptic plasticity both intracortically and subcortically. Relating these changes to the manifestation of symptoms presents a great challenge, requiring a constrained framework to capture the most salient elements. Here, a biologically-grounded computational model of basal ganglia-mediated action selection was used to explore two pathological processes that hypothetically underpin schizophrenia. These were a drop in the efficiency of cortical transmission, reducing both the signal-to-noise ratio (SNR) and overall activity levels; and an excessive compensatory upregulation of subcortical dopamine release. It was proposed that reduced cortical efficiency was the primary process, which led to a secondary disinhibition of subcortical dopamine release within the striatum. This compensation was believed to partly recover lost function, but could then induce disorganised-type symptoms - summarised as selection ”Instability” - if it became too pronounced. This overcompensation was argued to be countered by antipsychotic medication. The model’s validity was tested during an fMRI (functional magnetic resonance imaging) study of 16 healthy volunteers, using a novel perceptual decision-making task, and was found to provide a good account for pallidal activation. Its account for striatum was developed and improved with a small number of principled model modifications: the inclusion of fast spiking interneurons within striatum, and their inhibition by the basal ganglia’s key regulatory nucleus, external globus pallidus. A key final addition was the explicit modelling of dopaminergic midbrain, which is dynamically regulated by both cortex and the basal ganglia. This enabled hypotheses concerning the effects of cortical inefficiency, compensatory dopamine release and medication to be directly tested. The new model was verified with a second set of 12 healthy controls. Its pathological predictions were compared to data from 12 patients with schizophrenia. Model simulations suggested that Instability went hand-in-hand with cortical inefficiency and secondary dopamine upregulation. Patients with high Instability scores showed a loss of SNR within decision-related cortex (consistent with cortical inefficiency); an exaggerated response to task demands within substantia nigra (consistent with dopaminergic upregulation); and had an improved fit to simulated data derived from increasingly cortically-inefficient models. Simulations representing the healthy state provided a good account for patients’ motor putamen, but only cortically-inefficient simulations representing the ill state provided a fit for ventral-anterior striatum. This fit improved as the simulated model became more medicated (increased D2 receptor blockade). The relative improvement of this account correlated with patients’ medication dosage. In summary, by distilling the hypothetical neuropathology of schizophrenia into two simplified umbrella processes, and using a computational model to consider their effects within action selection, this work has successfully related patients’ fMRI activation to particular symptomatology and antipsychotic medication. This approach has the potential to improve patient care by enabling a neurobiological appreciation of their current illness state, and tailoring their medication level appropriately.

No description available.

Les systèmes sensorimoteurs autonomes, placés dans des environnements dynamiques, doivent répondre continuellement à la question ultime: comment contrôler les commandes motrices à partir des entrées sensorielles? Répondre à cette question est un problème très complexe, principalement à cause de l'énorme quantité d'informations qui doit être traitée, tout en respectant plusieurs restrictions: contraintes de temps réel, espace mémoire restreint, et capacité de traitement des données limitée. Un autre défi majeur consiste à traiter l'information incomplète et imprécise, habituellement présente dans des environnements dynamiques. Cette thèse s'intéresse au problème posé par la commande des systèmes sensorimoteurs autonomes et propose un enchaînement d'hypothèses et de simplifications. Ces hypothèses et simplifications sont définies dans un cadre mathématique précis et strict appelé programmation bayésienne, une extension des réseaux bayésiens. L'enchaînement se présente en cinq paliers: utilisation d'états internes; les hypothèses de Markov de premier ordre, de stationnarité et les filtres bayésiens; exploitation de l'indépendance partielle entre les variables d'état; addition d'un mécanisme de choix de comportement;la focalisation de l'attention guidée par l'intention de comportement. La description de chaque étape est suivie de son analyse selon les exigences de mémoire, de complexité de calcul, et de difficulté de modélisation. Nous présentons également des discussions approfondies concernant d'une part la programmation des robots et d'autre part les systèmes cognitifs. Enfin, nous décrivons l'application de ce cadre de programmation à un robot mobile.

Behaviour is usually assumed to depend on the reach of a critical intensitytermed reactivity thresholdby its motivation. This view represents a simple, predictive theoretical framework in ethology and psychology. However, it is here argued that only the influence of an isolated motivation on behaviour can be explained that way; that such a view fails to account for behaviour when several motivations are jointly activated. Upon analysis, the 'classical' theory of behaviour proves to be under-specified and thus leads to three conceptual problems that make it logically inconsistent for the study of multiple motivations. A revision of the 'classical' théory, called Anticipatory Dynamics Model (ADM), is then developed in order to bring a theoretical solution to these conceptual problems. The ADM hypothesizes that an organisms motivational interactions are due to the limitation of the organisms attentional resources. Several experiments are designed to test major predictions of the ADM. Tout comportement est censé dépendre du fait que sa motivation atteint une valeur critique d'intensité appelée seuil de réactivité. Cette approche constitue un cadre théorique simple et prédictif aussi bien en éthologie qu'en psychologie. Cependant, je tente de montrer que seule l'influence d'une motivation isolée sur le comportement peut être expliquée de cette façon ; que cette approche cesse d'être applicable lorsque plusieurs motivations sont activées simultanément. Après analyse, cette théorie 'classique' s'avère sous-spécifiée et conduit à trois problèmes conceptuels qui la rendent logiquement inconsistante pour l'étude des motivations multiples. Une révision de cette théorie 'classique', appelée Modèle de Dynamique Anticipatoire (MDA), est ainsi développée en vue d'apporter une solution théorique à ces problèmes conceptuels. Le MDA fait l'hypothèse que les interactions motivationnelles d'un organisme sont causées par la limitation de ses ressources attentionnelles. Plusieurs expériences ont été imaginées pour tester des prédictions importantes du MDA.

Μελέτη της υπόθεσης ότι τα βασικά γάγγλια, μία κεντρική δομή του εγκεφάλου, συμμετέχει ενεργά στην επιλογή της κατάλληλης δράσης. Προσπάθεια μοντελοποίησης της λειτουργίας των βασικών γαγγλίων κια προσομοίωση παθολογικών καταστάσεων του ανθρώπινου εγκεφάλου. / Stydy of the hypothesis that the basal ganglia, a major structure in the mamalian brain, plays a crusial role in the selection of the appropriate action. In this direction we try to simulate the functionality of the basal ganglia as well the pathophysiological conditions (Parkinson, Huntington disease) related to them.