The work presented in this dissertation investigates techniques for robot Learning from Demonstration (LFD). LFD is a well established approach where the robot is to learn from a set of demonstrations. The dissertation focuses on LFD where a human teacher demonstrates a behavior by controlling the robot via teleoperation. After demonstration, the robot should be able to reproduce the demonstrated behavior under varying conditions. In particular, the dissertation investigates techniques where previous behavioral knowledge is used as bias for generalization of demonstrations. The primary contribution of this work is the development and evaluation of a semi-reactive approach to LFD called Predictive Sequence Learning (PSL). PSL has many interesting properties applied as a learning algorithm for robots. Few assumptions are introduced and little task-specific configuration is needed. PSL can be seen as a variable-order Markov model that progressively builds up the ability to predict or simulate future sensory-motor events, given a history of past events. The knowledge base generated during learning can be used to control the robot, such that the demonstrated behavior is reproduced. The same knowledge base can also be used to recognize an on-going behavior by comparing predicted sensor states with actual observations. Behavior recognition is an important part of LFD, both as a way to communicate with the human user and as a technique that allows the robot to use previous knowledge as parts of new, more complex, controllers. In addition to the work on PSL, this dissertation provides a broad discussion on representation, recognition, and learning of robot behavior. LFD-related concepts such as demonstration, repetition, goal, and behavior are defined and analyzed, with focus on how bias is introduced by the use of behavior primitives. This analysis results in a formalism where LFD is described as transitions between information spaces. Assuming that the behavior recognition problem is partly solved, ways to deal with remaining ambiguities in the interpretation of a demonstration are proposed. The evaluation of PSL shows that the algorithm can efficiently learn and reproduce simple behaviors. The algorithm is able to generalize to previously unseen situations while maintaining the reactive properties of the system. As the complexity of the demonstrated behavior increases, knowledge of one part of the behavior sometimes interferes with knowledge of another parts. As a result, different situations with similar sensory-motor interactions are sometimes confused and the robot fails to reproduce the behavior. One way to handle these issues is to introduce a context layer that can support PSL by providing bias for predictions. Parts of the knowledge base that appear to fit the present context are highlighted, while other parts are inhibited. Which context should be active is continually re-evaluated using behavior recognition. This technique takes inspiration from several neurocomputational models that describe parts of the human brain as a hierarchical prediction system. With behavior recognition active, continually selecting the most suitable context for the present situation, the problem of knowledge interference is significantly reduced and the robot can successfully reproduce also more complex behaviors.

The work presented in this dissertation investigates techniques for robot Learning from Demonstration (LFD). LFD is a well established approach where the robot is to learn from a set of demonstrations. The dissertation focuses on LFD where a human teacher demonstrates a behavior by controlling the robot via teleoperation. After demonstration, the robot should be able to reproduce the demonstrated behavior under varying conditions. In particular, the dissertation investigates techniques where previous behavioral knowledge is used as bias for generalization of demonstrations. The primary contribution of this work is the development and evaluation of a semi-reactive approach to LFD called Predictive Sequence Learning (PSL). PSL has many interesting properties applied as a learning algorithm for robots. Few assumptions are introduced and little task-specific configuration is needed. PSL can be seen as a variable-order Markov model that progressively builds up the ability to predict or simulate future sensory-motor events, given a history of past events. The knowledge base generated during learning can be used to control the robot, such that the demonstrated behavior is reproduced. The same knowledge base can also be used to recognize an on-going behavior by comparing predicted sensor states with actual observations. Behavior recognition is an important part of LFD, both as a way to communicate with the human user and as a technique that allows the robot to use previous knowledge as parts of new, more complex, controllers. In addition to the work on PSL, this dissertation provides a broad discussion on representation, recognition, and learning of robot behavior. LFD-related concepts such as demonstration, repetition, goal, and behavior are defined and analyzed, with focus on how bias is introduced by the use of behavior primitives. This analysis results in a formalism where LFD is described as transitions between information spaces. Assuming that the behavior recognition problem is partly solved, ways to deal with remaining ambiguities in the interpretation of a demonstration are proposed. The evaluation of PSL shows that the algorithm can efficiently learn and reproduce simple behaviors. The algorithm is able to generalize to previously unseen situations while maintaining the reactive properties of the system. As the complexity of the demonstrated behavior increases, knowledge of one part of the behavior sometimes interferes with knowledge of another parts. As a result, different situations with similar sensory-motor interactions are sometimes confused and the robot fails to reproduce the behavior. One way to handle these issues is to introduce a context layer that can support PSL by providing bias for predictions. Parts of the knowledge base that appear to fit the present context are highlighted, while other parts are inhibited. Which context should be active is continually re-evaluated using behavior recognition. This technique takes inspiration from several neurocomputational models that describe parts of the human brain as a hierarchical prediction system. With behavior recognition active, continually selecting the most suitable context for the present situation, the problem of knowledge interference is significantly reduced and the robot can successfully reproduce also more complex behaviors.

La dopamine est un neurotransmetteur important dans le fonctionnement des ganglions de la base, région du cerveau impliquée dans la fonction motrice et l’apprentissage. Un dérèglement de la dynamique de la dopamine peut être à l’origine de différentes pathologies neurologiques, telles que la maladie de Parkinson et le trouble de déficit de l’attention avec ou sans hyperactivité (TDAH). La lévodopa, un précurseur de la dopamine, est utilisée pour réduire les symptômes associés à la maladie de Parkinson, sans action directe sur ses causes. La lévodopa est très efficace au début de la maladie, mais la durée de son effet ainsi que son index thérapeutique diminuent avec la progression de la dénervation induite par la maladie. Ces changements compliquent considérablement l’optimisation des régimes posologiques. Le méthylphénidate, quant à lui, est administré pour réduire les symptômes du TDAH et agit entre autres en bloquant la recapture de la dopamine. Bien que les données confirment une certaine implication de la dopamine dans le TDAH, son étiologie exacte demeure inconnue. Peu d’études ont cerné l’effet de la lévodopa sur le système dopaminergique des ganglions de la base et son évolution avec la progression de la maladie. Aussi, bien que le TDAH ait suscité beaucoup d’intérêt, rares sont les études quantitatives de nature mécanistiques sur le sujet. L’approche de modélisation mathématique utilisée dans cette thèse s’inscrit dans un effort global visant l’optimisation de la lévodopa et du méthylphénidate, appuyé par l’élucidation des mécanismes impliqués dans la maladie de Parkinson et dans le TDAH. En adoptant une approche de pharmacologie quantitative des systèmes (QSP), nous avons développé un modèle intégratif du système dopaminergique des ganglions de la base, avec l’objectif d’élucider les mécanismes impliqués, d’évaluer l’impact de la dopamine chez dessujets souffrant de Parkinson ou de TDAH, et recevant ou non un traitement, et enfin de guider objectivement l’exercice d’optimisation des régimes posologiques. À notre connaissance, c’est le premier cadre unificateur de modélisation qui s’adresse à ces deux pathologies. Le modèle développé dans cette thèse est composé de trois sous-modèles : le premier décrit la pharmacocinétique du médicament concerné, soit la lévodopa ou le méthylphénidate ; le deuxième exprime mathématiquement les différents mécanismes impliqués dans la dynamique de la dopamine ; le troisième représente la complexité de la neurotransmission dans les ganglions de la base. Avec des adaptations appropriées, nous avons appliqué ce même modèle au contexte de la maladie de Parkinson et au TDAH, ainsi qu’à leurs thérapies respectives. Pour représenter physiologiquement la maladie de Parkinson, nous avons intégré dans le modèle l’évolution de la perte neuronale ainsi que les différents mécanismes de compensation qui en résultent. La fréquence de tapotement des doigts est utilisée comme mesure clinique de la bradykinésie, définie comme le ralentissement des mouvements chez les patients parkinsoniens. Le modèle développé se base sur les connaissances actuelles de la pathophysiologie et pharmacologie du Parkinson, assurant ainsi sa validité en comparaison à des observations expérimentales et cliniques. Ensuite, à l’aide de ce modèle, les relations non-linéaires entre la concentration plasmatique de lévodopa, la concentration en dopamine dans le cerveau et la réponse à une tâche motrice sont étudiées. Le rétrécissement de l’index thérapeutique de la lévodopa au cours de la progression de la maladie dû à ces non-linéarités est investigué. Enfin, pour assurer l’aspect translationnel de notre approche, nous avons développé une application web à laquelle ce modèle a été intégré. Cette application sert de preuve de concept à un outil facilitant l’optimisation et l’individualisation des régimes posologiques. Pour l’étude du TDAH, nous avons adapté le modèle du système dopaminergique en y intégrant la libération tonique et phasique de la dopamine, cette dernière se produisant durant une tâche d’apprentissage par renforcement. Des individus virtuels ont été créés avec et sans déséquilibre du ratio tonique/phasique de la dopamine. En simulant une tâche de réponse à des stimuli dans un contexte de déséquilibre de la dopamine, le modèle nous a permis d’observer des symptômes similiaires à ceux de patients réels souffrant de TDAH. Finalement, la réponse au méthylphénidate résultant de l’inhibition de la recapture de la dopamine, à travers différents scénarios d’apprentissage a aussi été étudiée. Le développement d’une métrique nous a permis de différencier les répondants des non-répondants, et ainsi de mettre en évidence l’implication possible d’un apprentissage excessif chez les nonrépondants. Une meilleure compréhension de la réponse au méthylphénidate permettrait d’éviter la surmédication chez les non-répondants et d’aider les cliniciens dans leur pratique. Malgré la complexité du système dopaminergique et des traitements associés, cette thèse est un pas en avant dans la compréhension des mécanismes sous-jacents et de leur implication dans la thérapie. Ces avancées ont été réalisées en adoptant une approche de pharmacologie quantitative des systèmes, associée à une modélisation neurocomputationnelle du domaine du génie électrique, et complétée par un aspect de transfert au chevet du patient. Ce n’est qu’en transcendant ainsi les frontières disciplinaires qu’une visée aussi globale et intégrative est possible, afin de faire face aux défis multidimensionnels du système de la santé. / Dopamine is an important neurotransmitter of the basal ganglia, a region of the brain involved in motor function and learning. Disruption of dopamine dynamics can cause various neurological conditions, such as Parkinson’s disease and attention deficit hyperactivity disorder (ADHD). Levodopa, a dopamine precursor, is used to reduce the symptoms associated with Parkinson’s disease, without directly alleviating its causes. Levodopa is very effective in the early stages of the disease, but its effect duration along with its therapeutic index decrease with disease-induced denervation. These modifications further challenge determination of optimal dosing regimens of levodopa. In the case of ADHD, methylphenidate is administered to reduce its symptoms by, among other things, blocking dopamine recapture. Although evidence supports involvement of dopamine in ADHD, its exact etiology remains unknown. Few studies have investigated the effect of levodopa on the basal ganglia dopaminergic system and how it evolves with disease progression. Also, although ADHD has received a lot of interest, few quantitative studies of a mechanistic nature have been conducted on the subject. The mathematical modeling approach used in this thesis is part of an overall effort to optimize levodopa and methylphenidate, supported by the elucidation of the mechanisms involved in Parkinson’s disease and ADHD. Using a quantitative systems pharmacology (QSP) approach, we have developed an integrative model of the basal ganglia dopaminergic system, with the objective of elucidating the mechanisms involved, assessing the impact of dopamine in subjects with Parkinson’s or ADHD, with and without treatment, and objectively guiding the dosing regimens optimization. To the best of our knowledge, this is the first unifying modeling framework that addresses at the same time these two pathologies and their therapies. The model developed in this thesis includes three sub-models: the first one describes the drug pharmacokinetics, either levodopa or methylphenidate; the second one translates mathematically the different mechanisms involved in the dopamine dynamics; the third one is a computational representation of the complexity of neurotransmission in the basal ganglia. With appropriate adaptations, we have applied this same model to the context of Parkinson’s disease and ADHD, as well as to their respective pharmacotherapies. In order to physiologically represent Parkinson’s disease, we have integrated the denervation process in the model as well as the resulting compensation mechanisms. The finger tapping frequency is used as a clinical endpoint of bradykinesia, defined as the slowing of movements. The developed model is based on up-to-date knowledge of the pathophysiology and pharmacology of Parkinson’s disease, thus ensuring its validity in comparison with experimental and clinical observations. Using this model, the non-linear relationships between plasma levodopa concentration, dopamine concentration in the brain and response to a motor task were studied. The narrowing of levodopa therapeutic index during the progression of the disease due to these non-linearities was investigated. Finally, to ensure the translational aspect of our approach, we developed a web application in which this model was integrated. This application serves as a proof of concept for a tool aimed to facilitate the optimization and individualization of dosing regimens. For the study of ADHD, we adapted the developed model by integrating tonic and phasic dopamine release, the latter occurring during a reinforcement learning task. Virtual individuals were created with and without dopamine imbalance in the tonic/phasic ratio. By simulating a stimulus-response task, we observe ADHD-like symptoms among virtual patients with dopamine imbalance. Finally, the response to methylphenidate resulting from dopamine recapture inhibition, through different learning scenarios, was also studied. The development of a metric allowed us to differentiate responders from non-responders, and thus to highlight the possible implication of excessive learning in non-responders. A better understanding of methylphenidate response would help avoid overmedication in non-responders and assist clinicians in their practice. Despite the complexity of the dopaminergic system and its associated therapies, this thesis is a step forward in understanding the underlying mechanisms and their involvement in pharmacotherapy. These advances were achieved by adopting a quantitative systems pharmacology approach, combined with neurocomputational modeling borrowed from the electrical engineering field, and complemented by a translational bedside aspect. It is only by transcending disciplinary boundaries and adopting such an integrative approach that this ultimate goal of having a real impact on the multifaceted health system is possible.