Facial motion provides an abundance of information necessary for mediating social communication. Emotional expressions, head rotations and eye-gaze patterns allow us to extract categorical and qualitative information from others (Blake & Shiffrar, 2007). Autism Spectrum Disorder (ASD) is a neurodevelopmental condition characterised by a severe impairment in social cognition. One of the causes may be related to a fundamental deficit in perceiving human movement (Herrington et al., 2007). This hypothesis was investigated more closely within the current thesis. In neurotypical controls, the visual processing of facial motion was analysed via EEG alpha waves. Participants were tested on their ability to discriminate between successive animations (exhibiting rigid and nonrigid motion). The appearance of the stimuli remained constant over trials, meaning decisions were based solely on differential movement patterns. The parieto-occipital region was specifically selective to upright facial motion while the occipital cortex responded similarly to natural and manipulated faces. Over both regions, a distinct pattern of activity in response to upright faces was characterised by a transient decrease and subsequent increase in neural processing (Girges et al., 2014). These results were further supported by an fMRI study which showed sensitivity of the superior temporal sulcus (STS) to perceived facial movements relative to inanimate and animate stimuli. The ability to process information from dynamic faces was assessed in ASD. Participants were asked to recognise different sequences, unfamiliar identities and genders from facial motion captures. Stimuli were presented upright and inverted in order to assess configural processing. Relative to the controls, participants with ASD were significantly impaired on all three tasks and failed to show an inversion effect (O'Brien et al., 2014). Functional neuroimaging revealed atypical activities in the visual cortex, STS and fronto-parietal regions thought to contain mirror neurons in participants with ASD. These results point to a deficit in the visual processing of facial motion, which in turn may partly cause social communicative impairments in ASD.

Human driving decisions are the leading cause of road fatalities. Autonomous driving naturally eliminates such incompetent decisions and thus can improve traffic safety and efficiency. Deep reinforcement learning (DRL) has shown great potential in learning complex tasks. Recently, researchers investigated various DRL-based approaches for autonomous driving. However, exploiting multi-modal fusion to generate pixel-wise perception and motion prediction and then leveraging these predictions to train a latent DRL has not been targeted yet. Unlike other DRL algorithms, the latent DRL algorithm distinguishes representation learning from task learning, enhancing sampling efficiency for reinforcement learning. In addition, supplying the latent DRL algorithm with accurate perception and motion prediction simplifies the surrounding urban scenes, improving training and thus learning a better driving policy. To that end, this Ph.D. research initially develops LiCaNext, a novel real-time multi-modal fusion network to produce accurate joint perception and motion prediction at a pixel level. Our proposed approach relies merely on a LIDAR sensor, where its multi-modal input is composed of bird's-eye view (BEV), range view (RV), and range residual images. Further, this Ph.D. thesis proposes leveraging these predictions with another simple BEV image to train a sequential latent maximum entropy reinforcement learning (MaxEnt RL) algorithm. A sequential latent model is deployed to learn a more compact latent representation from high-dimensional inputs. Subsequently, the MaxEnt RL model trains on this latent space to learn a driving policy. The proposed LiCaNext is trained on the public nuScenes dataset. Results demonstrated that LiCaNext operates in real-time and performs better than the state-of-the-art in perception and motion prediction, especially for small and distant objects. Furthermore, simulation experiments are conducted on CARLA to evaluate the performance of our proposed approach that exploits LiCaNext predictions to train sequential latent MaxEnt RL algorithm. The simulated experiments manifest that our proposed approach learns a better driving policy outperforming other prevalent DRL-based algorithms. The learned driving policy achieves the objectives of safety, efficiency, and comfort. Experiments also reveal that the learned policy maintains its effectiveness under different environments and varying weather conditions.

Cette thèse s'est intéressée aux relations entre les mouvements biologiques et la perception sonore en considérant le cas spécifique des mouvements graphiques et des sons de frottement qu'ils génèrent. L'originalité de ces travaux réside dans l'utilisation d'un modèle de synthèse sonore basé sur un principe perceptif issu de l'approche écologique de la perception et contrôlé par des modèles de gestes. Des stimuli sonores dont le timbre n'est modulé que par des variations de vitesse produites par un geste ont ainsi pu être générés permettant de se focaliser sur l'influence perceptive de cet invariant transformationel. Une première étude a ainsi montré que l'on reconnait la cinématique des mouvements biologiques (la loi en puissance 1/3), et que l'on peut discriminer des formes géométriques simples juste à partir des sons de frottement produits. Une seconde étude a montré l'existence de prototypes dynamiques sonores caractérisant les trajectoires elliptiques, mettant ainsi en évidence que les prototypes géométriques peuvent émerger d'un couplage sensorimoteur. Enfin, une dernière étude a montré qu'une cinématique évoquée par un sonore influence significativement la cinématique et la géométrie d'un geste dans une tâche de reproduction graphique du mouvement d'un point lumineux. Ce résultat révèle l'importance de la modalité auditive dans l'intégration multisensorielle des mouvements continus dans une situation jamais explorée. Ces résultats ont permis le contrôle de modèles de synthèse par des descriptions gestuelles et la création d'outils de sonification pour l'apprentissage de gestes et la réhabilitation d'une pathologie motrice, la dysgraphie. / This thesis focused on the relations between biological movements and auditory perception in considering the specific case of graphical movements and the friction sounds they produced. The originality of this work lies in the use of sound synthesis processes that are based on a perceptual paradigm and that can be controlled by gesture models. The present synthesis model made it possible to generate acoustic stimuli which timbre was directly modulated by the velocity variations induced by a graphic gesture in order to exclusively focus on the perceptual influence of this transformational invariant. A first study showed that we can recognize the biological motion kinematics (the 1/3 power law) and discriminate simple geometric shapes simply by listening to the timbre variations of friction sounds that solely evoke velocity variations. A second study revealed the existence of dynamic prototypes characterized by sounds corresponding to the most representative elliptic trajectory, thus revealing that prototypical shapes may emerged from sensorimotor coupling. A final study showed that the kinematics evoked by friction sounds may significantly affect the dynamic and geometric dimension in the visuo-motor coupling. This shed critical light on the relevance of auditory perception in the multisensory integration of continuous motion in a situation never explored. All of these theoretical results enabled the gestural control of sound synthesis models from a gestural description and the creation of sonification tools for gesture learning and rehabilitation of a graphomotor disease, dysgraphia.

Thèse numérisée par la Direction des bibliothèques de l'Université de Montréal. / Parmi la multitude de fonctions que le système visuel effectue, la perception du mouvement est l'une des plus importantes. Il a été clairement démontré qu'il existe des sites cérébraux spécifiques pour la détection, l'analyse et l'intégration du mouvement. De façon classique, les mécanismes neurophysiologiques qui sous-tendent ces processus sont attribués aux aires corticales. Le thalamus quant à lui, est généralement considéré comme un « relais passif », c'est à dire qui transmet l'information sensorielle vers le cortex sans modifier le signal entrant. Le but de ce projet sera de préciser les mécanismes nerveux impliqués dans la perception et l'intégration du mouvement et plus précisément, la contribution des régions cérébrales sous-corticales et corticales intimement liés par des connexions réciproques. Ces régions sont: le complexe LP-pulvinar, situé dans le thalamus, l'aire extra-striée postero-médiane suprasylvienne (PMLS) et le cortex ectosylvien visuel antérieur (AEV); deux régions corticales ayant un rôle spécialisé dans l'analyse du mouvement. Les expériences ont été réalisées sur des chats adultes normaux anesthésiés. Une microélectrode d'enregistrement a été descendue dans ces trois sites afin d'enregistrer l'activité des neurones. Les réponses neuronales à des réseaux sinusoïdaux, des patrons texturés (« bruit visuel ») et des « plaids » ont été caractérisé pour étudier les mécanismes qui sous-tendent l'intégration du mouvement au niveau cellulaire. Afin de caractériser d'avantage ce lien, l'influence des aires corticales sur les propriétés thalamiques a été déterminée par inactivation locale réversible (i.e. micro-injection de l'acide y-aminobutyrique; GABA) ou par inactivation permanente plus vaste (i.e. ablation chirurgicale). Les résultats de cette étude se résument comme suit : 1) les propriétés des réponses neuronales du PMLS au bruit visuel sont similaires à celles du LP-pulvinar. Ce résultat suggère que les processus d'analyse impliquant une boucle cortico-thalamique PMLS-LP sont comparables au niveau cortical et sous-cortical. 2) les neurones du PMLS et du LP peuvent coder le mouvement relatif entre un objet et son arrièreplan. De plus, l'inactivation réversible du LP perturbe ces réponses au niveau du PMLS. Ces résultats sont essentiels dans l'établissement d'un lien fonctionnel entre ces deux régions quant à l'analyse de certains aspects du mouvement. 3) certains neurones du complexe LP-pulvinar sont capables d'intégrer l'information directionnelle telle que définie par des « plaids ». Ceci constitue la première démonstration de propriétés de haut-niveau en dehors du cortex. De plus, cette découverte suggère que le LP-pulvinar participe de façon parallèle et en coopération avec le cortex dans l'analyse de scènes visuelles complexes via l'exploitation des boucles cortico-thalamiques. Les résultats de cette étude sont importants non seulement pour appuyer des notions théoriques novatrices sur le rôle du thalamus, mais aussi dans le but de réévaluer et de préciser les mécanismes nerveux qui sous-tendent la perception du mouvement et l'intégration sensorielle en général. / Among the multitude of functions the visual system carries out, the perception of motion is one of the most important. It has been clearly demonstrated that the visual system contains numerous specialised areas implicated in the detection, analysis, and integration of motion. Classically, the neurophysiological mechanisms underlying these processes have been uniquely attributed to regions of the cerebral cortex. The thalamus for its part, has generally been regarded as a passive relay transferring information to the cortex without any modification of the sensory signal. The purpose of this study is to investigate the neurophysiological mechanisms implicated in the perception and integration of motion and more specifically, delineate the contribution of cortical and subcortical structures that are intimately related via reciprocal connections. These areas are: the LP-pulvinar complex; located in the thalamus, and the extrastriate posteromedial lateral suprasylvian (PMLS) and anterior ectosylvian visual (AEV) cortical areas; two regions whose role in motion analysis are well established. Experiments were carried out on normal adult anaesthetised cats. A recording microelectrode was descended in one of the aforementioned areas to record neuronal activity. Neuronal responses to drifting sine-wave gratings, moving texture patterns ("visual noise"), and "plaid patterns" were recorded in order to investigate the mechanisms underlying the integration of motion information at the neuronal level. As a continuation of the study, the influence of cortical motion areas on recorded thalannic responses will be determined by local reversible deactivation (i.e. microinjection of y-aminobutyric acid; GABA) or by irreversible deactivation (i.e. surgical ablation). The results of the study are as follows: 1) Response properties of PMLS neurons to moving texture patterns are similar to those found in the LP-pulvinar connplex. These results suggest that motion processing along both components of the PMLS-LP cortico-thalamic loop is carried out within a similar envelope of analysis. 2) Neurons in both PMLS and LP are able to code the relative motion of an object with respect to its background. Furthermore, reversible deactivation of LP disrupts these responses in PMLS. These results are important in establishing that both these areas are functionally linked in the analysis of specific aspects of motion. 3) The fact that pattern-selective responses to moving plaids can be found in the LP-pulvinar complex suggests that this area is capable of carrying out higher-order motion computations. The importance of this later results is two-fold. First, these findings represent the first demonstration that higher-order properties exists outside extrastriate cortical areas. Second, they further suggest that certain thalamic nuclei, via the establishment of cortico-thalamic loops, participate in parallel and in co-operation with the cortex in the analysis of complex visual scenes. The results of this study are important not only to reinforce current and novel theoretical notions regarding the role of the thalamus, but also in the re-evaluation of the neurophysiological mechanisms involved in motion perception and sensory integration as a whole.