The grounding of language in humanoid robots is a fundamental problem, especially in social scenarios which involve the interaction of robots with human beings. Indeed, natural language represents the most natural interface for humans to interact and exchange information about concrete entities like KNIFE, HAMMER and abstract concepts such as MAKE, USE. This research domain is very important not only for the advances that it can produce in the design of human-robot communication systems, but also for the implication that it can have on cognitive science. Abstract words are used in daily conversations among people to describe events and situations that occur in the environment. Many scholars have suggested that the distinction between concrete and abstract words is a continuum according to which all entities can be varied in their level of abstractness. The work presented herein aimed to ground abstract concepts, similarly to concrete ones, in perception and action systems. This permitted to investigate how different behavioural and cognitive capabilities can be integrated in a humanoid robot in order to bootstrap the development of higher-order skills such as the acquisition of abstract words. To this end, three neuro-robotics models were implemented. The first neuro-robotics experiment consisted in training a humanoid robot to perform a set of motor primitives (e.g. PUSH, PULL, etc.) that hierarchically combined led to the acquisition of higher-order words (e.g. ACCEPT, REJECT). The implementation of this model, based on a feed-forward artificial neural networks, permitted the assessment of the training methodology adopted for the grounding of language in humanoid robots. In the second experiment, the architecture used for carrying out the first study was reimplemented employing recurrent artificial neural networks that enabled the temporal specification of the action primitives to be executed by the robot. This permitted to increase the combinations of actions that can be taught to the robot for the generation of more complex movements. For the third experiment, a model based on recurrent neural networks that integrated multi-modal inputs (i.e. language, vision and proprioception) was implemented for the grounding of abstract action words (e.g. USE, MAKE). Abstract representations of actions ("one-hot" encoding) used in the other two experiments, were replaced with the joints values recorded from the iCub robot sensors. Experimental results showed that motor primitives have different activation patterns according to the action's sequence in which they are embedded. Furthermore, the performed simulations suggested that the acquisition of concepts related to abstract action words requires the reactivation of similar internal representations activated during the acquisition of the basic concepts, directly grounded in perceptual and sensorimotor knowledge, contained in the hierarchical structure of the words used to ground the abstract action words.

No description available.

Interpretação semântica de imagens tem se mostrado uma das fronteiras mais promissoras da área de Visão Computacional, especificamente aplicada a interpretação imagens. Nas abordagens que estão sendo propostas atualmente, conhecimento visual explicitamente modelado é utilizado com algoritmos de raciocínio simbólico combinados a algoritmos de processamento de imagem a fim de se extrair o conteúdo de imagens e associá-lo a modelos semanticamente ricos. Este trabalho apresenta uma abordagem de interpretação semântica de imagens especificamente voltada para interpretação de gráficos de linhas, chamada S-Chart. Ela consiste um conjunto de modelos de conhecimento e algoritmos que podem ser instanciados para interpretação de gráficos em diversos domínios. Os modelos são representados em três níveis semânticos e aplicam o conceito de ancoramento simbólico (symbol grounding) para mapear as primitivas entre os níveis. Os algoritmos de interpretação propostos fazem a interação entre o raciocínio simbólico de alto nível e os algoritmos de processamento de sinal para os dados brutos dos gráficos analisados. Para demonstrar a aplicabilidade do framework S-Chart, foi desenvolvido o sistema InteliStrata, uma aplicação no domínio da Geologia, voltada para interpretação semântica de gráficos de perfis de poço. Utilizando a aplicação, foram interpretados dois perfis de raios gama capturados em poços de exploração, de modo que o sistema identificasse a presença de Sequências Estratigráficas e superfícies de inundação máximas. Os resultados foram comparados com a interpretação de um geólogo especialista sobre os mesmos dados. O sistema aponta as mesmas sequências já identificadas e oferece outras opções de interpretação compatíveis com as do geólogo utilizando os mesmos dados. O framework S-Chart tem seus pontos fortes nos seus modelos representação de conhecimento visual independentes de domínio, que permitem a utilização do mesmo arcabouço em diferentes aplicações e, em especial, no seu modelo de ancoramento simbólico entre primitivas de representação. / Semantic image interpretation is one of the most promising frontiers in the Computer Vision area, specifically when applied to Image Interpretation. To reach semantic interpretation, visual knowledge explicitly represented is applied by symbolic reasoning algorithms combined with image processing algorithms in order to extract the content of the images and associate it with semantically rich models. This work describes the S-Chart approach, a semantic image interpretation approach designed for interpretation of line charts. It is structured as a set of knowledge models and algorithms that can be instantiated to accomplish chart interpretation in other domains. The models are represented in three semantic levels and apply the concept of symbol grounding in order to map the primitives between the levels. The interpretation algorithms carry out the interaction between the symbolic reasoning in the high level, and the signal processing algorithms in the low level data. In order to demonstrate the applicability of the S-Chart framework, we developed the InteliStrata system, an application in Geology for the semantic interpretation of well log profiles. Using the application, we have interpreted the graphs of two gamma-ray profiles captured in exploration wells, to indicate the position of Stratigraphic Sequences and the maximum flooding surfaces. The results were compared with the interpretation of an experienced geologist using the same data input. The system was able to point the same identified sequences and offered alternative interpretation that were compatible with the geologist interpretation over the data. The S-Chart framework demonstrates its effectiveness on interpretation of pictorial information in knowledge intensive domains. The stronger points of the approach are its domain independent models for visual knowledge representation and, specially, the application of a symbol grounding model to provide a correlation between representation primitives.

Interpretação semântica de imagens tem se mostrado uma das fronteiras mais promissoras da área de Visão Computacional, especificamente aplicada a interpretação imagens. Nas abordagens que estão sendo propostas atualmente, conhecimento visual explicitamente modelado é utilizado com algoritmos de raciocínio simbólico combinados a algoritmos de processamento de imagem a fim de se extrair o conteúdo de imagens e associá-lo a modelos semanticamente ricos. Este trabalho apresenta uma abordagem de interpretação semântica de imagens especificamente voltada para interpretação de gráficos de linhas, chamada S-Chart. Ela consiste um conjunto de modelos de conhecimento e algoritmos que podem ser instanciados para interpretação de gráficos em diversos domínios. Os modelos são representados em três níveis semânticos e aplicam o conceito de ancoramento simbólico (symbol grounding) para mapear as primitivas entre os níveis. Os algoritmos de interpretação propostos fazem a interação entre o raciocínio simbólico de alto nível e os algoritmos de processamento de sinal para os dados brutos dos gráficos analisados. Para demonstrar a aplicabilidade do framework S-Chart, foi desenvolvido o sistema InteliStrata, uma aplicação no domínio da Geologia, voltada para interpretação semântica de gráficos de perfis de poço. Utilizando a aplicação, foram interpretados dois perfis de raios gama capturados em poços de exploração, de modo que o sistema identificasse a presença de Sequências Estratigráficas e superfícies de inundação máximas. Os resultados foram comparados com a interpretação de um geólogo especialista sobre os mesmos dados. O sistema aponta as mesmas sequências já identificadas e oferece outras opções de interpretação compatíveis com as do geólogo utilizando os mesmos dados. O framework S-Chart tem seus pontos fortes nos seus modelos representação de conhecimento visual independentes de domínio, que permitem a utilização do mesmo arcabouço em diferentes aplicações e, em especial, no seu modelo de ancoramento simbólico entre primitivas de representação. / Semantic image interpretation is one of the most promising frontiers in the Computer Vision area, specifically when applied to Image Interpretation. To reach semantic interpretation, visual knowledge explicitly represented is applied by symbolic reasoning algorithms combined with image processing algorithms in order to extract the content of the images and associate it with semantically rich models. This work describes the S-Chart approach, a semantic image interpretation approach designed for interpretation of line charts. It is structured as a set of knowledge models and algorithms that can be instantiated to accomplish chart interpretation in other domains. The models are represented in three semantic levels and apply the concept of symbol grounding in order to map the primitives between the levels. The interpretation algorithms carry out the interaction between the symbolic reasoning in the high level, and the signal processing algorithms in the low level data. In order to demonstrate the applicability of the S-Chart framework, we developed the InteliStrata system, an application in Geology for the semantic interpretation of well log profiles. Using the application, we have interpreted the graphs of two gamma-ray profiles captured in exploration wells, to indicate the position of Stratigraphic Sequences and the maximum flooding surfaces. The results were compared with the interpretation of an experienced geologist using the same data input. The system was able to point the same identified sequences and offered alternative interpretation that were compatible with the geologist interpretation over the data. The S-Chart framework demonstrates its effectiveness on interpretation of pictorial information in knowledge intensive domains. The stronger points of the approach are its domain independent models for visual knowledge representation and, specially, the application of a symbol grounding model to provide a correlation between representation primitives.

Interpretação semântica de imagens tem se mostrado uma das fronteiras mais promissoras da área de Visão Computacional, especificamente aplicada a interpretação imagens. Nas abordagens que estão sendo propostas atualmente, conhecimento visual explicitamente modelado é utilizado com algoritmos de raciocínio simbólico combinados a algoritmos de processamento de imagem a fim de se extrair o conteúdo de imagens e associá-lo a modelos semanticamente ricos. Este trabalho apresenta uma abordagem de interpretação semântica de imagens especificamente voltada para interpretação de gráficos de linhas, chamada S-Chart. Ela consiste um conjunto de modelos de conhecimento e algoritmos que podem ser instanciados para interpretação de gráficos em diversos domínios. Os modelos são representados em três níveis semânticos e aplicam o conceito de ancoramento simbólico (symbol grounding) para mapear as primitivas entre os níveis. Os algoritmos de interpretação propostos fazem a interação entre o raciocínio simbólico de alto nível e os algoritmos de processamento de sinal para os dados brutos dos gráficos analisados. Para demonstrar a aplicabilidade do framework S-Chart, foi desenvolvido o sistema InteliStrata, uma aplicação no domínio da Geologia, voltada para interpretação semântica de gráficos de perfis de poço. Utilizando a aplicação, foram interpretados dois perfis de raios gama capturados em poços de exploração, de modo que o sistema identificasse a presença de Sequências Estratigráficas e superfícies de inundação máximas. Os resultados foram comparados com a interpretação de um geólogo especialista sobre os mesmos dados. O sistema aponta as mesmas sequências já identificadas e oferece outras opções de interpretação compatíveis com as do geólogo utilizando os mesmos dados. O framework S-Chart tem seus pontos fortes nos seus modelos representação de conhecimento visual independentes de domínio, que permitem a utilização do mesmo arcabouço em diferentes aplicações e, em especial, no seu modelo de ancoramento simbólico entre primitivas de representação. / Semantic image interpretation is one of the most promising frontiers in the Computer Vision area, specifically when applied to Image Interpretation. To reach semantic interpretation, visual knowledge explicitly represented is applied by symbolic reasoning algorithms combined with image processing algorithms in order to extract the content of the images and associate it with semantically rich models. This work describes the S-Chart approach, a semantic image interpretation approach designed for interpretation of line charts. It is structured as a set of knowledge models and algorithms that can be instantiated to accomplish chart interpretation in other domains. The models are represented in three semantic levels and apply the concept of symbol grounding in order to map the primitives between the levels. The interpretation algorithms carry out the interaction between the symbolic reasoning in the high level, and the signal processing algorithms in the low level data. In order to demonstrate the applicability of the S-Chart framework, we developed the InteliStrata system, an application in Geology for the semantic interpretation of well log profiles. Using the application, we have interpreted the graphs of two gamma-ray profiles captured in exploration wells, to indicate the position of Stratigraphic Sequences and the maximum flooding surfaces. The results were compared with the interpretation of an experienced geologist using the same data input. The system was able to point the same identified sequences and offered alternative interpretation that were compatible with the geologist interpretation over the data. The S-Chart framework demonstrates its effectiveness on interpretation of pictorial information in knowledge intensive domains. The stronger points of the approach are its domain independent models for visual knowledge representation and, specially, the application of a symbol grounding model to provide a correlation between representation primitives.

Negation is a part of language that humans engage in pretty much from the onset of speech. Negation appears at first glance to be harder to grasp than object or action labels, yet this thesis explores how this family of ‘concepts’ could be acquired in a meaningful way by a humanoid robot based solely on the unconstrained dialogue with a human conversation partner. The earliest forms of negation appear to be linked to the affective or motivational state of the speaker. Therefore we developed a behavioural architecture which contains a motivational system. This motivational system feeds its state simultaneously to other subsystems for the purpose of symbol-grounding but also leads to the expression of the robot’s motivational state via a facial display of emotions and motivationally congruent body behaviours. In order to achieve the grounding of negative words we will examine two different mechanisms which provide an alternative to the established grounding via ostension with or without joint attention. Two large experiments were conducted to test these two mechanisms. One of these mechanisms is so called negative intent interpretation, the other one is a combination of physical and linguistic prohibition. Both mechanisms have been described in the literature on early child language development but have never been used in human-robot-interaction for the purpose of symbol grounding. As we will show, both mechanisms may operate simultaneously and we can exclude none of them as potential ontogenetic origin of negation.

This body of work explores an emerging aspect of human-robot interaction, transparency. Socially guided machine learning has proven that highly immersive robotic behaviors have yielded better results than lesser interactive behaviors for performance and shorter training time. While other work explores this transparency in learning by demonstration using non-verbal cues to point out the importance or preference users may have towards behaviors, my work follows this argument and attempts to extend it by offering cues to the internal task representation. What I show is that task-transparency, or the ability to connect and discuss the task in a fluent way implores the user to shape and correct the learned goal in ways that may be impossible by other present day learning by demonstration methods. Additionally, some participants are shown to prefer task-transparent robots which appear to have the ability of "introspection" in which it can modify the learned goal by other methods than just demonstration.

The semantic web aims to make the meaning of data on the web explicit and machine processable. Harking back to Leibniz in its vision, it imagines a world of interlinked information that computers `understand' and `know' how to process based on its meaning. Spearheaded by the World Wide Web Consortium, ontology languages OWL and RDF form the core of the current technical offerings. RDF has successfully enabled the construction of virtually unlimited webs of data, whilst OWL gives the ability to express complex relationships between RDF data triples. However, the formal semantics of these languages limit themselves to that aspect of meaning that can be captured by mechanical inference rules, leaving many open questions as to other aspects of meaning and how they might be made machine processable. The Semantic Web has faced a number of problems that are addressed by the included publications. Its germination within academia, and logical semantics has seen it struggle to become familiar, accessible and implementable for the general IT population, so an overview of semantic technologies is provided. Faced with competing `semantic' languages, such as the ISO's Topic Map standards, a method for building ISO-compliant Topic Maps in the OWL DL language has been provided, enabling them to take advantage of the more mature OWL language and tools. Supplementation with rules is needed to deal with many real-world scenarios and this is explored as a practical exercise. The available syntaxes for OWL have hindered domain experts in ontology building, so a natural language syntax for OWL designed for use by non-logicians is offered and compared with similar offerings. In recent years, proliferation of ontologies has resulted in far more than are needed in any given domain space, so a mechanism is proposed to facilitate the reuse of existing ontologies by giving contextual information and leveraging social factors to encourage wider adoption of common ontologies and achieve interoperability. Lastly, the question of meaning is addressed in relation to the need to define one's terms and to ground one's symbols by anchoring them effectively, ultimately providing the foundation for evolving a `Pragmatic Web' of action.

The semantic web aims to make the meaning of data on the web explicit and machine processable. Harking back to Leibniz in its vision, it imagines a world of interlinked information that computers `understand' and `know' how to process based on its meaning. Spearheaded by the World Wide Web Consortium, ontology languages OWL and RDF form the core of the current technical offerings. RDF has successfully enabled the construction of virtually unlimited webs of data, whilst OWL gives the ability to express complex relationships between RDF data triples. However, the formal semantics of these languages limit themselves to that aspect of meaning that can be captured by mechanical inference rules, leaving many open questions as to other aspects of meaning and how they might be made machine processable. The Semantic Web has faced a number of problems that are addressed by the included publications. Its germination within academia, and logical semantics has seen it struggle to become familiar, accessible and implementable for the general IT population, so an overview of semantic technologies is provided. Faced with competing `semantic' languages, such as the ISO's Topic Map standards, a method for building ISO-compliant Topic Maps in the OWL DL language has been provided, enabling them to take advantage of the more mature OWL language and tools. Supplementation with rules is needed to deal with many real-world scenarios and this is explored as a practical exercise. The available syntaxes for OWL have hindered domain experts in ontology building, so a natural language syntax for OWL designed for use by non-logicians is offered and compared with similar offerings. In recent years, proliferation of ontologies has resulted in far more than are needed in any given domain space, so a mechanism is proposed to facilitate the reuse of existing ontologies by giving contextual information and leveraging social factors to encourage wider adoption of common ontologies and achieve interoperability. Lastly, the question of meaning is addressed in relation to the need to define one's terms and to ground one's symbols by anchoring them effectively, ultimately providing the foundation for evolving a `Pragmatic Web' of action.

Cette thèse considère l'apprentissage de motifs récurrents dans la perception multimodale. Elle s'attache à développer des modèles robotiques de ces facultés telles qu'observées chez l'enfant, et elle s'inscrit en cela dans le domaine de la robotique développementale.Elle s'articule plus précisément autour de deux thèmes principaux qui sont d'une part la capacité d'enfants ou de robots à imiter et à comprendre le comportement d'humains, et d'autre part l'acquisition du langage. A leur intersection, nous examinons la question de la découverte par un agent en développement d'un répertoire de motifs primitifs dans son flux perceptuel. Nous spécifions ce problème et établissons son lien avec ceux de l'indétermination de la traduction décrit par Quine et de la séparation aveugle de source tels qu'étudiés en acoustique.Nous en étudions successivement quatre sous-problèmes et formulons une définition expérimentale de chacun. Des modèles d'agents résolvant ces problèmes sont également décrits et testés. Ils s'appuient particulièrement sur des techniques dites de sacs de mots, de factorisation de matrices et d'apprentissage par renforcement inverse. Nous approfondissons séparément les trois problèmes de l'apprentissage de sons élémentaires tels les phonèmes ou les mots, de mouvements basiques de danse et d'objectifs primaires composant des tâches motrices complexes. Pour finir nous étudions le problème de l'apprentissage d'éléments primitifs multimodaux, ce qui revient à résoudre simultanément plusieurs des problèmes précédents. Nous expliquons notamment en quoi cela fournit un modèle de l'ancrage de mots acoustiques / This thesis focuses on learning recurring patterns in multimodal perception. For that purpose it develops cognitive systems that model the mechanisms providing such capabilities to infants; a methodology that fits into thefield of developmental robotics.More precisely, this thesis revolves around two main topics that are, on the one hand the ability of infants or robots to imitate and understand human behaviors, and on the other the acquisition of language. At the crossing of these topics, we study the question of the how a developmental cognitive agent can discover a dictionary of primitive patterns from its multimodal perceptual flow. We specify this problem and formulate its links with Quine's indetermination of translation and blind source separation, as studied in acoustics.We sequentially study four sub-problems and provide an experimental formulation of each of them. We then describe and test computational models of agents solving these problems. They are particularly based on bag-of-words techniques, matrix factorization algorithms, and inverse reinforcement learning approaches. We first go in depth into the three separate problems of learning primitive sounds, such as phonemes or words, learning primitive dance motions, and learning primitive objective that compose complex tasks. Finally we study the problem of learning multimodal primitive patterns, which corresponds to solve simultaneously several of the aforementioned problems. We also details how the last problems models acoustic words grounding.