Human languages are not just tools for transmitting cultural ideas, they are themselves culturally transmitted. This single observation has major implications for our understanding of how and why languages around the world are structured the way they are, and also for how scientists should be studying them. Accounting for the origins of what turns out to be such a uniquely human ability is, and should be, a priority for anyone interested in what makes us different from every other lifeform on Earth. The way the scientific community thinks about language has seen considerable changes over the years. In particular, we have witnessed movements away from a purely descriptive science of language, towards a more explanatory framework that is willing to embrace the difficult questions of not just how individual languages are currently structured and used, but also how and why they got to be that way in the first place. Seeing languages as historical entities is, of course, nothing new in linguistics. Seeing languages as complex adaptive systems, undergoing processes of evolution at multiple levels of interaction however, is. Broadly speaking, this thesis explores some of the implications that this perspective on language has, and argues that in addition to furthering our understanding of the processes of biological evolution and the mechanisms of individual learning required specifically for language, we also need to be mindful of the less well-understood cultural processes that mediate between the two. Human communication systems are not just direct expressions of our genes. Neither are they independently acquired by learners anew at every generation. Instead, languages are transmitted culturally from one generation to another, creating an opportunity for a different kind of evolutionary channel to exist. It is a central aim of this thesis to explore some of the adaptive dynamics that such a cultural channel has, and investigate the extent to which certain structural and statistical properties of language can be directly explained as adaptations to the transmission process and the learning biases of speakers. In order to address this aim, this thesis takes an experimental approach. Building on a rich set of empirical results from various computational simulations and mathematical models, it presents a novel methodological framework for exploring one type of cultural transmission mechanism, iterated learning, in the laboratory using human participants. In these experiments, we observe the evolution of artificial languages as they are acquired and then transmitted to new learners. Although there is no communication involved in these studies, and participants are unaware that their learning efforts are being propagated to future learners, we find that many functional features of language emerge naturally from the different constraints imposed upon them during transmission. These constraints can take a variety of forms, both internal and external to the learner. Taken collectively, the data presented here suggest several points: (i) that iterated language learning experiments can provide us with new insights about the emergence and evolution of language; (ii) that language-like structure can emerge as a result of cultural transmission alone; and (iii) that whilst structure in these systems has the appearance of design, and is in some sense ‘created’ by intentional beings, its emergence is in fact wholly the result of non-intentional processes. Put simply, cultural evolution plays a vital role in language. This work extends our framework for understanding it, and offers a new method for investigating it.

This thesis addresses a major problem facing any attempt to account for language structure through a cultural mechanism: The processes required by such a mechanism are only possible if we assume the existence of a range of preconditions. These preconditions are not trivial, and themselves require an explanation. In this thesis I address the nature and origin of these preconditions. I approach this topic in three stages. In the first stage, I pull-apart the functioning of one prominent cultural account of language evolution—the Iterated Learning Model —to identify the preconditions it assumes. These preconditions cluster into two main groups. The first concerns the traditional transmission of the communication system. The second relates to the emergence of particular skills of social cognition that make learned symbols and language-like communication a possibility. In the second stage, I turn to comparative evidence, looking for evolutionary analogies that might shed light on the emergence of these preconditions. Two case studies—the Bengalese finch and the domestic dog—are considered in detail, both of which show aspects of one of the preconditions emerging in the context of domestication. In each case I examine what it is about the domestication process that led to this outcome. In the final stage, I consider whether this same context might explain the emergence of these preconditions in humans. The claim that humans are a self-domesticated species has a long history, and is increasingly invoked in contemporary discussions of language evolution. However, it is often unclear exactly what this claim entails. I present a synthesis and critique of a range of empirical and theoretical perspectives on self-domestication. I conclude that human self-domestication is a coherent concept, and that there are several plausible accounts of how it might have occurred. The realisation that humans are a self-domesticated species can, therefore, provide some insight into how a cultural account of language structure might be possible at all.

Languages are culturally transmitted through a repeated cycle of learning and communicative interaction. These two aspects of cultural transmission impose (at least) three interacting pressures that can shape the evolution of linguistic structure: a pressure for learnability, a pressure for expressivity, and a pressure for coordination amongst users in a linguistic community. This thesis considers how these sometimes competing pressures impact linguistic complexity across cultural time. Using artificial language and iterated learning experimental paradigms, I investigate the conditions under which complexity in morphological and syntactic systems emerges, spreads, and reduces. These experiments illustrate the interaction of transmission, learning and use in hitherto understudied domains - morphosyntax and word order. In a first study (Chapter 2), I report the first iterated learning experiments to investigate the evolution of complexity in compositional structure at the word and sentence level. I demonstrate that a complex meaning space paired with pressures for learnability and communication can result in compositional hierarchical constituent structure, including fixed combinatorial rules of word formation and word order. This structure grants a productive and productively interpretable language and only requires learners to acquire a finite lexicon and a finite set of combinatorial rules (i.e., a grammar). In Chapter 3, I address the unique effect of communicative interaction on linguistic complexity, by removing language learning completely. Speakers use their native language to express novel meanings either in isolation or during communicative interaction. I demonstrate that even in this case, communicative interaction leads to more efficient and overall simpler linguistic systems. These first two studies provide support for the claim that morphological and syntactic complexity are shaped by an overarching drive towards simplicity (or learnability) in language learning and communication. Chapter 4 reports a series of experiments assessing the possibility that the simplicity bias found in the first two studies operates at a different strength depending on the linguistic level. Studies in natural language learning and in pidgin/creole genesis suggest that while morphological variation seems to be highly susceptible to regularisation, variation in other syntactic features, like word order, appears more likely to be reproduced. I test this experimentally by comparing regularisation of unconditioned variation across morphology and word order in the context of artificial language learning. I show that language users in fact regularise unconditioned variation in a similar way across linguistic levels, suggesting that the simplicity bias may be driven by a single, non-level-specific mechanism. Taken together, the experimental evidence presented in this thesis supports the hypothesis that the cultural and cognitive pressures acting on language users during learning and communicative interaction - for learnability, expressivity and coordination - are at least partially responsible for the evolution of linguistic complexity. Specifically, they are responsible for the emergence of linguistic complexity which maximises learnability and communicative efficiency, and for the reduction of complexity which does not. More generally, the approach taken in this thesis promotes a view of complexity in linguistic systems as an evolving variable determined by the biases of language learners and users as languages are culturally transmitted.

Non-representational cognitive science is a promising research field that provides an alternative to the view of the brain as a “computer” filled with symbolic representations of the world and cognition as “calculations” performed on those symbols. Autopoiesis is a biological, bottom-up, non-representational theory of cognition, in which representations and meaning are framed as explanatory concepts that are constituted in an observer’s description of a cognitive system, not operational concepts in the system itself. One of the problems of autopoiesis, and all non-representational theories, is that they struggle with scaling up to high-level cognitive behaviour such as language. The Iterated Learning Model is a theory of language evolution that shows that certain features of language are explained not because of something happening in the linguistic agent’s brain, but as the product of the evolution of the linguistic system itself under the pressures of learnability and expressivity. Our goal in this work is to combine an autopoietic approach with the cultural transmission chains that the ILM uses, in order to provide the first step in an autopoietic explanation of the evolution of language. In order to do that, we introduce a simple, joint action physical task in which agents are rewarded for dancing around each other in either of two directions, left or right. The agents are simulated e-pucks, with continuous-time recurrent neural networks as nervous systems. First, we adapt a biologically plausible reinforcement learning algorithm based on spike-timing dependent plasticity tagging and dopamine reward signals. We show that, using this algorithm, our agents can successfully learn the left/right dancing task and examine how learning time influences the agents’ task success rates. Following that, we link individual learning episodes in cultural transmission chains and show that an expert agent’s initial behaviour is successfully transmitted in long chains. We investigate the conditions under which these transmission chains break down, as well as the emergence of behaviour in the absence of expert agents. By using long transmission chains, we look at the boundary conditions for the re-establishment of transmitted behaviour after chain breakdowns. Bringing all the above experiments together, we discuss their significance for non-representational cognitive science and draw some interesting parallels to existing Iterated Learning research; finally, we close by putting forward a number of ideas for additions and future research directions.

Cette thèse explore le développement de modèles d'apprentissage automatique robustes et compositionnels en exploitant les connaissances issues des goulots d'étranglement cognitifs qui structurent l'apprentissage humain et la représentation des connaissances. Notre recherche explore des concepts ancrés dans trois goulots d'étranglement cognitifs : l'apprentissage itéré, l'oubli et le réapprentissage, et l'attention sélective. Tout d'abord, nous examinons l'apprentissage itéré (IL), une théorie qui explique l'émergence de la compositionnalité dans les langues humaines. Traditionnellement étudiée dans des jeux référentiels simples imitant des expériences de sciences cognitives, nous étendons son application à un contexte plus large de question-réponse visuelle (VQA) en utilisant des réseaux de modules neuronaux (NMNs). En traitant la communication entre un générateur de programme et le moteur d'exécution dans un NMN comme une langue émergente, nous encourageons la compositionnalité dans sa structure, ce qui conduit à une généralisation systématique améliorée. Nos résultats démontrent que l'IL amplifie la compositionnalité là où une préférence pour celle-ci existe, améliorant ainsi la performance des tâches. Ensuite, nous investiguons le goulot d'étranglement cognitif de l'oubli et du réapprentissage, qui permet aux humains d'intégrer de nouvelles connaissances plus efficacement avec les connaissances existantes. L'oubli est souvent émulé dans l'apprentissage automatique par des réinitialisations partielles de paramètres. Cependant, nous proposons que la minimisation consciente de la netteté (SAM) incarne le paradigme de l'oubli et du réapprentissage, en effectuant un oubli ciblé des biais indésirables uniquement lors de la perturbation pour calculer les gradients de mise à jour. Notre perspective de l'oubli perturbé résout les contradictions dans le récit de la minimisation de la netteté et propose également une perturbation améliorée qui surpasse le SAM standard. Nous concluons que l'oubli ciblé sans compromettre l'état appris du modèle peut significativement améliorer la généralisation et la robustesse. Enfin, nous explorons l'attention sélective, un mécanisme cognitif qui permet aux humains de se concentrer sur les aspects saillants de leur environnement. La taille limitée de notre mémoire de travail impose la simplicité des concepts auxquels nous prêtons attention, et des représentations plus larges nécessitent de prêter attention séparément à différents sous-ensembles de nos stimuli. Cette capacité à construire des représentations complexes à partir de plus simples permet en partie la compréhension compositionnelle et la généralisation humaine. Nous appliquons ce concept pour proposer SPARO, un nouveau module de lecture qui structure les encodages dans CLIP et DINO en collections de concepts auxquels on prête attention séparément. Chaque concept SPARO est produit par une seule tête d'attention avec une dimensionnalité limitée, imitant le goulot d'étranglement de l'attention humaine. Nous démontrons que la structuration des encodages avec un a priori pour l'attention sélective améliore la compositionnalité, la généralisation en aval et la capacité d'intervention manuelle pour filtrer les concepts pertinents. / This thesis explores the development of robust and compositional machine learning models by harnessing insights from cognitive bottlenecks that structure human learning and knowledge representation. Our research explores concepts rooted in three cognitive bottlenecks: iterated learning, forgetting and relearning, and selective attention. First, we examine iterated learning (IL), a theory that explains the emergence of compositionality in human languages. Traditionally studied in simple referential games mimicking cognitive science experiments, we extend its application to a broader context of visual question-answering (VQA) using neural module networks (NMNs). By treating the communication between a program generator and the execution engine in an NMN as an emergent language, we encourage compositionality in its structure, leading to improved systematic generalization. Our findings demonstrate that IL amplifies compositionality where a preference for it exists, thereby enhancing task performance. Next, we investigate the cognitive bottleneck of forgetting and relearning, which enables humans to integrate new knowledge more effectively with existing knowledge. Forgetting is often emulated in machine learning through partial parameter resets. However, we propose that sharpness-aware minimization (SAM) embodies the forget-and-relearn paradigm, performing targeted forgetting of undesirable biases, only during perturbation to compute update gradients. Our perturbed forgetting perspective addresses contradictions in the sharpness minimization narrative and also offers an improved perturbation that outperforms standard SAM. We conclude that targeted forgetting without compromising the model's learned state can significantly enhance generalization and robustness. Finally, we explore selective attention, a cognitive mechanism that enables humans to focus on salient aspects of their environment. The limited size of our working memory forces the simplicity of the attended concepts, and broader representations require separately attending to different subsets of our stimuli. This ability to construct complex representations from simpler ones in part enables human compositional understanding and generalization. We apply this concept to propose SPARO, a new read-out module that structures encodings in CLIP and DINO as collections of separately attended concepts. Each SPARO concept is produced through a single attention head with limited dimensionality, emulating the human attention bottleneck. We demonstrate that structuring encodings with a prior for selective attention enhances compositionality, downstream generalization, and the capacity for manual intervention to filter relevant concepts.

Dans le traitement de langue et des système de dialogue, il est courant de pré-entraîner des modèles de langue sur corpus humain avant de les affiner par le biais d'un simulateur et de résolution de tâches. Malheuresement, ce type d'entrainement tend aussi à induire un phénomène connu sous le nom de dérive du langage. Concrétement, les propriétés syntaxiques et sémantiques de la langue intiallement apprise se détériorent: les agents se concentrent uniquement sur la résolution de la tâche, et non plus sur la préservation de la langue. En s'inspirant des travaux en sciences cognitives, et notamment l'apprentigssage itératif Kirby and Griffiths (2014), nous proposons ici une approche générique pour contrer cette dérive du langage. Nous avons appelé cette méthode Seeded iterated learning (SIL), ou apprentissage itératif capitalisé. Ce travail a été publié sous le titre (Lu et al., 2020b) et est présenté au chapitre 2. Afin d'émuler la transmission de la langue entre chaque génération d'agents, un agent étudiant est d'abord pré-entrainé avant d'être affiné de manière itérative, et ceci, en imitant des données échantillonnées à partir d'un agent enseignant nouvellement formé. À chaque génération, l'enseignant est créé en copiant l'agent étudiant, avant d'être de nouveau affiné en maximisant le taux de réussite de la tâche sous-jacente. Dans un second temps, nous présentons Supervised Seeded iterated learning (SSIL) dans le chapitre 3, où apprentissage itératif capitalisé avec supervision, qui a été publié sous le titre (Lu et al., 2020b). SSIL s'appuie sur SIL en le combinant avec une autre méthode populaire appelée Supervised SelfPlay (S2P) (Gupta et al., 2019), où apprentissage supervisé par auto-jeu. SSIL est capable d'atténuer les problèmes de S2P et de SIL, i.e. la dérive du langage dans les dernier stades de l'entrainement tout en préservant une plus grande diversité linguistique. Tout d'abord, nous évaluons nos méthodes dans sous la forme d'une preuve de concept à traver le Jeu de Lewis avec du langage synthetique. Dans un second temps, nous l'étendons à un jeu de traduction se utilisant du langage naturel. Dans les deux cas, nous soulignons l'efficacité de nos méthodes par rapport aux autres méthodes de la litterature. Dans le chapitre 1, nous discutons des concepts de base nécessaires à la compréhension des articles présentés dans les chapitres 2 et 3. Nous décrivons le problème spécifique du dialogue orienté tâche, y compris les approches actuelles et les défis auxquels ils sont confrontés : en particulier, la dérive linguistique. Nous donnons également un aperçu du cadre d'apprentissage itéré. Certaines sections du chapitre 1 sont empruntées aux articles pour des raisons de cohérence et de facilité de compréhension. Le chapitre 2 comprend les travaux publiés sous le nom de (Lu et al., 2020b) et le chapitre 3 comprend les travaux publiés sous le nom de (Lu et al., 2020a), avant de conclure au chapitre 4. / In task-oriented dialogue, pretraining on human corpus followed by finetuning in a simulator using selfplay suffers from a phenomenon called language drift. The syntactic and semantic properties of the learned language deteriorates as the agents only focuses on solving the task. Inspired by the iterative learning framework in cognitive science Kirby and Griffiths (2014), we propose a generic approach to counter language drift called Seeded iterated learning (SIL). This work was published as (Lu et al., 2020b) and is presented in Chapter 2. In an attempt to emulate transmission of language between generations, a pretrained student agent is iteratively refined by imitating data sampled from a newly trained teacher agent. At each generation, the teacher is created by copying the student agent, before being finetuned to maximize task completion.We further introduce Supervised Seeded iterated learning (SSIL) in Chapter 3, work which was published as (Lu et al., 2020a). SSIL builds upon SIL by combining it with the other popular method called Supervised SelfPlay (S2P) (Gupta et al., 2019). SSIL is able to mitigate the problems of both S2P and SIL namely late-stage training collapse and low language diversity. We evaluate our methods in a toy setting of Lewis Game, and then scale it up to the translation game with natural language. In both settings, we highlight the efficacy of our methods compared to the baselines. In Chapter 1, we talk about the core concepts required for understanding the papers presented in Chapters 2 and 3. We describe the specific problem of task-oriented dialogue including current approaches and the challenges they face: particularly, the challenge of language drift. We also give an overview of the iterated learning framework. Some sections in Chapter 1 are borrowed from the papers for coherence and ease of understanding. Chapter 2 comprises of the work published as (Lu et al., 2020b) and Chapter 3 comprises of the work published as (Lu et al., 2020a). Chapter 4 gives a conclusion on the work.