Brain size varies greatly among vertebrates. It has been proposed that the diversity of brain size is produced and maintained through a balance of adaptations to different types and levels of cognitive ability and constraints for adaptive evolution. Phylogenetic comparative studies have made major contributions to our understanding of brain size evolution. However, previous studies have nearly exclusively focused on mammalian and avian taxa and almost no attempts have been made to investigate brain size evolution in ectothermic vertebrates. In my thesis, I studied brain size evolution in two groups of fish with extreme diversity in ecology, morphology and life history, Cichlidae and Syngnathidae. Using phylogenetic comparative methods, I investigated four key questions in vertebrate brain size evolution; cognitive adaptation, sexual selection, phenotypic integration and energetic constraints. I have demonstrated i) that phenotypic integration can link functionally unrelated traits, and this may constrain independent evolution of each part involved or promote concerted evolution of an integrated whole, ii) that brain-body static allometry constrains the direction of brain size evolution, even though the static-allometry showed ability to evolve, allowing evolution of relative brain size under allometric constraints, iii) that the energetic constraints of development and maintenance of brain tissue is an important factor in forming the diversity in brain size in cichlids and syngnathids, both at macroevolutionary and microevolutionary time scales, and iv) that adaptation for feeding and female mating competition may have played key roles in the adaptive evolution of brain size in pipefishes and seahorses. To conclude, my thesis shows the strong benefit of using fish as a model system to study brain size evolution with a phylogenetic comparative framework.

Competition over access for mates has led to the evolution of many striking examples of morphological traits and behaviour in animals. The rapid development of the sexual selection field in recent decades have dramatically advanced our understanding of what traits make individuals more successful in attracting mates and how preferences for mates evolve over time. However, till now, research in this field has put less emphasis on the mechanisms that underlie variation in mate choice and sexual behaviour. Cognitive processes could potentially be key drivers of individual variation in mating preferences and sexual behaviours and therefore deserve further investigation. In this thesis, I used guppies artificially selected for relative brain size as the model system to study the association between brain size, cognitive ability and various aspects of mate choice. Previous studies in this model system showed that large-brained individuals of both sexes outperformed small-brained individuals in cognitive tests. Here I quantified their sexual behaviours and mating preferences to provide novel empirical data concerning the association between brain size, cognitive ability and sexual selection. In dichotomous choice preference tests based on visual cues, comparisons between large-brained and small-brained guppies showed important differences in their assessment of mate quality. These results are not driven by pre-existing visual biases caused by the artificial selection since further investigation of the visual capacity of these fish detected no differences between large-brained and small-brained individuals in their sensitivity to colour or in their capacity to resolve spatial detail. I also quantified sexual behaviour in male guppies artificially selected for relative brain size and found no difference in the behaviours of large-brained and small-brained males in a single male-single female non-competitive scenario. On the contrary, in a more complex social setting I found a reduction in large-brained males in the rate of courtship towards females and dominance displays towards other males when exposed to different degrees of predation threat and different numbers of male competitors. However, this reduction in behavioural intensity did not result in a lower access to copulation with females for large-brained males. I likewise evaluated female sexual behaviour and found that large-brained females had higher behavioural flexibility such that they decreased their receptiveness towards males more strongly under higher levels of predation threat. Together, these results provide novel empirical evidence that brain size and cognitive ability are tightly linked to mating preferences and sexual behaviours. These findings suggest that brain size and cognitive ability might be important mechanisms behind variation in mating preferences and in sexually selected traits across and within species. / <p>At the time of the doctoral defense, the following papers were unpublished and had a status as follows: Paper 2: Manuscript. Paper 3: Manuscript. Paper 5: Manuscript.</p>