Brain size varies greatly among vertebrates. It has been proposed that the diversity of brain size is produced and maintained through a balance of adaptations to different types and levels of cognitive ability and constraints for adaptive evolution. Phylogenetic comparative studies have made major contributions to our understanding of brain size evolution. However, previous studies have nearly exclusively focused on mammalian and avian taxa and almost no attempts have been made to investigate brain size evolution in ectothermic vertebrates. In my thesis, I studied brain size evolution in two groups of fish with extreme diversity in ecology, morphology and life history, Cichlidae and Syngnathidae. Using phylogenetic comparative methods, I investigated four key questions in vertebrate brain size evolution; cognitive adaptation, sexual selection, phenotypic integration and energetic constraints. I have demonstrated i) that phenotypic integration can link functionally unrelated traits, and this may constrain independent evolution of each part involved or promote concerted evolution of an integrated whole, ii) that brain-body static allometry constrains the direction of brain size evolution, even though the static-allometry showed ability to evolve, allowing evolution of relative brain size under allometric constraints, iii) that the energetic constraints of development and maintenance of brain tissue is an important factor in forming the diversity in brain size in cichlids and syngnathids, both at macroevolutionary and microevolutionary time scales, and iv) that adaptation for feeding and female mating competition may have played key roles in the adaptive evolution of brain size in pipefishes and seahorses. To conclude, my thesis shows the strong benefit of using fish as a model system to study brain size evolution with a phylogenetic comparative framework.