The processes and mechanisms that govern brain size evolution remain a widely discussed topic in evolutionary biology. How relative brain size relates to animal behaviour and cognition is even more controversial. Recent comparative and experimental studies have shown a positive relationship between relative brain size and complexity of behaviour. Some of the most important behaviours that have direct consequences for an individual’s fitness are reproductive behaviours, and they sometimes require quite complex behavioural repertoires. Selection for complex behaviour might therefore induce an expansion of brain size to allow for cognitively demanding tasks during courtship and mating. In the present study we investigated the effect of relative brain size on reproductive behaviour in male guppies (Poecilia reticulata), using fish from a recently established brain size artificial selection experiment. Females were paired with either a large- or a small-brained male, and we collected data on a suite of male courtship behaviours including sneak copulation attempts, courtship display, gonopodial swings and time spent following the female. Although the extent of orange colouration, a trait that varies across large- and small-brained males, affected male behaviour, we were not able to detect any difference in reproductive behaviour between the brain size selection lines. These results suggest that there is no strong association between male mating behaviour and relative brain size, and future studies will examine this question further. But currently, our results indicate that relative brain size might not be linked to reproductive behaviour to any significant extent, at least not in the guppy.