Psychosocial problems are exceedingly common following moderate-to-severe traumatic brain injury (TBI), and are thought to be the major predictor of long-term functional outcome. However, current rehabilitation protocols have shown little success in improving interpersonal and social abilities of individuals with TBI, revealing a critical need for new and more effective treatments. Recent research has shown that neuro-modulatory treatments (e.g., non-invasive brain stimulation, lifestyle interventions) targeting the functionality of specific brain systems—as opposed to focusing on re-teaching individuals with TBI the impaired behaviors— hold the potential to succeed where past behavioral protocols have failed. However, in order to implement such treatments it is crucial to gain a better knowledge of the neural systems underlying social functioning secondary to TBI. It is well established that in TBI populations the inability to identify and interpret social cues, and in particular to engage in successful recognition of facial affects, is one of the factors driving impaired social functioning following TBI. The aims of the work here described were threefold: (1) to determine the degree of impairment in individuals with moderate-to-severe TBI on tasks measuring different sub-types of facial affect recognition skills, (2) to determine the relationship between white matter integrity and different facial affect recognition ability in individuals with TBI by using diffusion tensor imaging, and (3) to determine the patterns of brain activation associated with facial affect recognition ability in individuals with TBI by using task-related functional magnetic resonance imaging (MRI). Our results revealed that individuals with TBI are impaired at both perceptual and verbal categorization facial affect recognition tasks, although they are significantly more impaired in the latter. Moreover, performance on tasks tapping into different types of emotion recognition abilities showed different white-matter neural correlates, with more individuals with TBI showing more extensive damage in the left inferior fronto-occipital fasciculus, uncinate fasciculus and inferior longitudinal fasciculus more likely to perform poorly on verbal categorization tasks. Lastly, our functional MRI study suggests an involvement of left dorsolateral prefrontal regions in the disruption of more perceptual emotion recognition skills, and involvement on the fusiform gyrus and of the ventromedial prefrontal cortex in more interpretative facial affect recognition deficits. The findings here presented further out understanding of the neurobiological mechanisms underlying facial affect impairment following TBI, and have the potential to inform the development of new and more effective treatments.