This research outlines a conceptual framework and data analysis process to examine multisource feedback (MSF) rater group differences from a leadership assessment survey, after testing the measures for equivalence. MSF gathers and compares ratings from supervisors, peer, followers and self and is the predominant leadership assessment tool in the United States. The results of MSF determine significant professional outcomes such as leadership development opportunities, promotions and compensation. An underlying belief behind the extensive use of MSF is that each rater group has a different set of implicit leadership theories (ILTs) they use when assessing the leader, and therefore each group is able to contribute unique insight. If this is true, research findings would find rater group consistency in leadership assessment outcomes, but they do not. A review of group comparison research reveals that most empirical MSF studies fail to perform preliminary data exploration, employ consistent models or adequately test for measurement equivalence (ME); yet industry standards strongly suggest exploratory methods whenever data sets undergo changes, and misspecified models cause biased results. Finally, ME testing is critical to ascertain if rater groups have similar conceptualizations of the factors and items in an MSF survey. If conceptual ME is not established, substantive group comparisons cannot be made. This study draws on the extant MSF, ILT and ME literature and analyzes rater group data from a large, application-based MSF leadership database. After exploring the data and running the requisite MI tests, I found that the measures upheld measurement invariance and were suitable for group comparison. Additional MI tests for substantive hypotheses support found that significant mean differences did exist among certain rater groups and dimensions, but only direct report and peer groups were consistently significantly different in all four dimensions (analytical, interpersonal, courageous and leadership effectiveness). Additionally, the interpersonal dimension was the most highly correlated with leadership effectiveness in all five rater groups. The overall findings of this study address the importance of MSF data exploration, offer alternative explanations to the disparate leadership MSF research findings to date and question the application use of MSF tools in their current form.