Rater training (RT) improves the reliability of assessment tools, but has not been well studied for technical skills. This study assessed whether RT improved the psychometric properties of surgical skill assessments. Surgeons (N=47) were randomized to RT or non-training groups. The RT group underwent frame-of-reference training. Participants assessed trainees performing a suturing and knot-tying task using four assessment tools. Inter-rater reliability, initial and delayed rater agreement, and construct validity were assessed between groups. There was no significant effect of RT on the assessment tools’ reliability and validity. Reliability and validity were most robust for the global rating scale. Although there were trends towards improved reliability and validity with RT, confidence intervals were wide and overlapping. Reliability remained below the minimum desired level of 0.8 required for high-stakes testing. Although RT may represent a way to improve reliability, further study is needed to determine effective training methods. / February 2017