Training and empirically evaluating caregivers’ implementation of behavior management skills is a particularly challenging task in large residential contexts. A pyramidal training approach provides an efficient and effective way to conduct large-scale competency-based behavior skills training. The purpose of this project was to develop and evaluate a large-scale pyramidal staff training program for behavior management skills. One hundred nine caregivers and 11 behavior service professionals at a large, residential care facility participated in this project. Interobserver agreement was utilized to develop and refine measurements systems to detect caregiver acquisition of skills, behavior service professionals’ ability to score caregiver performance and behavior service professionals’ ability to deliver a specified portion of the curriculum. Pre- and post-test probes were conducted utilizing standard role play scenarios and checklists to evaluate caregiver acquisition of three specific behavior management skills. The results supported the following conclusions: first, interobserver agreement measures were useful to develop a reliable measurement system, to refine some curriculum elements, and to evaluate measurement conducted by behavior service professionals. Second, behavior skills training (BST) resulted in caregiver acquisition of all three behavior management techniques. Third, the pyramidal training approach was effective to teach behavior service professionals to deliver BST and accurately measure the performances of trainees.