The extent to which interventions are implemented as intended is called treatment integrity (TI). Given that it is unreasonable to expect 100% TI in applied settings, a through understand of TI is essential to appropriately train teachers. This understanding must include the types of TI errors that may be committed and how these effect student learning. It is essential to study TI errors because of the real-world implications for students, including eligibility decisions for special education services, which are based upon students’ responses to inventions. It is not possible to make educational decisions on intervention effectiveness unless it is clear that evidence-based practices have been implemented accurately. If TI is low, it is impossible to determine which services and interventions a student requires. Further, measuring the fidelity with which interventions are applied allows for a more thorough and accurate understanding of which components of an intervention are effective, necessary, and feasible. A broader understanding of which TI errors are most significant, as well as measuring the necessary levels of TI, will lead to more accurate information about how to implement evidence-based practices. The purpose of this study is to gain a more nuanced understanding of TI failures in the form of errors of commission and the role commission errors have on participant responsiveness (Power, 2005). This study extends the results of DiGennaro Reed et al. (2011) by including an intermediate (80%) level of TI which may more accurately represent an attainable level of TI in applied settings. For two of four students, more errors of commission were related to lower skill acquisition. For the other two students, idiosyncratic patterns of responding emerged.

The purpose of this study was to examine the similarity of outcomes from three different treatment integrity (TI) methods, and to identify the method which best corresponded to the assessment of a child’s behavior. Six raters were recruited through individual contact via snowball sampling. A modified intervention component list and 19 video clips were derived from Stahr’s (2005) study, “An Intervention for Children with Autism Spectrum Disorders (ASD) who Have Food Selectivity.” The raters, randomly and evenly assigned to three dyads. Each dyad received an average of six hours training and reached 85% interobserver agreement (IOA) with a 0.60 kappa score. After training, each dyad watched 5 video clips per day and measured both the child’s behavior and TI. The percentages of IOA, kappas, and indices of dependability for assessment of the child’s behavior and TI were analyzed. The data revealed that all raters reached over 80% IOA and the whole interval (WI) and yes/no (Y/N) dyads reached .60 kappa, but the two raters in the Likert-type (LIK) dyad could not reach .60 kappa. The indices of dependability indicated that the six raters consistently observed and rated both the child’s behavior and TI, but there was a discrepancy in scores (i.e., percentages of TI) between the two raters in the two indirect measure dyads (i.e., Y/N and LIK). An analysis of the percentages of total variance showed that the two indirect TI methods may affect the discrepancy between the two raters’ rating scores. A comparison of the three different TI methods and correlation between the child’s behavior and TI were examined using the PASW Statistics 18 software program. There was no significant difference between the WI and the Y/N dyads, while the assessments from the LIK dyad indicated a significant difference from the other two dyads. Both the WI and the Y/N dyads showed correlations between the degree of the child’s behavior and the degree of TI, but there was no significant difference between the two correlation coefficients. Questions about reliability with the indirect TI measures suggest one should be careful in considering these results.

Evidence-based policing (EBP) has emerged as a key strand of police innovation since Sherman’s (1998) Police Foundation lecture. However, for others EBP raises as many questions as answers. One of the most contentious areas is the role advocated for randomised controlled trials in testing practice and developing knowledge to support EBP. RCTs are controversial with some scholars who argue that policing is not comparable to medicine and that RCTs are unable to reflect the complexity of the police role and context. Even those who advocate the use of RCTs recognise that there are significant challenges in achieving the high dosage and high fidelity that a successful experiment requires. This dissertation responds to these challenges by analysing the completed randomised controlled trials in policing and using a case study, Operation Turning Point, to identify the factors that may contribute to the conduct and management of police field trials with high levels of treatment integrity. In the introduction, Chapter 1, the approach is set out, framed around grounded theory, to be developed in four, linked, chapters. Chapter 2 is focused on understanding treatment integrity in RCTs involving the police: A search for police RCTs is produced 122 Police RCTs completed and reported by 2016. The levels of treatment integrity are analysed. 78 of the 122 RCTs exceeded a 60% threshold, with 49 being above 90%. In Chapter 3, a “novice theory” is developed and tested as an explanation for levels of treatment integrity in police randomised controlled trials: Analysis of the 122 RCTs suggests that “novice theory” can provide an explanation for the general patterns of treatment integrity. Further detailed analysis suggested that there are, however, other factors which may be important in determining the treatment integrity. These are developed in Chapter 4, which centres on a case study of Operation Turning Point. Using published case studies and an analysis of juvenile justice RCTs, a potential framework of operational factors is developed that appear to be important in effective conduct and management. The Turning Point case study is used to develop and expand on those operational factors. Finally, taking the two together, the analysis concluded that, beyond the operational factors, there were some more strategic, “protective factors” that were also critical. These are developed in Chapter 5, by using the coding and analysis of interviews with a sample of key staff involved in Turning Point Our analysis suggests that novice theory needs to be understood in the context of both the operational and protective factors that we have identified. Taken together these findings indicate the potential advantages of building institutional frameworks in which the development of practitioners and researchers and the conduct and management of experimental research could be brought closer together. We conclude with ten recommendations designed to improve the treatment integrity of police RCTs.

Treatment integrity—the extent to which a treatment is delivered as it was intended—has long been recognized as critically important in treatment evaluation research, but has garnered increased attention in recent years within the context of dissemination and implementation science. However, the field’s development has been hindered by inadequate measurement tools. This project is focused on developing and evaluating the psychometric strength of two measures of treatment integrity. To evaluate the psychometric strength of the Cognitive-Behavioral Therapy for Youth Anxiety Therapist Adherence Scale (CBAY-A)and the Cognitive-Behavioral Therapy for Youth Anxiety Therapist Competence Scale (CBAY-C), 954 psychotherapy sessions from two treatment evaluation studies were coded. Analysis of the evidence for reliability and validity of the item scores for each measure provide substantial support for each measure, while also highlighting areas in need of further evaluation. The discussion focuses on interpreting the psychometric strength of the CBAY-A and CBAY-C compared to other measures of treatment integrity, next steps for evaluating the psychometric strength of the two measures, and potential applications of the CBAY-A and CBAY-C.

Treatment integrity has a direct impact on early intensive behavioral intervention (EIBI) outcomes for children with autism. Research suggests that providing feedback can improve treatment integrity in EIBI. The purpose of this study was to compare the effects of two remote feedback methods, videoconference feedback and email feedback, on treatment integrity. Participants included six teachers who were providing EIBI to children with autism in China. Using a single-case alternating treatment design, each feedback method was associated with a specific teaching procedure, either discrete trial training or incidental teaching. The teachers implemented the two teaching procedures daily but only received feedback during intervention. Results showed that videoconference feedback produced faster mastery of the teaching procedures and better treatment integrity sustainability after the intervention was removed than email feedback. Treatment integrity deterioration was observed during maintenance and follow up. Results of generalization were mixed. In social validity evaluations, teachers expressed preference of videoconference feedback over email feedback in terms of acceptance and effectiveness of the intervention, but they considered email feedback a more efficient use of their time.

A teacher in a mainstream classroom identified six children with behaviour difficulties or learning difficulties who were not receiving any assistance outside of the classroom. The teacher was to independently complete a functional assessment of each child to reach a hypothesis about the cause of the difficulty, and then choose and implement an appropriate intervention. Treatment integrity was monitored throughout the interventions to ensure correct implementation. It became apparent that the teacher was unable to complete adequate functional assessments or maintain appropriate interventions because of the time constraints associated with being in a classroom and her lack of specialist knowledge.

Treatment integrity is an important component of behavioural interventions, however few studies have examined methods to teach supervisors to evaluate the treatment integrity of such interventions applied by front-line staff. The purpose of the current study was to evaluate the effectiveness of a self-instructional package to teach individuals to evaluate the treatment integrity of discrete-trials teaching (DTT) sessions using the Discrete-Trials Teaching Evaluation Form (DTTEF). Participants consisted of six staff from the St.Amant Autism Programs. In a modified multiple-baseline design across a pair of participants, and replicated across two more pairs, at Baseline, a participant observed a confederate who role-played an instructor teaching three tasks to a confederate who role-played a child with autism spectrum disorder (ASD). Each participant was required to evaluate sessions taught with (a) low integrity, (b) moderate integrity, or (c) high integrity. During training, participants studied a self-instructional package. At Post-training, participants were assessed on the same tasks as during Baseline. During Generalization, participants were assessed evaluating the treatment integrity of three videos of an Autism Tutor administering DTT to a child with ASD. Finally, during a seven-month Follow-up, four participants were available and were assessed evaluating the treatment integrity of a confederate instructor teach a confederate child with low, moderate, and high integrity. Results demonstrated that after an average of 1 hour and 16 minutes of training, there was an immediate increase in accuracy across all participants. Specifically, mean accuracy increased from 47.6% in Baseline to 84.7% at the Post-training assessment (a 37.1% increase). All participants showed excellent generalization results, and three of the four participants who were available at the Follow-up assessment performed at a high level. These results suggest that the training package has potential to be used as an effective method to train staff who work with children with ASD to evaluate the treatment integrity of DTT sessions. / February 2016

Treatment integrity has always had a presence in research, but now more than ever must become a priority owing to the changes in Special Education Law. The present study intends to explore the relationship between factors of implementation, treatment integrity of intervention implementation, and reading fluency. Participants included students in grades 2 through 5 and their teachers enrolled in an urban elementary school in the southwest area of the United States. Participants were chosen for possible inclusion on the basis of their fall performance relative to oral reading fluency on a universal screening measure used as part of the district's Response to Intervention (RTI) plan. Classroom teachers were observed implementing reading interventions and asked to respond to surveys aimed at summarizing their opinions regarding factors related to choice of intervention and implementation. Path analysis was conducted to explore the relationship between two factors of implementation (intervention complexity and acceptability), treatment integrity (adherence to intervention protocol) and student outcomes (oral reading fluency scores). It was hypothesized that low scores for intervention complexity would be inversely related to levels of treatment integrity, which would subsequently be positively related to reading fluency. Moreover, it was hypothesized that intervention acceptability and treatment integrity would be positively related, which would subsequently be positively related to reading fluency. Lastly, it was hypothesized that there would be an inverse relationship between intervention complexity and reading fluency, and a positive relationship between intervention acceptability and reading fluency. Results indicated an inverse relationship between intervention complexity and treatment integrity, suggesting that when complexity was low, treatment integrity was high. A positive relationship was found between intervention acceptability and treatment integrity, suggesting that when acceptability was high, treatment integrity was high. Furthermore, when treatment integrity was high, reading fluency scores were found to be high. An inverse relationship was found between complexity and reading fluency, suggesting that when complexity was low, reading fluency scores were high. Lastly, a positive relationship was found between acceptability and reading fluency, suggesting that when acceptability was high, reading fluency scores were high.

No description available.

Discrete Trial Teaching (DTT) is a teaching method that involves fast-paced trials designed to teach basic skills by breaking them into smaller components, typically conducted in a one-on-one setting. Treatment integrity has proven to be of great importance in DTT, with skill acquisition occurring at higher rates when treatment integrity is high. While research has shown that verbal and written feedback are effective in training staff to conduct DTT, there is still a need for research on the use of individualized interventions based on performance assessments. This study used a multiple-probe across participants design, and demonstrated that a one-on-one session including interventions such as feedback, practice, treatment integrity checklists, and/or antecedent interventions is an effective method for increasing treatment integrity and implementation of DTT. The three participants that took part in the individualized interventions all displayed increases in proficiency of delivering DTT trials. / Applied Behavioral Analysis