While enrolling in a college for a degree is still considered to be more of the expectation than the rule in households across the United States it is facing public criticism regarding its overall cost versus long-term benefits. This has prompted researchers to reevaluate practices in higher education to determine ways in which post-secondary institutions can improve their overall outcomes. In part, research indicates that this should include revisions to assessment, evaluation, and research (AER) practices. Currently, higher education faces many challenges associated with rigorous AER practices, which include an institutional focus on accountability measures, organizational challenges, and difficulties in operationally defining and measuring constantly changing definition for students, faculty, and higher education. These challenges have forced higher education to move away from rigorous quantitative designs that show causality. Additionally, higher education’s focus on large N research struggles to capture the unique identities and experiences of today’s college campus, which also poses challenges for the use of AER by practitioners across academic and student affairs. At present, new approaches to AER must be considered. The purpose of the current study was to propose and investigate one potential way to supplement large N AER with the use of single-subject research design (SSRD). Using visual analysis and the calculation of effect sizes with nonoverlap of all pairs (NAP), archival data from a unique behavioral program in a post-secondary transition program located at a land-grant institution in the southeastern United States was evaluated to determine if its implementation reduced student referrals at the program (N =25) and cohort level between Fall 2019 to Spring 2020. Results indicated that the behavioral program was most effective at the program level but had variable results at the cohort level. These results indicate that SSRD can be an effective approach to AER practices in higher education and would adequately supplement and potentially inform further large N research with its ability address to assess smaller populations.

This study contributes to the existing literature on treatment integrity (TI) by presenting TI findings across interventions aimed at the development of social emotional skills in learners with emotional and behavioural disorders. Social and emotional skills were selected as the target for our investigation given the significance of these skills in relation to the academic and behavioural success of learners and the challenges most often faced in these skill areas by students with emotional and behaviour disorders (E/BD). The study analysed single-subject experimental studies from 2000 to 2009 in two leading journals in the field of emotional and behaviour disorders: Behavior Disorders and The Journal of Emotional Behavioral Disorders. The degree to which studies operationally defined independent variables and evaluated and reported measures of treatment integrity and associated risk factors is reported. Thirty-three studies met the inclusion criteria for the present study and TI was evaluated across six variables (1) year published, (2) dependent variable(s), (3) independent variables(s), (4) participant characteristics, (5) treatment agent and (6) assessment of TI. Results indicated that approximately 49% of the studies monitored and reported TI, meaning that they provided a description of the TI procedure and resultant data. Findings from the study point to the need for attention to TI both in the description of methods used and in the reporting of TI data.

In single-case research design (SCRD), experimental control is demonstrated when the researcher’s application of an intervention, known as the independent variable, reliably produces a change in behavior, known as the dependent variable, and the change is not otherwise explained by confounding or extraneous variables. SCRD studies that fail to demonstrate experimental control may not be published because researchers may be unwilling to submit these papers for publication due to null findings and journals may be unwilling and unlikely to publish null outcomes (i.e., publication bias). The lack of submission and publication of null findings, leading to a disproportion of positive studies in the published research literature, is known as the “file drawer effect” (Rosenthal, 1979; Ferguson & Heene, 2012). Recently, researchers and policy organizations have identified evidence-based practices (EBPs) for children with autism spectrum disorder (ASD) based on systematic reviews of SCRD studies (Odom, Collet-Klingenberg, Rogers, & Hatton, 2010). However, if SCRD studies that do not demonstrate experimental control (i.e., null studies) are disproportionately unpublished due to the file drawer effect, this may result in a misrepresentation of positive findings, leading interventions to be deemed evidence-based that, actually, lack sufficient empirical support (Sham & Smith, 2014; Shadish, Zelinsky, Vevea, & Kratochwill, 2016). Social narratives, exercise, self-management, and response interruption/redirection are interventions for children with ASD that has been named EBPs according to the National Autism Standards (NAC; 2009) and National Professional Development Center on Autism Spectrum Disorder (NPDC; 2010); however, these interventions have not yet been evaluated for potential publication bias. The study employed and extended methods similar to Sham and Smith (2014), comparing the procedures and results of published articles and unpublished dissertations and theses for interventions identified as EBPs to evaluate the methodological rigor and evaluate the possibility of publication bias, file drawer effect, and lack of replication. Specifically, the results of published and unpublished studies were compared to determine if published studies showed greater treatment effect, which would indicate the file drawer effect. Also, SCRD quality indicators were employed to evaluate whether studies that were published tend to be of higher quality, as this would mitigate possible publication bias shown by larger effect sizes (ES) in published studies. The outcome resulted in three out of four EBPs (social narratives, antecedent exercise, and response interruption and redirection), yielding different ES when published studies were compared to unpublished studies; in contrast, self-management yielded a similar ES for published and unpublished studies. For social narratives and antecedent exercise, unpublished studies presented at lower estimated ES than published studies; whereas for response interruption and redirection, unpublished studies presented at a higher estimated ES compared to published studies. Generally, study quality presented at similar levels for published and unpublished studies for each EBP, with the exception of antecedent exercise. Differences were identified for antecedent exercise study quality based upon visual and statistical analyses. Lastly, there do not appear to be observed differences in treatment outcomes between published and unpublished studies when study quality was considered in the analysis. Implications of the results are discussed with respect to the file drawer effect and publication bias in EBPs, and the call to increase publications in peer-reviewed journals of negative findings and replication studies, which leads to identifying and establishing boundary criteria for EBPs. / Special Education