When an exam is administered across dual platforms, such as paper-and-pencil and computer-based testing simultaneously, individual items may become more or less difficult in the computer version (CBT) as compared to the paper-and-pencil (P&P) version, possibly resulting in a shift in the overall difficulty of the test (Mazzeo & Harvey, 1988). Using 38,955 examinees' response data across five forms of the National Nurse Aide Assessment Program (NNAAP) administered in both the CBT and P&P mode, three methods of differential item functioning (DIF) detection were used to detect item DIF across platforms. The three methods were Mantel-Haenszel (MH), Logistic Regression (LR), and the 1-Parameter Logistic Model (1-PL). These methods were compared to determine if they detect DIF equally in all items on the NNAAP forms. Data were reported by agreement of methods, that is, an item flagged by multiple DIF methods. A kappa statistic was calculated to provide an index of agreement bet ween paired methods of the LR, MH, and the 1-PL based on the inferential tests. Finally, in order to determine what, if any, impact these DIF items may have on the test as a whole, the test characteristic curves for each test form and examinee group were displayed. Results indicated that items behaved differently and the examinee's odds of answering an item correctly were influenced by the test mode administration for several items ranging from 23% of the items on Forms W and Z (MH) to 38% of the items on Form X (1-PL) with an average of 29%. The test characteristic curves for each test form were examined by examinee group and it was concluded that the impact of the DIF items on the test was not consequential. Each of the three methods detected items exhibiting DIF in each test form (ranging from 14 items to 23 items). The Kappa statistic demonstrated a strong degree of agreement between paired methods of analysis for each test form and each DIF method pairing (reporting good to excell ent agreement in all pairings). Findings indicated that while items did exhibit DIF, there was no substantial impact at the test level.

English learners (ELs) represent one of the fastest growing student populations in the United States. Given that language can serve as a barrier in EL performance, test accommodations are provided to help level the playing field and allow ELs to better demonstrate their true performance level. Test accommodations on the computer offer the ability to collect new types of data difficult to obtain via paper-and-pencil tests. Specifically, these data can be used as additional sources of validity evidence when examining test accommodations. To date, limited research has examined computer-based accommodations, thus limiting these additional sources of validity evidence. The purpose of this study was to evaluate the validity of computer-based test accommodations on high school History and Math assessments using evidence based on response processes, specifically accommodation use and response time. Two direct linguistic accommodations, non-ELs, two EL groups, and five research questions were investigated in this study. Accommodation use results indicated significant differences in use across the three student groups, with ELs using accommodations more frequently than non-ELs. However, there were still high percentages of all three groups not accessing any accommodations on individual items. Accommodation use was more common on History than on Math, and decreased as the assessment progressed. Results suggest future research focus on students actually using the accommodations when conducting research on the effectiveness of accommodations. Response time results showed ELs taking longer to process test items as compared to non-ELs regardless of receiving test accommodations. Receiving accommodations significantly impacted processing time for some of the items on History, but not on Math. Similarly, History showed a relationship between the number of accommodations on test items and response time, but Math did not. These results suggested that the Math content knowledge may have played a larger role in response time than the accommodations. Positive relationships between test performance and response time were found in both subject areas. The most common predictors of both accommodation use and response time across both subject areas were sex, Hispanic status, and socioeconomic status. Implications of the results and suggestions for future research are discussed.

This case study examined the efficiency and precision of computer classification and adaptive testing to elicit responses from test coordinators on implementing a high-stakes computer-based testing. Test coordinators from five elementary schools located in a Georgia school district participated in the study. The school district administered state-made, high-stakes tests using paper and pencil; locally developed tests via the computer or paper and pencil. A post-hoc simulation program, Comprehensive Simulation of Computerized Adatpive Testing, used 586 student item responses to produce results with a variable termination point and classification termination point. Results from the simulation were analyzed and used in the case study to elicit interview responses from test coordinators. The photographs of computer-labs and test schedule documents were collected and analyzed to validate school test coordinators' responses. Test coordinators responded positively to the efficiency and precision of simulation results. Some test coordinators preferred the use of computer-adaptive tests for diagnostic purposes only. Test coordinators experiences focused on the security, the emotions, and the management of testing. The findings of this study will benefit those interested in implementing a high-stakes, computer-based testing program by recommending a simulation study be conducted and feedback by solicited from test coordinators prior to an operational test administration.

Made available in DSpace on 2016-03-15T19:39:44Z (GMT). No. of bitstreams: 1 Vera Rocha Reis Lellis.pdf: 3028729 bytes, checksum: 995a8687ff78d6788c884890cf7a3ba1 (MD5) Previous issue date: 2011-02-23 / Coordenação de Aperfeiçoamento de Pessoal de Nível Superior / Attention can be understood as a group of neural processes that allow an adequate selection of information from the environment, in this way orienting adaptive responses. Thus, its study is made necessary so that cognitive processes and how their alterations express themselves through adaptive difficulties can be comprehended, such as it is observed in cases of Attention Deficit Disorder and/or Hyperactivity (ADDH). This study aims at performing a neuropsychological and behavioral evaluation, centered in the analysis of the attentional process of 78 junior high children, enrolled in a private school, with ages between 6 and 11. Traditional manuscript tests and computer-based tests were employed for the evaluation of attention. The instruments used were: WISC-III; Wisconsin; Focused Attention (AC and TECON) and Diffuse Attention (TEDIF) In behavioral evaluation the Behavioral Scale (CBCL/6-18), applied to parents, was used. For the teachers, the Signs of Disattention and Hyperactivity Scale (Benzic) was employed, along with the Behavioral Scale (TRF). The computer-based tests evaluated the voluntary and automatic orientation, temporal direction an attention management. The results from the computer-based instruments were compared to the results from the traditional pencil and paper tests, along with the behavioral and the Signs of Disattention and Hyperactivity scales. No indicators of behavioral problems were observed, and there was no report from either parents (CBCL) or teachers (TRF) on that matter. The results of the traditional tests related to Focused Attention (FC, TECON-1 and TECON-2, TEDIF 1 and TEDIF-3) showed significantly higher scores and points related to schooling. The computer-based tests, on the other hand, showed a systematic lowering of the reaction time related to the increase in schooling. However, as schooling increases, there is a decrease in the differences between the valid and the invalid conditions, making participants more efficient in perceiving stimuli outside the indicated location. Another factor that presented significant difference was the Clue Target relation factor, in which it is possible to observe that the reaction time for the condition in which the target and the clue come from the same side (ipsolateral condition) is lower than the reaction time for the condition in which the target and the clue come from opposite sides (contralateral condition). With the Wisconsin test is possible to verify that with the increase in schooling there is a decrease in the number of tests administered and also an increase in the number of categories completed, as well as a decrease on the total amount of errors and the number of preservative answers, and that this fact correlates with the decrease in the reaction time in computer-based tests, demonstrating that the skills that were evaluated by each one of this tests are also correlated. / A atenção pode ser compreendida enquanto conjunto de processos neurais que possibilitam a seleção adequada de informações do ambiente e que por isso orientam respostas adaptativas. Desse modo, seu estudo é necessário para a compreensão dos processos cognitivos e de como suas alterações se expressam por dificuldades adaptativas, tais como as observadas em casos de Transtorno de Déficit de Atenção e/ou Hiperatividade (TDAH). O objetivo deste estudo foi o de fazer uma avaliação neuropsicológica e comportamental centrada na análise do processo atencional de 78 crianças do Ensino Fundamental I, matriculadas em uma escola particular e com idade entre 6 e 11 anos. Foram empregados testes tradicionais de realização em lápis e papel e testes computadorizados para avaliação da atenção. Os instrumentos utilizados foram: WISC-III; Wisconsin; Atenção Concentrada (AC e TECON) e Atenção Difusa (TEDIF). Na avaliação comportamental foi utilizada a Escala Comportamental (CBCL/6 18) aplicada aos pais. Para os professores foram empregadas a Escala de Sinais de Desatenção e Hiperatividade (Benczik), além de Escala Comportamental (TRF). Os testes computadorizados avaliaram a orientação voluntária e automática, o direcionamento temporal e a manutenção da atenção. Foram comparados os resultados dos instrumentos computadorizados com os resultados dos testes tradicionais de respostas em lápis e papel e com as escalas comportamentais e de sinais de desatenção e hiperatividade. Não foram observados nos participantes indicadores de problemas de comportamento relato dos pais (CBCL) ou pelos professores (TRF). Os resultados dos testes tradicionais relacionados à Atenção Concentrada (AC, TECON-1 e TECON -2, TEDIF-1 e TEDIF-3) demonstram aumento significativo de acertos e pontos em função da escolaridade. Já nos testes computadorizados houve uma diminuição sistemática do tempo de reação em função do aumento da escolaridade. Entretanto, à medida que a escolaridade aumenta há uma diminuição das diferenças entre a condição válida e inválida, tornando os participantes mais eficientes em perceber estímulos fora dos locais indicados. Outro fator que apresentou diferença significativa foi quanto ao tempo de reação para a condição aonde pista alvo vêm na mesma posição (condição ipsolateral) é menor do que o tempo de reação para a condição na qual a pista e o alvo vêm em lados opostos (condição contralateral). No teste Wisconsin é possível verificar que com o aumento da escolaridade há uma diminuição no número de ensaios administrados e também um aumento de número de categorias completadas bem como diminuição no total de erros e de número de respostas perseverativas, e que tal fato se correlaciona com a diminuição do tempo de reação nos testes computadorizados, demonstrando que as habilidades avaliadas por cada um desses testes também estão correlacionadas.