Cognitive diagnostic assessment (CDA) is a testing format that employs a cognitive model to, first, develop or identify items measuring specific knowledge and skills and, then, use this model to direct psychometric analyses of examinees item response patterns to promote diagnostic inferences. The attribute hierarchy method (AHM, Leighton, Gierl, & Hunka, 2004) is a psychometric procedure for classifying examinees test item responses into a set of structured attribute patterns associated with different components from a cognitive model of task performance. Attribute reliability is a fundamental concept in cognitive diagnostic assessment because it refers to the consistency of the decisions made in diagnostic test about examinees mastery of specific attributes. In this study, an adapted attribute-based reliability estimate was evaluated in comparison of the standard Cronbachs alpha using simulated data. Factors expected to influence attribute reliability estimates, including test length, sample size, model structure, and model-data fit level, were also studied. Results of this study revealed that the performances of the two attribute-based reliability estimation indices are comparable; however, the adapted index is conceptually more meaningful. Test length, model structure, and model-data fit were shown to impact attribute reliability estimates differentially. Implications to researchers and practitioners were given based on the simulation results. Limitations of the present study and future directions were also discussed. / Measurement, Evaluation, and Cognition

No description available.

When applying cognitive diagnostic models, the goal is to accurately estimate students’ diagnostic profiles. The accuracy of these estimates may be enhanced by looking at the types of incorrect options a student selects. This thesis research examines the additional diagnostic information available from the distractors in multiple-choice items used in large-scale achievement assessments and identifies optimal conditions for extracting diagnostic information. The study is based on the analyses of both real student responses and simulated data. The real student responses are from a large-scale provincial math assessment for grade 6 students in Ontario. Data were then simulated under different skill dimensionality and item discrimination conditions. Comparisons were made between student profile estimates when using the DINA and MC-DINA models. The MC-DINA model is a newly developed cognitive diagnostic model where the probability of a student choosing a particular item option depends on how closely the student’s cognitive skill profile matches the skills tapped by that option. The results from the simulation data analysis suggested that when the simulated data included additional diagnostic information in the distractors, the MC-DINA model was able to use that information to improve the estimation of the student profiles, which shows the utility of the additional information obtained from item distractors. The value of adding information from distractors was greater when there was lower item discrimination and more skill multidimensionality. However, in the real data, the keyed options provided more diagnostic information than the distractors, and there was little information in the distractors that could be utilized by the MC-DINA model. This implies that current math test items could be further developed to include diagnostically rich distractors. The study offers some suggestions for a design of multiple-choice test items and its formative use.

When applying cognitive diagnostic models, the goal is to accurately estimate students’ diagnostic profiles. The accuracy of these estimates may be enhanced by looking at the types of incorrect options a student selects. This thesis research examines the additional diagnostic information available from the distractors in multiple-choice items used in large-scale achievement assessments and identifies optimal conditions for extracting diagnostic information. The study is based on the analyses of both real student responses and simulated data. The real student responses are from a large-scale provincial math assessment for grade 6 students in Ontario. Data were then simulated under different skill dimensionality and item discrimination conditions. Comparisons were made between student profile estimates when using the DINA and MC-DINA models. The MC-DINA model is a newly developed cognitive diagnostic model where the probability of a student choosing a particular item option depends on how closely the student’s cognitive skill profile matches the skills tapped by that option. The results from the simulation data analysis suggested that when the simulated data included additional diagnostic information in the distractors, the MC-DINA model was able to use that information to improve the estimation of the student profiles, which shows the utility of the additional information obtained from item distractors. The value of adding information from distractors was greater when there was lower item discrimination and more skill multidimensionality. However, in the real data, the keyed options provided more diagnostic information than the distractors, and there was little information in the distractors that could be utilized by the MC-DINA model. This implies that current math test items could be further developed to include diagnostically rich distractors. The study offers some suggestions for a design of multiple-choice test items and its formative use.

No description available.

Background: The amount of time that it takes to get a breast cancer diagnosis is very important to patients. The Ontario diagnostic assessment unit (DAU) is designed to improve the quality and timeliness of care during a breast cancer diagnosis. This study described and examined the association between the length of the diagnostic interval and DAU use in Ontario, Canada. Methods: This was a retrospective cohort study among all breast cancer patients diagnosed between Jan 1st, 2011 and Dec 31st, 2011 in Ontario, Canada. DAU use and diagnostic intervals were described. The association between DAU use and the diagnostic interval was examined separately in a cohort of 2499 screen-detected patients and a cohort of 4381 symptomatic patients. Study data sources included administrative databases available at the Institute for Clinical Evaluative Sciences (ICES) and Cancer Care Ontario (CCO). The diagnostic interval was defined as the time from the index contact to the cancer diagnosis. DAU use was determined based on the payment record within the organized screening program as well as the hospital where patients were diagnosed. Multivariate median regressions were used to control for possible confounders. Results: On average, Ontario breast cancer patients waited 4.6 weeks to be diagnosed. Forty-eight percent were diagnosed in a DAU and 52% were diagnosed in the usual care route. In screen-detected patients, DAUs had a higher rate in meeting national timeliness targets compared to usual care (79.1% vs. 70.2%, p<0.001). DAU use was significantly associated with an 8.3-day decrease in the time to diagnosis (95% CI: 6.5-10.2) after controlling for potential confounders. In symptomatic patients, DAUs also had a higher rate in achieving the Canadian timeliness targets compare to usual care (71.7% vs. 58.1%, p<0.001). DAUs significantly reduced the time to diagnosis by 10 days (95% CI: 7.8-11.9) after controlling for possible confounders. Conclusions: We observed considerable variation in breast cancer diagnostic intervals and DAU use in Ontario. Use of Ontario DAUs was associated with improved diagnostic timeliness for breast cancer patients. / Thesis (Master, Community Health & Epidemiology) -- Queen's University, 2013-11-18 13:19:03.231

CoordenaÃÃo de AperfeiÃoamento de Pessoal de NÃvel Superior / nÃo hà / Atualmente, alcanÃar qualidade na educaÃÃo à uma meta que nÃo està desvinculada da realizaÃÃo de procedimentos que envolvem alguma modalidade de avaliaÃÃo. O tipo, a periodicidade e o formato dessas avaliaÃÃes dependem dos fins a que ela servirà em cada situaÃÃo. Uma avaliaÃÃo com funÃÃes de diagnÃstico tem como finalidade possibilitar um conhecimento prÃvio sobre a caracterizaÃÃo individual e coletiva de uma turma de alunos, facilitando o planejamento pedagÃgico para determinado perÃodo letivo. O objetivo principal desse estudo concentrou-se em investigar como a avaliaÃÃo diagnÃstica poderia contribuir para o Ãxito em alfabetizar crianÃas, bem como identificar fatores intervenientes nesse processo. Esse tipo de avaliaÃÃo à praticado, desde o ano de 2007, no CearÃ, como uma das aÃÃes do Eixo de AvaliaÃÃo Externa do Programa AlfabetizaÃÃo na Idade Certa (PAIC), no qual a pesquisadora atua como especialista. Ao longo da experiÃncia profissional nesse programa, foram se delineando vÃrias questÃes que implicavam uma investigaÃÃo mais profunda sobre o que, de fato, causava algumas discrepÃncias nos resultados dos municÃpios envolvidos. Em se tratando de um processo padronizado de avaliaÃÃo e de procedimentos que garantiam o acesso igualitÃrio aos resultados, podia-se imaginar que havia outros fatores envolvidos no problema, pois nÃo bastava apenas haver a avaliaÃÃo e a divulgaÃÃo e discussÃo dos resultados. Portanto, a hipÃtese principal deste trabalho asseverou que a prÃtica da avaliaÃÃo diagnÃstica da alfabetizaÃÃo contribui para a melhoria da aprendizagem quando os sujeitos envolvidos no processo desenvolvem atitudes positivas em relaÃÃo a essa prÃtica e Ãs aÃÃes dela decorrentes. Sendo assim, fatores ligados Ãs atitudes dos sujeitos envolvidos no processo de alfabetizaÃÃo poderiam exercer maiores influÃncias do que os fatores socioeconÃmicos. Dessa forma, foi realizado um estudo, em dois municÃpios cearenses, com uma amostra de 72 alunos e 4 professoras de duas escolas privadas, mas com perfil equivalente aos alunos e professores participantes do PAIC, o que garantiu a aplicabilidade da pesquisa. Aos alunos foram aplicados os mesmos instrumentos de avaliaÃÃo utilizados no Programa, bem como outros que foram desenvolvidos para serem aplicados como teste. Os resultados do desempenho dos alunos nas avaliaÃÃes foram registrados, analisados e apresentados nÃo apenas na dimensÃo quantitativa, mas, sobretudo, numa dimensÃo com um expressivo teor pedagÃgico que auxiliou na compreensÃo dos avanÃos dos alunos. AlÃm dessa anÃlise, o estudo procurou investigar, atravÃs do discurso dos sujeitos, os possÃveis fatores que estariam imbricados a todo o processo de aprendizagem dos alunos. Para esta etapa foi utilizado o mÃtodo hermenÃutico, prÃprio da pesquisa de enfoque fenomenolÃgico. A pesquisa apresentou como resultados gerais os seguintes pontos: (1) Foi possÃvel identificar os avanÃos dos alunos na consolidaÃÃo de habilidades bÃsicas da alfabetizaÃÃo em leitura e escrita, devido à aplicaÃÃo de uma avaliaÃÃo diagnÃstica em maio de 2009 e uma avaliaÃÃo comparativa em novembro de 2009; (2) Houve um expressivo avanÃo qualitativo no desempenho dos alunos avaliados; (3) Das trÃs turmas investigadas, uma apresentou melhores resultados dos alunos de nÃvel socioeconÃmico inferior. Nas outras duas, os resultados dos alunos de ambos os nÃveis foram bastante similares; (4) Alguns aspectos subjetivos das professoras e dos pais emergiram como fatores que contribuÃram para uma melhor aprendizagem dos alunos, foram eles: vontade, compromisso, acolhimento e reflexividade; (5) Ficou evidente que um trabalho de acompanhamento que envolva atividades de supervisÃo do trabalho docente e de formaÃÃo continuada à essencial para a qualidade das aÃÃes pedagÃgicas desenvolvidas. A pesquisa, por fim, encontrou evidÃncias de que a prÃtica da avaliaÃÃo diagnÃstica pode contribuir significativamente para nortear os caminhos do Ãxito na alfabetizaÃÃo de crianÃas, quando houver um verdadeiro compromisso dos sujeitos envolvidos no processo em utilizar adequadamente os resultados, planejar e executar as intervenÃÃes pedagÃgicas e acompanhar o trabalho docente. / Nowadays, reaching quality in education is a target that cannot be separated of any kind of assessment procedures. The type, the frequency and the conditions of those assessments depend on their purpose in any situation. A diagnostic assessment can provide a previous knowledge about a class of students, making easier the pedagogical plans to a certain period of school time. The main objective of this study was to investigate how the diagnostic assessment could contribute to the success in literacy process of children, as well as, to identify some intervening factors in this process. In Ceara, since 2007, the Literacy in Right Age Program (PAIC) has been practiced this kind of assessment, in which the researcher works as a specialist. During the professional experience in this program, some questions about the causes of the differences in results among some cities, schools and classes were asking for a further investigation. The standardized procedures on how to use the results of diagnostic assessment were not sufficient to guarantee that all classes and students had good individual results. So, probably, there were any other factors involved. The main hypothesis of this study said that the diagnostic assessment contributes to the increase of learning when people involved in such process develop positive attitudes in relation of this practice and the actions that occur after it. Therefore, some factors linked to the individualsâ attitudes could influence more than socioeconomic factors. The study took place in two cities of Ceara, which involved 72 students and 4 teachers of two private schools, but having equivalent profile with the students and teachers of PAIC, which guaranteed the applicability of the study. The tests applied to the 72 students were the same applied by the Program. The results of students learning performance were registered, analyzed and presented not only in quantitative dimension, but also in a pedagogic one that helped in the comprehension of studentsâ development. Besides of this analysis, this study investigated, through the individualsâ speeches, the possible factors which would be causing influence in learning process. In this part, the hermeneutic method was used. This study presented as the main results: (1) It was possible to identify the learning development in reading and writing skills, because the appliance of a diagnostic assessment in May, 2009 and a comparative one in November, 2009; (2) The students showed a qualitative development of their performance; (3) The students from a lower socioeconomic level in one of the three investigated classes had better results. In the other two classes, the studentsâ results were very similar; (4) Some subjective aspects of the teachers and parents appeared as factors that contributed to a better learning process. They were: will, commitment, acceptance and praxis; (5) Monitoring activities which involve supervision and teachersâ continuing formation is essential to the quality of the pedagogic actions. This study found evidences that the diagnostic assessment can contribute meaningfully to guide the ways of the success in children literacy acquisition, when individuals involved into this process develop commitment to use the assessmentsâ results, plan and execute the pedagogic actions and monitoring didactic work.

No description available.

A simulation study was conducted to explore the performance of the linear logistic test model (LLTM) when the relationships between items and cognitive components were misspecified. Factors manipulated included percent of misspecification (0%, 1%, 5%, 10%, and 15%), form of misspecification (under-specification, balanced misspecification, and over-specification), sample size (20, 40, 80, 160, 320, 640, and 1280), Q-matrix density (60% and 46%), number of items (20, 40, and 60 items), and skewness of person ability distribution (-0.5, 0, and 0.5). Statistical bias, root mean squared error, confidence interval coverage, confidence interval width, and pairwise cognitive components correlations were computed. The impact of the design factors were interpreted for cognitive components, item difficulty, and person ability parameter estimates. The simulation provided rich results and selected key conclusions include (a) SAS works superbly when estimating LLTM using a marginal maximum likelihood approach for cognitive components and an empirical Bayes estimation for person ability, (b) parameter estimates are sensitive to misspecification, (c) under-specification is preferred to over-specification of the Q-matrix, (d) when properly specified the cognitive components parameter estimates often have tolerable amounts of root mean squared error when the sample size is greater than 80, (e) LLTM is robust to the density of Q-matrix specification, (f) the LLTM works well when the number of items is 40 or greater, and (g) LLTM is robust to a slight skewness of the person ability distribution. In sum, the LLTM is capable of identifying conceptual knowledge when the Q-matrix is properly specified, which is a rich area for applied empirical research.

This study built and supported arguments for the use of diagnostic assessment in English as a second language (ESL) academic writing. In the two-phase study, a new diagnostic assessment scheme, called the Empirically-derived Descriptor-based Diagnostic (EDD) checklist, was developed and validated for use in small-scale classroom assessment. The checklist assesses ESL academic writing ability using empirically-derived evaluation criteria and estimates skill parameters in a way that overcomes the problems associated with the number of items in diagnostic models. Interpretations of and uses for the EDD checklist were validated using five assumptions: (a) that the empirically-derived diagnostic descriptors that make up the EDD checklist are relevant to the construct of ESL academic writing; (b) that the scores derived from the EDD checklist are generalizable across different teachers and essay prompts; (c) that performance on the EDD checklist is related to performance on other measures of ESL academic writing; (d) that the EDD checklist provides a useful diagnostic skill profile for ESL academic writing; and (e) that the EDD checklist helps teachers make appropriate diagnostic decisions and has the potential to positively impact teaching and learning ESL academic writing. Using a mixed-methods research design, four ESL writing experts created the EDD checklist from 35 descriptors of ESL academic writing. These descriptors had been elicited from nine ESL teachers’ think-aloud verbal protocols, in which they provided diagnostic feedback on ESL essays. Ten ESL teachers utilized the checklist to assess 480 ESL essays and were interviewed about its usefulness. Content reviews from ESL writing experts and statistical dimensionality analyses determined that the underlying structure of the EDD checklist consists of five distinct writing skills: content fulfillment, organizational effectiveness, grammatical knowledge, vocabulary use, and mechanics. The Reduced Reparameterized Unified Model (Hartz, Roussos, & Stout, 2002) then demonstrated the diagnostic quality of the checklist and produced fine-grained writing skill profiles for individual students. Overall teacher evaluation further justified the validity claims for the use of the checklist. The pedagogical implications of the use of diagnostic assessment in ESL academic writing were discussed, as were the contributions that it would make to the theory and practice of second language writing instruction and assessment.