Thesis (Ed.M.)--Boston University.

The use of constructed response (CR) items or performance tasks to assess test takers' ability has grown tremendously over the past decade. Examples of CR items in psychological and educational measurement range from essays, works of art, and admissions interviews. However, unlike multiple-choice (MC) items that have predetermined options, CR items require test takers to construct their own answer. As such, they require the judgment of multiple raters that are subject to differences in perception and prior knowledge of the material being evaluated. As with any scoring procedure, the scores assigned by raters must be comparable over time and over different test administrations and forms; in other words, scores must be reliable and valid for all test takers, regardless of when an individual takes the test. This study examines how longitudinal patterns or changes in rater behavior affect model-based classification accuracy. Rater drift refers to changes in rater behavior across different test administrations. Prior research has found evidence of drift. Rater behavior in CR scoring is examined using two measurement models - latent class signal detection theory (SDT) and item response theory (IRT) models. Rater effects (e.g., leniency and strictness) are partly examined with simulations, where the ability of different models to capture changes in rater behavior is studied. Drift is also examined in two real-world large scale tests: teacher certification test and high school writing test. These tests use the same set of raters for long periods of time, where each rater's scoring is examined on a monthly basis. Results from the empirical analysis showed that rater models were effective to detect changes in rater behavior over testing administrations in real-world data. However, there were differences in rater discrimination between the latent class SDT and IRT models. Simulations were used to examine the effect of rater drift on classification accuracy and on differences between the latent class SDT and IRT models. Changes in rater severity had only a minimal effect on classification. Rater discrimination had a greater effect on classification accuracy. This study also found that IRT models detected changes in rater severity and in rater discrimination even when data were generated from the latent class SDT model. However, when data were non-normal, IRT models underestimated rater discrimination, which may lead to incorrect inferences on the precision of raters. These findings provide new and important insights into CR scoring and issues that emerge in practice, including methods to improve rater training.

This research aims to develop an MCMC algorithm for estimating the Q-matrix in a Bayesian framework. A saturated multinomial model was used to estimate correlated attributes in the DINA model and rRUM. Closed-forms of posteriors for guess and slip parameters were derived for the DINA model. The random walk Metropolis-Hastings algorithm was applied to parameter estimation in the rRUM. An algorithm for reducing potential label switching was incorporated into the estimation procedure. A method for simulating data with correlated attributes for the DINA model and rRUM was offered. Three simulation studies were conducted to evaluate the algorithm for Bayesian estimation. Twenty simulated data sets for simulation study 1 were generated from independent attributes for the DINA model and rRUM. A hundred data sets from correlated attributes were generated for the DINA and rRUM with guess and slip parameters set to 0.2 in simulation study 2. Simulation study 3 analyzed data sets simulated from the DINA model with guess and slip parameters generated from Uniform (0.1, 0.4). Results from simulation studies showed that the Q-matrix recovery rate was satisfactory. Using the fraction-subtraction data, an empirical study was conducted for the DINA model and rRUM. The estimated Q-matrices from the two models were compared with the expert-designed Q-matrix.

There has been growing interest in recent years in using cognitive diagnosis models for diagnostic measurement, i.e., classification according to multiple discrete latent traits. The Q-matrix, an incidence matrix specifying the presence or absence of a relationship between each item in the assessment and each latent attribute, is central to many of these models. Important applications include educational and psychological testing; demand in education, for example, has been driven by recent focus on skills-based evaluation. However, compared to more traditional models coming from classical test theory and item response theory, cognitive diagnosis models are relatively undeveloped and suffer from several issues limiting their applicability. This thesis exams several issues related to statistical inference and experimental design for Q-matrix based cognitive diagnosis models. We begin by considering one of the main statistical issues affecting the practical use of Q-matrix based cognitive diagnosis models, the identifiability issue. In statistical models, identifiability is prerequisite for most common statistical inferences, including parameter estimation and hypothesis testing. With Q-matrix based cognitive diagnosis models, identifiability also affects the classification of respondents according to their latent traits. We begin by examining the identifiability of model parameters, presenting necessary and sufficient conditions for identifiability in several settings. Depending on the area of application and the researcher's degree of control over the experiment design, fulfilling these identifiability conditions may be difficult. The second part of this thesis proposes new methods for parameter estimation and respondent classification for use with non-identifiable models. In addition, our framework allows consistent estimation of the severity of the non-identifiability problem, in terms of the proportion of the population affected by it. The implications of this measure for the design of diagnostic assessments are also discussed.

The research of cognitive diagnostic models (CDMs) is becoming an important field of psychometrics. Instead of assigning one score, CDMs provide attribute profiles to indicate the mastering status of concepts or skills for the examinees. This would make the test result more informative. The implementation of many CDMs relies on the existing item-to-attribute relationship, which means that we need to know the concepts or skills each item requires. The relationships between the items and attributes could be summarized into the Q-matrix. Misspecification of the Q-matrix will lead to incorrect attribute profile. The Q-matrix can be designed by expert judgement, but it is possible that such practice can be subjective. There are previous researches about the Q-matrix estimation. This study proposes an estimation method for one of the most parsimonious CDMs, the DINA model. The method estimates the Q-matrix for DINA model by setting constraints on the generalized DINA model. In the simulation study, the results showed that the estimated Q-matrix fit better the empirical fraction subtraction data than the expert-design Q-matrix. We also show that the proposed method may still be applicable when the constraints were relaxed.

Thesis (Ed.D.)--Boston University.

This research was designed to ascertain the effect of student self-evaluation, teacher evaluation, and feedback, and the absence of formal evaluation of college drafting assignments upon student achievement and knowledge retention, and attitudes of students in drafting technology/CAD. More specifically, the study was designed to test the following hypotheses: (1) no significant difference existed among scores representing achievement of college drafting students experiencing three evaluation approaches, (2) no significant difference existed among the scores representing the cognitive achievement (retention) of college drafting students experiencing three evaluation approaches five weeks after treatment, and, (3) no significant difference existed among the attitude scores of students experiencing different evaluation approaches. / The participants in the study were 39 undergraduate students enrolled in the Industrial Studies 123 Technical Drafting class in the Department of Industrial Studies at the University of Wisconsin-Platteville during the fall semester of the 1990-91 academic year. The students were randomly assigned to the three groups as follows: (1) a control group with no-evaluation; and the two experimental groups, (2) a student self-evaluation group; and (3) a teacher-evaluation group. / A one-way analysis of covariance was utilized to test the hypotheses at a 0.05 level of significance. Based on the findings and conditions of this study, the following conclusions were made: (1) the no-evaluation and teacher-evaluation groups did demonstrate an increased growth of achievement and knowledge retention over the self-evaluation group; (2) the evaluation method has no effect on the achievement and knowledge retention of college drafting students; and, (3) the attitude of the students toward CAD was found to be very positive. / Source: Dissertation Abstracts International, Volume: 53-04, Section: A, page: 1136. / Major Professor: Hollie B. Thomas. / Thesis (Ph.D.)--The Florida State University, 1992.

Thesis (M. Ed.)--University of Hong Kong, 2006. / Title proper from title frame. Also available in printed format.

Educational institutions are exposed to a continuously changing environment. As in business, they experiment with their methods and techniques to improve their outputs. Performance assessment, both in education and business is a gray science. It is not just a question of evaluating performance. Performance is comprised of a number of components such as motivation, ability, organizational support, and rewards. This study looked at the relationship between educational assessment and the performance components. An assessment tool called the Balanced Scorecard, which has seen great success in the world of business, was adapted for use in the classroom. The study utilized quasi-experimental design to compare the effects of the broad-based holistic measurement associated with a balanced scorecard, and a traditional grading structure in two topics-based college courses. The study found that motivation, individual equity, satisfaction, and student engagement were all significantly higher in the experimental group by comparison. This evidence suggests that by utilizing a broad base of performance measures, one can increase student motivation and engagement in the learning process. / Graduation date: 1999

Thesis (Ph. D.)--University of North Carolina at Greensboro, 2006. / Title from PDF title page screen. Advisor: Terry Ackerman, Robert Henson; submitted to the School of Education. Includes bibliographical references (p. 121-125).