The assessment of noncognitive constructs poses a number of challenges that set it apart from traditional cognitive ability measurement. Of particular concern is the influence of response biases and response styles that can influence the accuracy of scale scores. One strategy to address these concerns is to use alternative item presentation formats (such as multidimensional forced choice (MFC) pairs, triads, and tetrads) that may provide resistance to such biases. A variety of strategies for constructing and scoring these forced choice measured have been proposed, though they often require large sample sizes, are limited in the way that statements can vary in location, and (in some cases) require a separate precalibration phase prior to the scoring of forced-choice responses. This dissertation introduces new item response theory models for estimating item and person parameters from rank-order responses indicating preferences among two or more alternatives representing, for example, different personality dimensions. Parameters for this new model, called the Hyperbolic Cosine Model for Rank order responses (HCM-RANK), can be estimated using Markov chain Monte Carlo (MCMC) methods that allow for the simultaneous evaluation of item properties and person scores. The efficacy of the MCMC parameter estimation procedures for these new models was examined via three studies. Study 1 was a Monte Carlo simulation examining the efficacy of parameter recovery across levels of sample size, dimensionality, and approaches to item calibration and scoring. It was found that estimation accuracy improves with sample size, and trait scores and location parameters can be estimated reasonably well in small samples. Study 2 was a simulation examining the robustness of trait estimation to error introduced by substituting subject matter expert (SME) estimates of statement location for MCMC item parameter estimates and true item parameters. Only small decreases in accuracy relative to the true parameters were observed, suggesting that using SME ratings of statement location for scoring might be a viable short-term way of expediting MFC test deployment in field settings. Study 3 was included primarily to illustrate the use of the newly developed IRT models and estimation methods with real data. An empirical investigation comparing validities of personality measures using different item formats yielded mixed results and raised questions about multidimensional test construction practices that will be explored in future research. The presentation concludes with a discussion of MFC methods and potential applications in educational and workforce contexts.

This dissertation employs modern psychometric techniques to estimate the item characteristics of the Conditional Reasoning Test of Aggression (CRT-A) and three related subscales. Using a sample size of 5,511 respondents, this dissertation provides an accurate assessment of the capability of the CRT-A to measure latent aggression. Additionally, a differential item functioning analysis compares student and applied samples. Results suggest that 21 of the 22 CRT-A items work as intended for the measurement of aggression and all 22 of the CRT-A items are good indicators of their respective factor subscales. Information curves indicate that the CRT-A is best suited for use with individuals who are high on latent aggression.

Researchers interested in metacognition of text comprehension (metacomprehension) have investigated both a knowledge and a monitoring component. Knowledge of comprehension consists of one’s awareness of person, strategy, and task variables and is investigated primarily through interviews and questionnaires. Monitoring of comprehension consists of two equally important abilities: evaluation and regulation. Evaluation involves adults’ ability to assess their understanding during reading, whereas regulation involves their ability to use compensatory strategies to resolve comprehension failures. Monitoring of comprehension is assessed through a variety of paradigms, such as on-line performance measures, error detection, and calibration. Researchers interested in adults’ evaluation ability have frequently employed a calibration paradigm in which adults are asked to take a comprehension test after reading one or more passages and make confidence judgments about their future test performance (predictions) or past test performance (postdictions). Findings indicate that adults are generally poor at evaluating their comprehension, and that a number of variables may influence their performance. However, findings have often been inconsistent, and a clearer picture of adults’ ability is needed. Item Response Theory (IRT) is a modern psychometric approach that has been successfully applied in psychological and educational research. An IRT-based comprehension test may provide a better measure of comprehension than those used in prior research. The main purpose of this study was to develop an IRT-based comprehension test for use in calibration studies. Students were also asked to report their guessing behavior, which was analyzed to determine if guessing influenced postdiction accuracy. Undergraduate and graduate students (n = 1,006) completed a comprehension test, made postdictions after each item, and reported their guessing behavior. Calibration accuracy was measured by comparing students’ test scores and postdictions. Factor analysis and a scree test were used to determine unidimensionality of the data, and chi square statistics were used to determine item fit. The comprehension test was found to be appropriate for distinguishing students at the low end of the ability continuum, but additional items need to be developed to discriminate among students at higher ability levels. Guessing scores were moderately, but significantly (p < .01) correlated with both comprehension performance and postdiction accuracy.

No description available.

El texto completo de este trabajo no está disponible en el Repositorio Académico UPC por restricciones de la casa editorial donde ha sido publicado. / The study evaluated the psychometric properties of the Spanish version of the Obsession with COVID-19 Scale (OCS) in 214 police and members of the armed forces (M age = 29.33 years, SD = 11.28). The one-dimensionality and satisfactory reliability of OCS were confirmed with confirmatory factor analysis, Item Response Theory analysis, Cronbach’s alpha, and McDonald’s omega. The scale is useful for identifying individuals with low levels of persistent and disturbing thoughts about COVID-19. COVID-19 obsession was associated with COVID-19 fear, anxiety, and depression. The OCS is suitable for investigating the psychological impact of COVID-19 on members of the police and armed forces.

Dimensions of a polytomous employee attitude survey were examined for the presence of differential item functioning (DIF) and differential test functioning (DTF) utilizing Raju, van der Linden, & Fleer's (1995) differential functioning of items and tests (DFIT) framework. Comparisons were made between managers and non-managers on the 'Management' dimension and between medical staff and nurse staff employees on both the 'Management' and 'Quality of Care and Service' dimensions. 2 out of 21 items from the manager/non-manager comparison were found to have significant DIF, supporting the generalizability of Lynch, Barnes-Farell, and Kulikowich (1998). No items from the medical staff/nurse staff comparisons were found to have DIF. The DTF results indicated that in two out of the three comparisons 1 item could be removed to create dimensions free from DTF. Based on the current findings implications and future research are discussed. / Master of Science

The fully Bayesian estimation via the use of Markov chain Monte Carlo (MCMC) techniques has become popular for estimating item response theory (IRT) models. The current development of MCMC includes two major algorithms: Gibbs sampling and the No-U-Turn sampler (NUTS). While the former has been used with fitting various IRT models, the latter is relatively new, calling for the research to compare it with other algorithms. The purpose of the present study is to evaluate the performances of these two emerging MCMC algorithms in estimating two two-parameter logistic (2PL) IRT models, namely, the 2PL unidimensional model and the 2PL multi-unidimensional model under various test situations. Through investigating the accuracy and bias in estimating the model parameters given different test lengths, sample sizes, prior specifications, and/or correlations for these models, the key motivation is to provide researchers and practitioners with general guidelines when it comes to estimating a UIRT model and a multi-unidimensional IRT model. The results from the present study suggest that NUTS is equally effective as Gibbs sampling at parameter estimation under most conditions for the 2PL IRT models. Findings also shed light on the use of the two MCMC algorithms with more complex IRT models.

No description available.

No description available.

No description available.