In the theory of measurement, there are two competing measurement frameworks, classical test theory and item response theory. The present study empirically examined, using large scale norm-referenced data, how the item and person statistics behaved under the two competing measurement frameworks. The study focused on two central themes: (1) How comparable are the item and person statistics derived from the item response and classical test framework? (2) How invariant are the item statistics from each measurement framework across examinee samples? The findings indicate that, in a variety of conditions, the two measurement frameworks produce similar item and person statistics. Furthermore, although proponents of item response theory have centered their arguments for its use on the property of invariance, classical test theory statistics, for this sample, are just as invariant.

In the theory of measurement, there are two competing measurement frameworks, classical test theory and item response theory. The present study empirically examined, using large scale norm-referenced data, how the item and person statistics behaved under the two competing measurement frameworks. The study focused on two central themes: (1) How comparable are the item and person statistics derived from the item response and classical test framework? (2) How invariant are the item statistics from each measurement framework across examinee samples? The findings indicate that, in a variety of conditions, the two measurement frameworks produce similar item and person statistics. Furthermore, although proponents of item response theory have centered their arguments for its use on the property of invariance, classical test theory statistics, for this sample, are just as invariant.

Item Response Theory (IRT) methodologies were employed in order to examine the Myers-Briggs Type Indicator (MBTI) for differential item functioning (DIF) on the basis of crossed gender and ethnicity variables. White males were the reference group, and the focal groups were: black females, black males, and white females. The MBTI was predicted to show DIF in all comparisons. In particular, DIF on the Thinking-Feeling scale was hypothesized especially in the comparisons between white males and black females and between white males and white females. A sample of 10,775 managers who took the MBTI at assessment centers provided the data for the present experiment. The Mantel-Haenszel procedure and an IRT-based area technique were the methods of DIF-detection. Results showed several biased items on all scales for all comparisons. Ethnicitybased bias was seen in the white male vs. black female and white male vs. black male comparisons. Gender-based bias was seen particularly in the white male vs. white female comparisons. Consequently, the Thinking-Feeling showed the least DIF of all scales across comparisons, and only one of the items differentially scored by gender was found to be biased. Findings indicate that the gender-based differential scoring system is not defensible in managerial samples, and there is a need for further research into the study of differential item functioning with regards to ethnicity. / Master of Science

Estágio realizado no INESC e orientado pelo Eng.º Pedro Miguel Carvalho / Tese de mestrado integrado. Engenharia Electrotécnica e de Computadores (Major Telecomunicações). Faculdade de Engenharia. Universidade do Porto. 2009

No description available.

Diese Publikation wurde im Rahmen des Seminars Psychometric Methods erstellt. Dabei handelt es sich um eine Lehrveranstaltung, die jedes Semester am Institut für Statistik und Mathematik der Wirtschaftsuniversität Wien ­ mit wechselnden thematischen Schwerpunkten ­ abgehalten wird. Im Wintersemester 2009/2010 lag der Fokus auf der Anwendung von Item-Response-Software. Zur Anwendung psychometrischer Methoden steht eine Vielzahl von Programmen zur Verfügung, die jeweils unterschiedliche Verfahren und Modelle anbieten. In diesem Seminar ging es im Wesentlichen darum, einen Überblick über die vorhandene Software zu bekommen, sowie die Stärken und Schwächen der einzelnen Programme herauszuarbeiten. Weiters sollten die Teilnehmer in die Lage versetzt werden, verschiedene psychometrische Modelle bei unterschiedlichen Problemstellungen praktisch anzuwenden. Im Rahmen des Seminars wurden von verschiedenen Teilnehmergruppen jeweils ein bestimmtes Programm vorgestellt. Einerseits wurden die theoretischen Hintergründe und Modelle aufbereitetet und andererseits die jeweiligen Programme mittels Live-Präsentationen von Datenanalysen vorgeführt. Dadurch bekamen alle Beteiligten einen Einblick, welche Modelle in den unterschiedlichen Softwarepaketen umgesetzt sind, wie man sie anwenden und interpretieren kann und auch, wie man praktisch mit ihnen umgeht. Damit die gewonnenen Erfahrungen auch für andere nutzbar werden haben wir die Gruppenbeiträge gesammelt herausgegeben. Die einzelnen Kapitel sollen jeweils eine Brücke zwischen den theoretisch-technischen Aspekten und anwendungsorientierten-praktischen Aspekten der einzelnen Progamme schlagen. Wichtig war uns auch die Auswahl der vorgestellten Softwarepakete, wobei sich der Bogen von etablierten und weitverbreiteten Programmen (z.B. BILOG oder MULTILOG) bis zu eher selten verwendenten Programmen (bspw. GGUM oder ScoRight) spannt. Ohne Anspruch auf Vollständigkeit hoffen wir mit diesem Buch einen Einblick in die wichtigsten Softwarepakete zu geben, wobei wir auf eine verständliche Erklärung theoretischer Hintergründe und möglichst interessante Anwendungsbeispiele großen Wert legten. Unser Ziel war es, interessierten Anwenderinnen und Anwendern eine kleine ,Landkarte' durch den Dschungel verfügbarer IRT- Software bereitzustellen, die zur weiteren Vertiefung anregen soll. Unser Dank gilt den Teilnehmerinnen und Teilnehmern des Seminars, die ihre Beiträge mit viel Engagement und Durchhaltevermögen (für nicht wenige war dieser Artikel die erste Begegnung mit LaTeX) verfasst und überarbeitet haben, sodass dieses Werk zustande kommen konnte.(author's abstract) / Series: Research Report Series / Department of Statistics and Mathematics

Psychometricians and test developers use DIF analysis to determine if there is possible bias in a given test item. This study examines the conditions under which two predominant methods for determining differential item function compare with each other in item bias detection using an effect size statistic as the basis for comparison. The main focus of the present research was to test whether or not incorporating an effect size for LR DIF will more accurately detect DIF and to compare the utility of an effect size index across MH DIF and LR DIF methods. A simulation study was used to compare the accuracy of MH DIF and LR DIF methods using a p value or supplemented with an effect size. Effect sizes were found to increase the accuracy of DIF and the possibility of the detection of DIF across varying ability distributions, population distributions, and sample size combinations. Varying ability distributions and sample size combinations affected the detection of DIF, while population distributions did not seem to affect the detection of DIF.

Psychometricians and test developers use DIF analysis to determine if there is possible bias in a given test item. This study examines the conditions under which two predominant methods for determining differential item function compare with each other in item bias detection using an effect size statistic as the basis for comparison. The main focus of the present research was to test whether or not incorporating an effect size for LR DIF will more accurately detect DIF and to compare the utility of an effect size index across MH DIF and LR DIF methods. A simulation study was used to compare the accuracy of MH DIF and LR DIF methods using a p value or supplemented with an effect size. Effect sizes were found to increase the accuracy of DIF and the possibility of the detection of DIF across varying ability distributions, population distributions, and sample size combinations. Varying ability distributions and sample size combinations affected the detection of DIF, while population distributions did not seem to affect the detection of DIF.

The Multidimensional Generalized Graded Unfolding Model (MGGUM) is a proximity-based, noncompensatory item response theory (IRT) model with applications in the context of attitude, personality, and preference measurement. Model development used fully Bayesian Markov Chain Monte Carlo (MCMC) parameter estimation (Roberts, Jun, Thompson, & Shim, 2009a; Roberts & Shim, 2010). Challenges can arise while estimating MGGUM parameters using MCMC where the meaning of dimensions may switch during the estimation process and difficulties in obtaining informative starting values may lead to increased identification of local maxima. Furthermore, researchers must contend with lengthy computer processing time. It has been shown alternative estimation methods perform just as well as, if not better than, MCMC in the unidimensional Generalized Graded Unfolding Model (GGUM; Roberts & Thompson, 2011) with marginal maximum a posteriori (MMAP) item parameter estimation paired with expected a posteriori (EAP) person parameter estimation being a viable alternative. This work implements MMAP/EAP parameter estimation in the multidimensional model. Additionally, item location initial values are derived from detrended correspondence analysis (DCA) based on previous implementation of correspondence analysis in the GGUM (Polak, 2011). A parameter recovery demonstrates the accuracy of two-dimensional MGGUM MMAP/EAP parameter estimates and a comparative analysis of MMAP/EAP and MCMC demonstrates equal accuracy, yet much improved efficiency of the former method. Analysis of real attitude measurement data also provides an illustrative application of the model.

This study compares item and examinee properties, studies the robustness of IRT models, and examines the difference in robustness when using model-data-fit as a robustness criterion. A conceptualization of robustness as a statistical relationship between model assumption violation and invariance properties has been created in this study based on current understanding on IRT models. Using real data from British Columbia Science Assessments, a series of regressional and canonical analyses were conducted. Scatterplots were used to study possible non-linear relationships. The means and standard deviations of "a" and "c" parameter estimates obtained by applying the three-parameter model to a data sample were used as indices of equal discrimination and non-guessing assumption violation for the Rasch model. The assumption of local independence was taken as being equivalent to the assumption of unidimensionality, and Humphreys' pattern index "p" was used to assess the degree of unidimensionality assumption violation. Means and standard deviations of Yen's Q [i subscript] were used to assess the model-data-fit of items at the total test level. Another statistic to assess the model-data-fit of examinees (D [i subscript]) was created and validated in this study. The mean and standard deviation of D [i subscript] were used to assess model-data-fit of examinees at the total test level. The statistics used in this study for assessing item and ability parameter estimate invariance properties were correlations between estimates obtained from a sample and the estimates obtained from an assessment data file. It was found that model-data-fit of items and model-data-fit of examinees are two statistically independent total test properties of model-data-fit. Therefore, there is a necessity in practice to differentiate model-data-fit of items and model-data-fit of examinees. It was also found that item estimate invariance and ability estimate invariance are statistically independent total test properties of invariance. Therefore, there is also a necessity in practice to differentiate item invariance and ability invariance. When invariance is used as a criterion for robustness, the three-parameter model is robust for all the combinations of sample size and test length. The Rasch model is not robust in terms of ability estimate invariance when a large sample size is combined with a moderate test length, or when a moderate sample size is combined with a long test length. Finally, no significant relationship between model-data-fit and invariance was found. Therefore, results of robustness studies obtained when model-data-fit is used as a criterion and the results when invariance is used as a criterion may be totally different, or even contradictory. Because invariance is the fundamental premise of IRT models, invariance properties rather than model-data-fit should be used as criteria for robustness.