In this report we give a brief introduction to Item Response Theory models and multilevel models. The general assumptions of two classical Item Response Theory, 1PL and 2PL models are discussed. We follow the discussion by introducing a multilevel level framework for these two Item Response Theory Models. We explain Bock and Aitkin's (1981) work to estimate item parameters for these two models. Finally we illustrate these models with a LSAT exam data and two statistical softwares; R project and Stata. / text

Recently, researchers have reformulated Item Response Theory (IRT) models into multilevel models to evaluate clustered data appropriately. Using a multilevel model to obtain item difficulty and person ability parameter estimates that correspond directly with IRT models’ parameters is often referred to as multilevel measurement modeling. Unlike conventional IRT models, multilevel measurement models (MMM) can handle, the addition of predictor variables, appropriate modeling of clustered data, and can be estimated using non-specialized computer software, including SAS. For example, a three-level model can model the repeated measures (level one) of individuals (level two) who are clustered within schools (level three). Limitations in terms of the minimum sample size and number of test items that permit reasonable one-parameter logistic (1-PL) IRT model’s parameters have not been examined for either the two- or three-level MMM. Researchers (Wright and Stone, 1979; Lord, 1983; Hambleton and Cook, 1983) have found that sample sizes under 200 and fewer than 20 items per test result in poor model fit and poor parameter recovery for dichotomous 1-PL IRT models with data that meet model assumptions. This simulation study tested the performance of the two-level and three-level MMM under various conditions that included three sample sizes (100, 200, and 400), three test lengths (5, 10, and 20), three level-3 cluster sizes (10, 20, and 50), and two generated intraclass correlations (.05 and .15). The study demonstrated that use of the two- and three-level MMMs lead to somewhat divergent results for item difficulty and person-level ability estimates. The mean relative item difficulty bias was lower for the three-level model than the two-level model. The opposite was true for the person-level ability estimates, with a smaller mean relative parameter bias for the two-level model than the three-level model. There was no difference between the two- and three-level MMMs in the school-level ability estimates. Modeling clustered data appropriately; having a minimum total sample size of 100 to accurately estimate level-2 residuals and a minimum total sample size of 400 to accurately estimate level-3 residuals; and having at least 20 items will help ensure valid statistical test results. / text

OBJECTIVES: (1) To assess prescribers' ability to recognize clinically significant DDIs, (2) to examine demographic and practice factors that may be associated with prescribers' DDI knowledge, and (3) to evaluate prescribers' perceived usefulness of various DDI information sources.METHODS: This study used a mailed questionnaire sent to a national sample of prescribers based on their past history of DDI prescribing which was determined using data from a pharmacy benefit manager covering over 50 million lives. The survey questionnaire included 14 drug-drug pairs that tested prescribers' ability to recognize clinically important DDIs and five 5-point Likert scale-type questions that assessed prescribers' perceived usefulness of DDI information provided by various sources. Demographic and practice characteristics were collected as well. Rasch analysis was used to evaluate the knowledge and usefulness questions.RESULTS: Completed questionnaires were obtained from 950 prescribers (overall response rate: 7.9%). The number of drug pairs correctly classified by the prescribers ranged from zero to thirteen, with a mean of 6 pairs (42.7%). The percentage of prescribers who correctly classified specific drug pairs ranged from 18.2% for warfarin-cimetidine to 81.2% for acetaminophen with codeine-amoxicillin. Half of the drug pair questions were answered "not sure" by over one-third of the respondents; among which, two were contraindicated. Rasch analysis of knowledge and usefulness questions revealed satisfactory model-data fit and person reliability of 0.72 and 0.61, respectively. A multiple regression analysis revealed that specialists were less likely to correctly identify interactions as compared to prescribers who were generalists. Other important predictors of DDI knowledge included the experience of seeing a harm caused by DDIs and the extent to which the risk of DDIs affected the prescribers' drug selection. ANOVA with the post-hoc Scheffe test indicated that prescribers considered DDI information provided by "other" sources to be more useful than that provided by computerized alert system. CONCLUSIONS: This study suggests that prescribers' DDI knowledge may be inadequate. The study found that for the drug interactions evaluated, generalists performed better than specialists. In addition, this study presents an application of IRT analysis to knowledge and attitude measurement in health science research.

ABSTRACT A MONTE CARLO STUDY INVESTIGATING MISSING DATA, DIFFERENTIAL ITEM FUNCTIONING, AND EFFECT SIZE by Phyllis Garrett The use of polytomous items in assessments has increased over the years, and as a result, the validity of these assessments has been a concern. Differential item functioning (DIF) and missing data are two factors that may adversely affect assessment validity. Both factors have been studied separately, but DIF and missing data are likely to occur simultaneously in real assessment situations. This study investigated the Type I error and power of several DIF detection methods and methods of handling missing data for polytomous items generated under the partial credit model. The Type I error and power of the Mantel and ordinal logistic regression were compared using within-person mean substitution and multiple imputation when data were missing completely at random. In addition to assessing the Type I error and power of DIF detection methods and methods of handling missing data, this study also assessed the impact of missing data on the effect size measure associated with the Mantel, the standardized mean difference effect size measure, and ordinal logistic regression, the R-squared effect size measure. Results indicated that the performance of the Mantel and ordinal logistic regression depended on the percent of missing data in the data set, the magnitude of DIF, and the sample size ratio. The Type I error for both DIF detection methods varied based on the missing data method used to impute the missing data. Power to detect DIF increased as DIF magnitude increased, but there was a relative decrease in power as the percent of missing data increased. Additional findings indicated that the percent of missing data, DIF magnitude, and sample size ratio also influenced the effect size measures associated with the Mantel and ordinal logistic regression. The effect size values for both DIF detection methods generally increased as DIF magnitude increased, but as the percent of missing data increased, the effect size values decreased.

In response to public concern over fairness in testing, conducting a differential item functioning (DIF) analysis is now standard practice for many large-scale testing programs (e.g., Scholastic Aptitude Test, intelligence tests, licensing exams). As highlighted by the Standards for Educational and Psychological Testing manual, the legal and ethical need to avoid bias when measuring examinee abilities is essential to fair testing practices (AERA-APA-NCME, 1999). Likewise, the development of statistical and substantive methods of investigating DIF is crucial to the goal of designing fair and valid educational and psychological tests. Douglas, Roussos and Stout (1996) introduced the concept of item bundle DIF and the implications of differential bundle functioning (DBF) for identifying the underlying causes of DIF. Since then, several studies have demonstrated DIF/DBF analyses within the framework of “unintended” multidimensionality (Oshima & Miller, 1992; Russell, 2005). Russell (2005), in particular, examined the effect of secondary traits on DBF/DTF detection. Like Russell, this study created item bundles by including multidimensional items on a simulated test designed in theory to be unidimensional. Simulating reference group members to have a higher mean ability than the focal group on the nuisance secondary dimension, resulted in DIF for each of the multidimensional items, that when examined together produced differential bundle functioning. The purpose of this Monte Carlo simulation study was to assess the Type I error and power performance of SIBTEST (Simultaneous Item Bias Test; Shealy & Stout, 1993a) for DBF analysis under various conditions with simulated data. The variables of interest included sample size and ratios of reference to focal group sample sizes, correlation between primary and secondary dimensions, magnitude of DIF/DBF, and angular item direction. Results showed SIBTEST to be quite powerful in detecting DBF and controlling Type I error for almost all of the simulated conditions. Specifically, power rates were .80 or above for 84% of all conditions and the average Type I error rate was approximately .05. Furthermore, the combined effect of the studied variables on SIBTEST power and Type I error rates provided much needed information to guide further use of SIBTEST for identifying potential sources of differential item/bundle functioning.

There are concerns regarding the risk of the inaccurate responses in the personality data. The inaccurate responses negatively affect in the individual selection contexts. Especially, in the military context, the personality score including inaccurate responses results in the selection of inappropriate personnel or allows enlistment dodgers to avoid their military duty. This study conducted IRT-based person-fit analysis with the dichotomous military dataset in the Korean Military Personality Inventory. In order for that, 2PL model was applied for the data and person-fit index l_(z) was used to detect aberrant respondents. Based on l_(z) values of each respondent, potentially inaccurate respondents was identified. In diagnosing possible sources of aberrant response patterns, PRCs was assessed. This study with the military empirical data shows that person-fit analysis using l_(z) is applicable and practical method for detecting inaccurate response patterns in the personnel selection contexts based on the personality measurement.

Alzheimer’s disease (AD) progressively impairs cognitive and functional abilities. Research on pharmacological treatment of AD is shifting to earlier forms of the disease, including preclinical stages. However, assessment methods traditionally used in clinical research may be inappropriate for these populations. The Alzheimer Disease Assessment Scale-cognitive (ADAS-cog), a commonly used cognitive battery in AD research, is most sensitive in the moderate range of cognitive impairment. It focuses on immediate recall and recognition aspects of memory rather than retention and delayed recall. As clinical trials for dementia continue to focus on prodromal stages of AD, instruments need to be retooled to focus on cognitive abilities more prone to change in the earliest stages of the disease. One such domain is delayed recall, which is differentially sensitive to decline in the earliest stages of AD. A supplemental delayed recall subtest for the ADAS-cog is commonly implemented, but we do not know precisely where along the spectrum of cognitive dysfunction this subtest yields incremental information beyond what is gained from the standard ADAS-cog. An item response theory (IRT) approach can analyze this in a psychometrically rigorous way. This study’s aims are twofold: (1) to examine where along the AD spectrum the delayed recall subtest yields optimal information about cognitive dysfunction, and (2) to determine if adding delayed recall to the ADAS-cog can improve prediction of functional outcomes, specifically patients’ ability to complete basic and instrumental activities of daily living. Results revealed differential functioning of ADAS-cog subtests across the dimension of cognitive impairment. The delayed recall subtest provided optimal information and increased the ADAS-cog’s measurement precision in the relatively mild range of cognitive dysfunction. Moreover, the addition of delayed recall to the ADAS- cog, consistent with my hypothesis, increased covariation with instrumental but not basic activities of daily living. These findings provide evidence that the delayed recall subtest slightly improves the ADAS-cog’s ability to capture information about cognitive impairment in the mild range of severity and thereby improves prediction of instrumental functional deficits.

With societies aging all around the world, the global burden of degenerative diseases is expected to increase exponentially. From the perspective drug development, degenerative diseases represent an especially challenging class. Clinical trials, in this context often termed disease progression studies, are long, costly, require many individuals, and have low success rates. Therefore, it is crucial to use informative study designs and to analyze efficiently the obtained trial data. The development of novel approaches intended towards facilitating both the design and the analysis of disease progression studies was the aim of this thesis. This aim was pursued in three stages (i) the characterization and extension of pharmacometric software, (ii) the development of new methodology around statistical power, and (iii) the demonstration of application benefits. The optimal design software PopED was extended to simplify the application of optimal design methodology when planning a disease progression study. The performance of non-linear mixed effect estimation algorithms for trial data analysis was evaluated in terms of bias, precision, robustness with respect to initial estimates, and runtime. A novel statistic allowing for explicit optimization of study design for statistical power was derived and found to perform superior to existing methods. Monte-Carlo power studies were accelerated through application of parametric power estimation, delivering full power versus sample size curves from a few hundred Monte-Carlo samples. Optimal design and an explicit optimization for statistical power were applied to the planning of a study in Alzheimer's disease, resulting in a 30% smaller study size when targeting 80% power. The analysis of ADAS-cog score data was improved through application of item response theory, yielding a more exact description of the assessment score, an increased statistical power and an enhanced insight in the assessment properties. In conclusion, this thesis presents novel pharmacometric methods that can help addressing the challenges of designing and planning disease progression studies.

国立情報学研究所で電子化したコンテンツを使用している。

国立情報学研究所で電子化したコンテンツを使用している。