Multistage tests are a widely used and efficient type of test presentation that aims to provide accurate ability estimates while keeping the test relatively short. Multistage tests typically rely on the psychometric framework of item response theory. Violations of item response models and other assumptions underlying a multistage test, such as differential item functioning, can lead to inaccurate ability estimates and unfair measurements. There is a practical need for methods to detect problematic model violations to avoid these issues. This study compares and evaluates three methods for the detection of differential item functioning with regard to continuous person covariates in data from multistage tests: a linear logistic regression test and two adaptations of a recently proposed score-based DIF test. While all tests show a satisfactory Type I error rate, the score-based tests show greater power against three types of DIF effects.

No description available.

No description available.

No description available.

No description available.

No description available.

O uso de provas elaboradas na forma de itens, em processos de avaliação para classificação, é uma herança histórica dos séculos XVI e XVII, ainda em uso nos dias atuais tanto na educação formal quanto em processos seletivos, a exemplo dos exames vestibulares. Elaboradas para mensurar conhecimentos, traços latentes que não podem ser medidos diretamente, as provas costumam ser corrigidas considerando unicamente o escore obtido pelo sujeito avaliado, sem contemplar informações importantes relacionadas aos itens das mesmas. O presente trabalho teve como objetivos: (i) utilizar a modelagem baseada na teoria da resposta ao item unidimensional - TRI e multidimensional - TRIM para construir escalas do conhecimento para a prova da FUVEST e (ii) classificar os candidatos aos seis cursos de graduação oferecidos pela Escola Superior de Agricultura \"Luiz de Queiroz\", unidade da Universidade de São Paulo, com base na escala construída. A hipótese imbutida no corpo do trabalho admitiu que o uso da TRIM classifica de forma diferente os candidatos que os atuais métodos utilizados pela FUVEST. Foram utilizados os padrões de respostas dos 2326 candidatos submetidos à prova, para que uma análise unidimensional fosse realizada, sob o enfoque da TRI, gerando uma escala de proficiências . Quatro traços latentes foram diagnosticados no processo avaliativo, por meio da modelagem multidimensional da TRIM, gerando uma escala das quatro dimensões. Uma proposta para classificação dos candidatos é apresentada, baseada na média das proficiências individuais ponderada pelas cargas fatoriais diagnosticadas pela modelagem. Análise comparativa entre os critérios de classificação utilizados pela FUVEST e pela TRIM foram realizados, identificando discordância entre os mesmos. O trabalho apresenta propostas de interpretação pedagógica para as escalas unidimensional e multidimensional e indica a TRIM como o critério complementar para classificação dos candidatos, valorizando informações individuais dos itens e, portanto, utilizando uma avaliação classificatória mais abrangente. / The use of elaborate exams in the form of items, in evaluation procedures for classification, is a historical legacy of the 16th and 17th centuries, still in use today both in formal education and in selective cases such as entrance examinations. Designed to measure knowledge, latent trait that can not be measured directly, the exams are usually corrected considering only the score obtained by the subject, without including important information related to the items of it. This study aimed to: (i) use the modeling approach unidimensional and multidimensional item response theory (IRT and MIRT, respectively), to build knowledge scales of the entrance examination FUVEST/2012; (ii) classifing candidates for the 6 undergraduate courses offered by the \"Luiz de Queiroz\" College of Agriculture , unit of the University of São Paulo, based on the scale then. The hypothesis supposes that the use of MIRT ranked candidates differently than current methods used by FUVEST. We used the patterns of responses of 2326 candidates submitted to the test, so that a one-dimensional analysis was performed under the IRT approach, generating a range of proficiencies. Four latent traits were diagnosed in the evaluation process by means of multidimensional modeling MIRT, generating a scale of four dimensions. A proposal for classification of the candidates is presented, based on the weighted average of the individual proficiencies by the factor loadings diagnosed by modeling. Comparative analysis of the classification criteria used by FUVEST and MIRT were performed by identifying discrepancies between them. This work presents the proposals of the pedagogical interpretation for one-dimensional and multidimensional scales and indicates the MIRT as additional criteria for the candidates, to valorize individual information of the items and therefore using a more comprehensive classification review.

Mixed-format tests containing both multiple-choice (MC) items and constructed-response (CR) items are now widely used in many testing programs. Mixed-format tests often are considered to be superior to tests containing only MC items although the use of multiple item formats leads to measurement challenges in the context of equating conducted under the common-item nonequivalent groups design (CINEG). The purpose of this dissertation was to investigate how various test characteristics and examinee characteristics influence CINEG mixed-format test score equating results. Simulated data were used in this dissertation. Simulees' item responses were generated using items selected from one MC item pool and one CR item pool which were constructed based on the College Board Advanced Placement examinations from various subject areas. Five main factors were investigated in this dissertation, including item-type dimensionality, group ability difference, within group ability difference, length and composition of the common-item set, and format representativeness of the common-item set. In addition, the performance of two equating methods, the presmoothed frequency estimation method (PreSm_FE) and the presmoothed chained equipercentile equating method (PreSm_CE), was compared under various conditions. To evaluate equating results, both conditional statistics and overall summary statistics were considered: absolute bias, standard error of equating, and root mean squared error. The difference that matters (DTM) also was used as a criterion for evaluating whether adequate equating results were obtained. The main findings based on the simulation studies are as follows: (1) For most situations, item-type multidimensionality did not have substantial impact on random error, regardless of the common-item set. However, its influence on bias depended on the composition of common-item sets; (2) Both the group ability difference factor and the within group ability difference factor had no substantial influence on random error. When group ability differences were simulated, the common-item set with more items or more total score points had less equating error. When a within group ability difference existed, conditions in which there was a balance of different item formats in the common-item set displayed more accurate equating results than did unbalanced common-item sets. (3) The relative performance of common-item sets with various lengths and compositions was dependent on the levels of group ability difference, within group ability difference, and test dimensionality. (4) The common-item set containing only MC items performed similarly to the common-item set with both item formats when the test forms were unidimensional and no within group ability difference existed or when groups of examinees did not differ in proficiency. (5) The PreSm_FE method was more sensitive to group ability difference than the PreSm_CE method. When the within group ability difference was non-zero, the relative performance of the two methods depended on the length and composition of the common-item set. The two methods performed almost the same in terms of random error. The studies conducted in this dissertation suggest that when equating multidimensional mixed-format test forms in practice, if groups of examinees differ substantially in overall proficiency, inclusion of both item formats should be considered for the common-item set. When within group ability differences are likely to exist, balancing different item formats in the common-item set appears to be even more important than the use of a larger number of common items for obtaining accurate equating results. Because only simulation studies were conducted in this dissertation, caution should be exercised when generalizing the conclusions to practical situations.

依據「多層次混合試題反應理論」與「隨機試題混合模型」，本研究提出「混合試題與受試者模型」。本研究旨在評估此模型在不同樣本數、不同試題差異功能的試題數下，偵測試題差異功能的表現，以及其參數回復性情形。研究結果顯示，「混合試題與受試者模型」在樣本數大、試題差異功能試題數較多之情境下，具有正確的參數回復性，能正確判斷出試題是否存在試題差異功能，且具有良好的難度估計值，並能將樣本正確地分群，其也與「隨機試題混合模型」的估計表現頗為相近。建議未來可將「混合試題與受試者模型」應用於大型教育資料庫相關研究上，並加入其他變項後進一步探討。 / Drawing upon the framework of the multilevel mixture item response theory model and the random item mixture model, the study attempts to propose one model, called the mixture items and examinees model(MIE model). The purpose of this study was to assess the respective performances of the model on different sample-sizes and differential item functioning (DIF) items. Particularly, the study assessed the model performances in the detection of DIF items, and the accurate parameters recovery. The results of the study revealed that with large sample-sizes and more DIF items, the MIE model had the good parameters recovery, the accurate detection of the DIF items, the good estimate of the item difficulty, and the accurate classifications of the sub-samples. These model performances appeared similar to those of the random item mixture model. The findings suggest that future studies should apply the MIE model to the analyses on large-scale education databases, and should add more variables to the MIE model.

O uso de provas elaboradas na forma de itens, em processos de avaliação para classificação, é uma herança histórica dos séculos XVI e XVII, ainda em uso nos dias atuais tanto na educação formal quanto em processos seletivos, a exemplo dos exames vestibulares. Elaboradas para mensurar conhecimentos, traços latentes que não podem ser medidos diretamente, as provas costumam ser corrigidas considerando unicamente o escore obtido pelo sujeito avaliado, sem contemplar informações importantes relacionadas aos itens das mesmas. O presente trabalho teve como objetivos: (i) utilizar a modelagem baseada na teoria da resposta ao item unidimensional - TRI e multidimensional - TRIM para construir escalas do conhecimento para a prova da FUVEST e (ii) classificar os candidatos aos seis cursos de graduação oferecidos pela Escola Superior de Agricultura \"Luiz de Queiroz\", unidade da Universidade de São Paulo, com base na escala construída. A hipótese imbutida no corpo do trabalho admitiu que o uso da TRIM classifica de forma diferente os candidatos que os atuais métodos utilizados pela FUVEST. Foram utilizados os padrões de respostas dos 2326 candidatos submetidos à prova, para que uma análise unidimensional fosse realizada, sob o enfoque da TRI, gerando uma escala de proficiências . Quatro traços latentes foram diagnosticados no processo avaliativo, por meio da modelagem multidimensional da TRIM, gerando uma escala das quatro dimensões. Uma proposta para classificação dos candidatos é apresentada, baseada na média das proficiências individuais ponderada pelas cargas fatoriais diagnosticadas pela modelagem. Análise comparativa entre os critérios de classificação utilizados pela FUVEST e pela TRIM foram realizados, identificando discordância entre os mesmos. O trabalho apresenta propostas de interpretação pedagógica para as escalas unidimensional e multidimensional e indica a TRIM como o critério complementar para classificação dos candidatos, valorizando informações individuais dos itens e, portanto, utilizando uma avaliação classificatória mais abrangente. / The use of elaborate exams in the form of items, in evaluation procedures for classification, is a historical legacy of the 16th and 17th centuries, still in use today both in formal education and in selective cases such as entrance examinations. Designed to measure knowledge, latent trait that can not be measured directly, the exams are usually corrected considering only the score obtained by the subject, without including important information related to the items of it. This study aimed to: (i) use the modeling approach unidimensional and multidimensional item response theory (IRT and MIRT, respectively), to build knowledge scales of the entrance examination FUVEST/2012; (ii) classifing candidates for the 6 undergraduate courses offered by the \"Luiz de Queiroz\" College of Agriculture , unit of the University of São Paulo, based on the scale then. The hypothesis supposes that the use of MIRT ranked candidates differently than current methods used by FUVEST. We used the patterns of responses of 2326 candidates submitted to the test, so that a one-dimensional analysis was performed under the IRT approach, generating a range of proficiencies. Four latent traits were diagnosed in the evaluation process by means of multidimensional modeling MIRT, generating a scale of four dimensions. A proposal for classification of the candidates is presented, based on the weighted average of the individual proficiencies by the factor loadings diagnosed by modeling. Comparative analysis of the classification criteria used by FUVEST and MIRT were performed by identifying discrepancies between them. This work presents the proposals of the pedagogical interpretation for one-dimensional and multidimensional scales and indicates the MIRT as additional criteria for the candidates, to valorize individual information of the items and therefore using a more comprehensive classification review.