O objetivo deste trabalho é apresentar os conceitos relacionados a Teste Adaptativo Informatizado, ou abreviadamente TAI, para o modelo logístico unidimensional da Teoria de Resposta ao Item. Utilizamos a abordagem bayesiana para a estimação do parâmetro de interesse, chamado de traço latente ou habilidade. Apresentamos os principais algoritmos de seleção de itens em TAI e realizamos estudos de simulação para comparar o desempenho deles. Para comparação, usamos aproximações numéricas para o Erro Quadrático Médio e para o Vício e também calculamos o tempo médio para o TAI selecionar um item. Além disso, apresentamos como instalar e usar a implementação de TAI desenvolvida neste projeto chamada de TAI2U, que foi desenvolvido no VBA-Excel usando uma interface com o R / The main of this work is to introduce the subjects related to Computerized Adaptive Testing, or breafly CAT, for the unidimensional three-parameter logistic model of Item Response Theory. We use bayesian approach to estimate the parameter of interest. We present several item selection algorithms and we perform simulations comparing them. The comparisons are made in terms of the mean square error, bias of the trait estimates, the average time for item selection and the average length of test. Furthermore, we show how to install e use the CAT implementation of this work called built in MIcrosoft Excel - VBA using interface with the statistical package R

This thesis aims to introduce the use of computerized adaptive testing (CAT) - a novel and ever increasingly used method of a test administration - applied to the field of Kinanthropology. By adapting a test to an individual respondent's latent trait level, computerized adaptive testing offers numerous theoretical and methodological improvements that can significantly advance testing procedures. In the first part of the thesis, the theoretical and conceptual basis of CAT, as well as a brief overview of its historical origins and basic general principles are presented. The discussion necessarily includes the description of Item Response Theory (IRT) to some extent, since IRT is almost exclusively used as the mathematical model in today's CAT applications. Practical application of CAT is then evaluated using Monte-Carlo simulations involving adaptive administration of the Physical Self-Description Questionnaire (PSDQ) (Marsh, Richards, Johnson, Roche, & Tremayne, 1994) - an instrument widely used to assess physical self-concept in the field of sport and exercise psychology. The Monte Carlo simulation of the PSDQ adaptive administration utilized a real item pool (N = 70) calibrated with a Graded Response Model (GRM, see Samejima, 1969, 1997). The responses to test items were generated based on item...

論文名稱：線上題庫與適性測驗整合系統之發展研究 頁數：337 校所系別：國立政治大學教育學系 畢業時間及摘要別：九十學年度第二學期博士論文摘要 指導教授：林邦傑博士、余民寧博士 研究生：陳新豐 論文摘要內容 本研究係結合工具研發、理論驗證與效能評估的研究，旨在開發一個建構在全球資訊網的「線上題庫與適性測驗整合系統」，以提供教師在網際網路環境下的輔助教學評量系統，除了可動態新增題庫之外，並能針對學生提供童身訂做的適性測驗。因此，研究的兩個核心主軸為「線上題庫與適性測驗整合系統」之開發與「線上題庫建置」的理論驗證。 依循這兩個核心主軸，本研究的研究目的有三：（一）開發線上題庫與適性測驗整合系統。（二）驗證建置題庫的相關理論。（三）評估整合系統運作效能與使用者滿意程度。 為達成這三個研究目的，研究者採用結構分析中，Sehlly、Cashmen和Rosenblatt (2001)所提出的「系統開發生命週期」，將「線上題庫與適性測驗整合系統」開發過程分為「系統規劃」、「系統分析」、「系統設計」、「系統建置」、「系統運行與支援」等五個階段逐步開發。 研究樣本方面，第一次預試選用台南市崇明國中三年級學生115人，第二次預試選用台南市建興國中三年級學生191人，正式施測樣本則是台灣地區北、中、南、東、離島等共計九校2567位國中三年級學生為研究對象。此外，在需求調查報告部分，共調查十五位專家對開發系統的意見。 就研究工具來說，本研究主要研究工具為「線上題庫與適性測驗之整合系統」，另外，「功能需求調查問卷」、「硬體設備」、「軟體工具」、「系統評估量表」也是本研究的研究工具。 就資料處理來說，本研究運用ITEMAN、BILOG、MatLab和SPSS套裝軟體進行資料處理，所採用的統計方法包含古典測驗理論與試題反應理論等理論來分析，計有試題分析、IRT三參數估計、因素結構分析等。 研究結果部分，本研究得到如下結論：（一）線上題庫與通性測驗兩個系統可整合為一。（二）線上題庫與遍性測驗之整合系統具有多項功能。（三）結構化分析中之系統開發生命週期是開發整合系統的理想方法。（四）題庫等化轉換常數方法以Mean/Mean和Haebara等方法較佳。（五）線上測驗與紙筆測驗的試題訊息量相近，但難度偏高。（六）線上測驗連結效益良好。（七）本整合系統運作效能良好。（八）使用者對整合系統之功能感到滿意。根據研究結論，本研究針對工具研發、題庫建置、效能評估等提出具體建議。 關鍵字：試題反應理論、題庫、等化、電腦化適性測驗、系統開發生命週期 / A Reserach In The Development of An Integrated On-Line Item Bank and Computerized Adaptive Testing System Abstract This research is to develop an integrated internet system of on-line item bank and computerized adaptive testing (the "System"), which is comprised of the teaching tool development, theory verification, and efficiency evaluation. Except for the addition of new item bank dynamically, the System, an auxiliary teaching evaluation system for teachers, can also provide customarily made adaptive testing for students. Therefore, to develop an integrated on-line item bank and computerized adaptive testing system and to verify the theory of on-line item bank development constitute the two core spindles of this research. Following the aforementioned research spindles, the main purposes of this research are going to: (A). Develop an integrated on-line item bank and computerized adaptive testing system. (B). Verify the related theories concerning the development of on-line item bank. (C). Evaluate the operating efficiency of such System and the degree of users' satisfaction. The "systems development life cycle" (Sehily, Cashmen Rosenblatt, 2001), a structured analysis method, is adopted to conduct the research. The development process of an integrated on-line item bank and computerized adaptive testing system is divided into 5 separate and successive stages, starting from system planning, system analysis, system design, system development, to system operation and support. In terms of research sampling, the selected samples in the first preliminary testing are 115 ninth-grade students of Chiung-Ming High School in Tainan City, Taiwan. The selected samples in the second preliminary testing are 191 ninth-grade students of Cheng-Sing High School in Tainan City, Taiwan. The third and official sampling is 2,567 ninth-grade students who were selected from a total of 9 high schools ranging from Northern, Central, Southern, and Eastern Taiwan, and islands adjacent to Taiwan. Furthermore, in the demand side, an investigation has been conducted to consult with 15 teaching experts for their professional opinions in regard to such System development. As far as research tools are concerned, except for the main research tool - the integrated on-line item bank and computerized adaptive testing system, other research tools employed consist of functional demand questionnaires, hardware equipments, software tools, and scales for system evaluation. In the aspect of data processing, ITEMAN, BILOQ MatLab, and SPSSapplication softwares are used to perform the data processing. The statistical method,like classical true score theory and item response theory and etc., is applied to conduct the following analyses: item analysis, IRT three-parameter estimate, structured analysis of elements, and etc. The results of this research lead to the following conclusion: 1. The on-line item bank: system and the adaptive testing system, the two separate systems, can be integrated into one system. 2. The integrated on-line item bank and adaptive testing system can play multiple functions. 3. "Systems development life cycle" in the structured analysis is an ideal manner to develop an integrated system. 4. "Mean/Mean", "Haebara" and, etc. are the better methods to perform the item bank equating and constant conversion. 5. For users, the information volume provided by on-line testing and traditional written testing are quite similar, but the degree of difficulty of on-line testing is higher than that of traditional written testing. 6. The linking effect of on-line testing is fair. 7. The operating efficiency of the integrated system is fair. 8. Users are satisfied with the functions of the integrated system. Based on research conclusions drawn thereon, suggestions for tool development, item bank development, and efficiency evaluation are also provided. Keywords: item response theory, item bank, equating, computerized adaptive testing,systems development life cycle

Mémoire numérisé par la Division de la gestion de documents et des archives de l'Université de Montréal.

Testes Adaptativos Computadorizados (CAT) baseados na Teoria de Resposta ao Item permitem fazer testes mais precisos com um menor número de questões em relação à prova clássica feita a papel. Porém a construção de CAT envolve alguns questionamentos-chave, que quando feitos de forma adequada, podem melhorar ainda mais a precisão e a eficiência na estimativa das habilidades dos respondentes. Um dos principais questionamentos está na escolha da Regra de Seleção de Itens (ISR). O CAT clássico, faz uso, exclusivamente, de uma ISR. Entretanto, essas regras possuem vantagens, entre elas, a depender do nível de habilidade e do estágio em que o teste se encontra. Assim, o objetivo deste trabalho é reduzir o comprimento de provas dicotômicas - que consideram apenas se a resposta foi correta ou incorreta - que estão inseridas no ambiente de um CAT que faz uso, exclusivo, de apenas uma ISR sem perda significativa de precisão da estimativa das habilidades. Para tal, cria-se a abordagem denominada ALICAT que personaliza o processo de seleção de itens em CAT, considerando o uso de mais de uma ISR. Para aplicar essa abordagem é necessário primeiro analisar o desempenho de diferentes ISRs. Um estudo de caso na prova de Matemática e suas tecnologias do ENEM de 2012, indica que a regra de seleção de Kullback-Leibler com distribuição a posteriori (KLP) possui melhor desempenho na estimativa das habilidades dos respondentes em relação as regras: Informação de Fisher (F); Kullback-Leibler (KL); Informação Ponderada pela Máxima Verossimilhança (MLWI); e Informação ponderada a posteriori (MPWI). Resultados prévios da literatura mostram que CAT utilizando a regra KLP conseguiu reduzir a prova do estudo de caso em 46,6% em relação ao tamanho completo de 45 itens sem perda significativa na estimativa das habilidades. Neste trabalho, foi observado que as regras F e a MLWI tiveram melhor desempenho nos estágios inicias do CAT, para estimar respondentes com níveis de habilidades extremos negativos e positivos, respectivamente. Com a utilização dessas regras de seleção em conjunto, a abordagem ALICAT reduziu a mesma prova em 53,3% / Computerized Adaptive Testing (CAT) based on Item Response Theory allows more accurate assessments with fewer questions than the classic paper test. Nonetheless, the CAT building involves some key questions that, when done properly, can further improve the accuracy and efficiency in estimating examinees\' abilities. One of the main questions is in regard to choosing the Item Selection Rule (ISR). The classic CAT makes exclusive use of one ISR. However, these rules have differences depending on the examinees\' ability level and on the CAT stage. Thus, the objective of this work is to reduce the dichotomous - which considers only correct and incorrect answers - test size which is inserted on a classic CAT without significant loss of accuracy in the estimation of the examinee\'s ability level. For this purpose, we create the ALICAT approach that personalizes the item selection process in a CAT considering the use of more than one ISR. To apply this approach, we first analyze the performance of different ISRs. The case study in textit test of the ENEM 2012 shows that the Kullback-Leibler Information with a Posterior Distribution (KLP) has better performance in the examinees\' ability estimation when compared with: Fisher Information (F); Kullback-Leibler Information (KL); Maximum Likelihood Weighted Information(MLWI); and Maximum Posterior Weighted Information (MPWI) rules. Previous results in the literature show that CAT using KLP was able to reduce this test size by 46.6% from the full size of 45 items with no significant loss of accuracy in estimating the examinees\' ability level. In this work, we observe that the F and the MLWI rules performed better on early CAT stages to estimate examinees proficiency level with extreme negative and positive values, respectively. With this information, we were able to reduce the same test by 53.3% using an approach that uses the best rules together

Testes Adaptativos Computadorizados (CAT) baseados na Teoria de Resposta ao Item permitem fazer testes mais precisos com um menor número de questões em relação à prova clássica feita a papel. Porém a construção de CAT envolve alguns questionamentos-chave, que quando feitos de forma adequada, podem melhorar ainda mais a precisão e a eficiência na estimativa das habilidades dos respondentes. Um dos principais questionamentos está na escolha da Regra de Seleção de Itens (ISR). O CAT clássico, faz uso, exclusivamente, de uma ISR. Entretanto, essas regras possuem vantagens, entre elas, a depender do nível de habilidade e do estágio em que o teste se encontra. Assim, o objetivo deste trabalho é reduzir o comprimento de provas dicotômicas - que consideram apenas se a resposta foi correta ou incorreta - que estão inseridas no ambiente de um CAT que faz uso, exclusivo, de apenas uma ISR sem perda significativa de precisão da estimativa das habilidades. Para tal, cria-se a abordagem denominada ALICAT que personaliza o processo de seleção de itens em CAT, considerando o uso de mais de uma ISR. Para aplicar essa abordagem é necessário primeiro analisar o desempenho de diferentes ISRs. Um estudo de caso na prova de Matemática e suas tecnologias do ENEM de 2012, indica que a regra de seleção de Kullback-Leibler com distribuição a posteriori (KLP) possui melhor desempenho na estimativa das habilidades dos respondentes em relação as regras: Informação de Fisher (F); Kullback-Leibler (KL); Informação Ponderada pela Máxima Verossimilhança (MLWI); e Informação ponderada a posteriori (MPWI). Resultados prévios da literatura mostram que CAT utilizando a regra KLP conseguiu reduzir a prova do estudo de caso em 46,6% em relação ao tamanho completo de 45 itens sem perda significativa na estimativa das habilidades. Neste trabalho, foi observado que as regras F e a MLWI tiveram melhor desempenho nos estágios inicias do CAT, para estimar respondentes com níveis de habilidades extremos negativos e positivos, respectivamente. Com a utilização dessas regras de seleção em conjunto, a abordagem ALICAT reduziu a mesma prova em 53,3% / Computerized Adaptive Testing (CAT) based on Item Response Theory allows more accurate assessments with fewer questions than the classic paper test. Nonetheless, the CAT building involves some key questions that, when done properly, can further improve the accuracy and efficiency in estimating examinees\' abilities. One of the main questions is in regard to choosing the Item Selection Rule (ISR). The classic CAT makes exclusive use of one ISR. However, these rules have differences depending on the examinees\' ability level and on the CAT stage. Thus, the objective of this work is to reduce the dichotomous - which considers only correct and incorrect answers - test size which is inserted on a classic CAT without significant loss of accuracy in the estimation of the examinee\'s ability level. For this purpose, we create the ALICAT approach that personalizes the item selection process in a CAT considering the use of more than one ISR. To apply this approach, we first analyze the performance of different ISRs. The case study in textit test of the ENEM 2012 shows that the Kullback-Leibler Information with a Posterior Distribution (KLP) has better performance in the examinees\' ability estimation when compared with: Fisher Information (F); Kullback-Leibler Information (KL); Maximum Likelihood Weighted Information(MLWI); and Maximum Posterior Weighted Information (MPWI) rules. Previous results in the literature show that CAT using KLP was able to reduce this test size by 46.6% from the full size of 45 items with no significant loss of accuracy in estimating the examinees\' ability level. In this work, we observe that the F and the MLWI rules performed better on early CAT stages to estimate examinees proficiency level with extreme negative and positive values, respectively. With this information, we were able to reduce the same test by 53.3% using an approach that uses the best rules together

Mémoire numérisé par la Division de la gestion de documents et des archives de l'Université de Montréal

In this thesis, we describe and analyze computerized adaptive tests (CAT), the class of psychometrics tests in which items are selected based on the actual estimate of respondent's ability. We focus on the tests based on di- chotomic IRT (item response theory) models. We present critera for item selection, methods for ability estimation and termination criteria, as well as methods for exposure rate control and content balancing. In the analytical part, the effect of CAT settings on the average length of the test and on absoulute bias of ability estimates is investigated using linear regression mo- dels. We provide post hoc analysis of real data coming from real admission test with unknown true values of abilities, as well as simulation study based on the simulated answers of respondents with known true values of ability. In the last chapter of the thesis we investigate the possibilities of analysing adaptive tests in R software and of creating a real CAT. 1

本論文探討當自變數存在測量誤差時，羅吉斯迴歸模型的估計問題，並將此結果應用在電腦化適性測驗中的線上校準問題。在變動長度電腦化測驗的假設下，我們證明了估計量的強收斂性。試題反應理論被廣泛地使用在電腦化適性測驗上，其假設受試者在試題的表現情形與本身的能力，可以透過試題特徵曲線加以詮釋，羅吉斯迴歸模式是最常見的試題反應模式。藉由適性測驗的施行，考題的選取可以依據不同受試者，選擇最適合的題目。因此，相較於傳統測驗而言，在適性測驗中，題目的消耗量更為快速。在題庫的維護與管理上，新試題的補充與試題校準便為非常重要的工作。線上試題校準意指在線上測驗進行中，同時進行試題校準。因此，受試者的能力估計會存在測量誤差。從統計的觀點，線上校準面臨的困難，可以解釋為在非線性模型下，當自變數有測量誤差時的實驗設計問題。我們利用序貫設計降低測量誤差，得到更精確的估計，相較於傳統的試題校準，可以節省更多的時間及成本。我們利用處理測量誤差的技巧，進一步應用序貫設計的方法，處理在線上校準中，受試者能力存在測量誤差的問題。 / In this dissertation, we focus on the estimate in logistic regression models when the independent variables are subject to some measurement errors. The problem of this dissertation is motivated by online calibration in Computerized Adaptive Testing (CAT). We apply the measurement error model techniques and adaptive sequential design methodology to the online calibration problem of CAT. We prove that the estimates of item parameters are strongly consistent under the variable length CAT setup. In an adaptive testing scheme, examinees are presented with different sets of items chosen from a pre-calibrated item pool. Thus the speed of attrition in items will be very fast, and replenishing of item pool is essential for CAT. The online calibration scheme in CAT refers to estimating the item parameters of new, un-calibrated items by presenting them to examinees during the course of their ability testing together with previously calibrated items. Therefore, the estimated latent trait levels of examinees are used as the design points for estimating the parameter of the new items, and naturally these designs, the estimated latent trait levels, are subject to some estimating errors. Thus the problem of the online calibration under CAT setup can be formulated as a sequential estimation problem with measurement errors in the independent variables, which are also chosen sequentially. Item Response Theory (IRT) is the most commonly used psychometric model in CAT, and the logistic type models are the most popular models used in IRT based tests. That's why the nonlinear design problem and the nonlinear measurement error models are involved. Sequential design procedures proposed here can provide more accurate estimates of parameters, and are more efficient in terms of sample size (number of examinees used in calibration). In traditional calibration process in paper-and-pencil tests, we usually have to pay for the examinees joining the pre-test calibration process. In online calibration, there will be less cost, since we are able to assign new items to the examinees during the operational test. Therefore, the proposed procedures will be cost-effective as well as time-effective.