Mixed-format tests containing both multiple-choice (MC) items and constructed-response (CR) items are now widely used in many testing programs. Mixed-format tests often are considered to be superior to tests containing only MC items although the use of multiple item formats leads to measurement challenges in the context of equating conducted under the common-item nonequivalent groups design (CINEG). The purpose of this dissertation was to investigate how various test characteristics and examinee characteristics influence CINEG mixed-format test score equating results. Simulated data were used in this dissertation. Simulees' item responses were generated using items selected from one MC item pool and one CR item pool which were constructed based on the College Board Advanced Placement examinations from various subject areas. Five main factors were investigated in this dissertation, including item-type dimensionality, group ability difference, within group ability difference, length and composition of the common-item set, and format representativeness of the common-item set. In addition, the performance of two equating methods, the presmoothed frequency estimation method (PreSm_FE) and the presmoothed chained equipercentile equating method (PreSm_CE), was compared under various conditions. To evaluate equating results, both conditional statistics and overall summary statistics were considered: absolute bias, standard error of equating, and root mean squared error. The difference that matters (DTM) also was used as a criterion for evaluating whether adequate equating results were obtained. The main findings based on the simulation studies are as follows: (1) For most situations, item-type multidimensionality did not have substantial impact on random error, regardless of the common-item set. However, its influence on bias depended on the composition of common-item sets; (2) Both the group ability difference factor and the within group ability difference factor had no substantial influence on random error. When group ability differences were simulated, the common-item set with more items or more total score points had less equating error. When a within group ability difference existed, conditions in which there was a balance of different item formats in the common-item set displayed more accurate equating results than did unbalanced common-item sets. (3) The relative performance of common-item sets with various lengths and compositions was dependent on the levels of group ability difference, within group ability difference, and test dimensionality. (4) The common-item set containing only MC items performed similarly to the common-item set with both item formats when the test forms were unidimensional and no within group ability difference existed or when groups of examinees did not differ in proficiency. (5) The PreSm_FE method was more sensitive to group ability difference than the PreSm_CE method. When the within group ability difference was non-zero, the relative performance of the two methods depended on the length and composition of the common-item set. The two methods performed almost the same in terms of random error. The studies conducted in this dissertation suggest that when equating multidimensional mixed-format test forms in practice, if groups of examinees differ substantially in overall proficiency, inclusion of both item formats should be considered for the common-item set. When within group ability differences are likely to exist, balancing different item formats in the common-item set appears to be even more important than the use of a larger number of common items for obtaining accurate equating results. Because only simulation studies were conducted in this dissertation, caution should be exercised when generalizing the conclusions to practical situations.

Validity is a crucial part of all forms of measurement, and especially in instruments that are high-stakes to the test takers. The aim of this thesis was to examine theory and validity evidence for a recently revised large-scale instrument used for selection to higher education in Sweden, the Swedish Scholastic Assessment Test (SweSAT), as well as identify threats to its validity. Previous versions of the SweSAT have been intensely studied but when it was revised in 2011, further research was needed to strengthen the validity arguments for the test. The validity approach suggested in the most recent version of the Standards for education and psychological testing, in which the theoretical basis and five sources of validity evidence are the key aspects of validity, was adopted in this thesis. The four studies that are presented in this thesis focus on different aspects of the SweSAT, including theory, score reporting, item functioning and linking of test forms. These studies examine validity evidence from four of the five sources of validity: evidence based on test content, response processes, internal structure and consequences of testing. The results from the thesis as a whole show that there is validity evidence that supports some of the validity arguments for the intended interpretations and uses of SweSAT scores, and that there are potential threats to validity that require further attention. Empirical evidence supports the two-dimensional structure of the construct scholastic proficiency, but the construct requires a more thorough definition in order to better examine validity evidence based on content and consequences for test takers. Section scores provide more information about test takers' strengths and weaknesses than what is already provided by the total score and can therefore be reported, but subtest scores do not provide additional information and should not be reported. All four quantitative subtests, as well as the Swedish reading comprehension subtest, are essentially free of differential item functioning (DIF) but there is moderate DIF that could be bias in two of the four verbal subtests. Finally, the equating procedure, although it appears to be appropriate, needs to be examined further in order to determine whether it is the best practice available or not for the SweSAT. Some of the results in this thesis are specific to the SweSAT because only SweSAT data was used but the design of the studies and the methods that were applied serve as practical examples of validating a test and are therefore likely useful to different populations of people involved in test development, test use and psychometric research. Suggestions for further research include: (1) a study to create a more clear and elaborate definition of the construct, scholastic proficiency; (2) a large and empirically focused study of subscore value in the SweSAT using repeat test takers and applying Haberman’s method along with recently proposed effect size measures; (3) a cross-validation DIF-study using more recently administered test forms; (4) a study that examines the causes for the recurring score differences between women and men on the SweSAT; and (5) a study that re-examines the best practice for equating the current version of the SweSAT, using simulated data in addition to empirical data.

Guidelines to screen and select common items for vertical scaling have been adopted from equating. Differences between vertical scaling and equating suggest that these guidelines may not apply to vertical scaling in the same way that they apply to equating. For example, in equating the examinee groups are assumed to be randomly equivalent, but in vertical scaling the examinee groups are assumed to possess different levels of proficiency. Equating studies that examined the characteristics of the common-item set stress the importance of careful item selection, particularly when groups differ in ability level. Since in vertical scaling cross-level ability differences are expected, the common items' psychometric characteristics become even more important in order to obtain a correct interpretation of students' academic growth. This dissertation applied two screening criteria and two selection approaches to investigate how changes in the composition of the linking sets impacted the nature of students' growth when creating vertical scales for two elementary mathematics tests. The purpose was to observe how well these equating guidelines were applied in the context of vertical scaling. Two separate datasets were analyzed to observe the impact of manipulating the common items' content area and targeted curricular grade level. The same Rasch scaling method was applied for all variations of the linking set. Both the robust z procedure and a variant of the 0.3-logit difference procedure were used to screen unstable common items from the linking sets. (In vertical scaling, a directional item-difficulty difference must be computed for the 0.3-logit difference procedure.) Different combinations of stable common items were selected to make up the linking sets. The mean/mean method was used to compute the equating constant and linearly transform the students' test scores onto the base scale. A total of 36 vertical scales were created. The results indicated that, although the robust z procedure was a more conservative approach to flagging unstable items, the robust z and the 0.3-logit difference procedure produced similar interpretations of students' growth. The results also suggested that the choice of grade-level-targeted common items affected the estimates of students' grade-to-grade growth, whereas the results regarding the choice of content-area-specific common items were inconsistent. The findings from the Geometry and Measurement dataset indicated that the choice of content-area-specific common items had an impact on the interpretation of students' growth, while the findings from the Algebra and Data Analysis/Probability dataset indicated that the choice of content-area-specific common items did not appear to significantly affect students' growth. A discussion of the limitations of the study and possible future research is presented.

論文名稱：線上題庫與適性測驗整合系統之發展研究 頁數：337 校所系別：國立政治大學教育學系 畢業時間及摘要別：九十學年度第二學期博士論文摘要 指導教授：林邦傑博士、余民寧博士 研究生：陳新豐 論文摘要內容 本研究係結合工具研發、理論驗證與效能評估的研究，旨在開發一個建構在全球資訊網的「線上題庫與適性測驗整合系統」，以提供教師在網際網路環境下的輔助教學評量系統，除了可動態新增題庫之外，並能針對學生提供童身訂做的適性測驗。因此，研究的兩個核心主軸為「線上題庫與適性測驗整合系統」之開發與「線上題庫建置」的理論驗證。 依循這兩個核心主軸，本研究的研究目的有三：（一）開發線上題庫與適性測驗整合系統。（二）驗證建置題庫的相關理論。（三）評估整合系統運作效能與使用者滿意程度。 為達成這三個研究目的，研究者採用結構分析中，Sehlly、Cashmen和Rosenblatt (2001)所提出的「系統開發生命週期」，將「線上題庫與適性測驗整合系統」開發過程分為「系統規劃」、「系統分析」、「系統設計」、「系統建置」、「系統運行與支援」等五個階段逐步開發。 研究樣本方面，第一次預試選用台南市崇明國中三年級學生115人，第二次預試選用台南市建興國中三年級學生191人，正式施測樣本則是台灣地區北、中、南、東、離島等共計九校2567位國中三年級學生為研究對象。此外，在需求調查報告部分，共調查十五位專家對開發系統的意見。 就研究工具來說，本研究主要研究工具為「線上題庫與適性測驗之整合系統」，另外，「功能需求調查問卷」、「硬體設備」、「軟體工具」、「系統評估量表」也是本研究的研究工具。 就資料處理來說，本研究運用ITEMAN、BILOG、MatLab和SPSS套裝軟體進行資料處理，所採用的統計方法包含古典測驗理論與試題反應理論等理論來分析，計有試題分析、IRT三參數估計、因素結構分析等。 研究結果部分，本研究得到如下結論：（一）線上題庫與通性測驗兩個系統可整合為一。（二）線上題庫與遍性測驗之整合系統具有多項功能。（三）結構化分析中之系統開發生命週期是開發整合系統的理想方法。（四）題庫等化轉換常數方法以Mean/Mean和Haebara等方法較佳。（五）線上測驗與紙筆測驗的試題訊息量相近，但難度偏高。（六）線上測驗連結效益良好。（七）本整合系統運作效能良好。（八）使用者對整合系統之功能感到滿意。根據研究結論，本研究針對工具研發、題庫建置、效能評估等提出具體建議。 關鍵字：試題反應理論、題庫、等化、電腦化適性測驗、系統開發生命週期 / A Reserach In The Development of An Integrated On-Line Item Bank and Computerized Adaptive Testing System Abstract This research is to develop an integrated internet system of on-line item bank and computerized adaptive testing (the "System"), which is comprised of the teaching tool development, theory verification, and efficiency evaluation. Except for the addition of new item bank dynamically, the System, an auxiliary teaching evaluation system for teachers, can also provide customarily made adaptive testing for students. Therefore, to develop an integrated on-line item bank and computerized adaptive testing system and to verify the theory of on-line item bank development constitute the two core spindles of this research. Following the aforementioned research spindles, the main purposes of this research are going to: (A). Develop an integrated on-line item bank and computerized adaptive testing system. (B). Verify the related theories concerning the development of on-line item bank. (C). Evaluate the operating efficiency of such System and the degree of users' satisfaction. The "systems development life cycle" (Sehily, Cashmen Rosenblatt, 2001), a structured analysis method, is adopted to conduct the research. The development process of an integrated on-line item bank and computerized adaptive testing system is divided into 5 separate and successive stages, starting from system planning, system analysis, system design, system development, to system operation and support. In terms of research sampling, the selected samples in the first preliminary testing are 115 ninth-grade students of Chiung-Ming High School in Tainan City, Taiwan. The selected samples in the second preliminary testing are 191 ninth-grade students of Cheng-Sing High School in Tainan City, Taiwan. The third and official sampling is 2,567 ninth-grade students who were selected from a total of 9 high schools ranging from Northern, Central, Southern, and Eastern Taiwan, and islands adjacent to Taiwan. Furthermore, in the demand side, an investigation has been conducted to consult with 15 teaching experts for their professional opinions in regard to such System development. As far as research tools are concerned, except for the main research tool - the integrated on-line item bank and computerized adaptive testing system, other research tools employed consist of functional demand questionnaires, hardware equipments, software tools, and scales for system evaluation. In the aspect of data processing, ITEMAN, BILOQ MatLab, and SPSSapplication softwares are used to perform the data processing. The statistical method,like classical true score theory and item response theory and etc., is applied to conduct the following analyses: item analysis, IRT three-parameter estimate, structured analysis of elements, and etc. The results of this research lead to the following conclusion: 1. The on-line item bank: system and the adaptive testing system, the two separate systems, can be integrated into one system. 2. The integrated on-line item bank and adaptive testing system can play multiple functions. 3. "Systems development life cycle" in the structured analysis is an ideal manner to develop an integrated system. 4. "Mean/Mean", "Haebara" and, etc. are the better methods to perform the item bank equating and constant conversion. 5. For users, the information volume provided by on-line testing and traditional written testing are quite similar, but the degree of difficulty of on-line testing is higher than that of traditional written testing. 6. The linking effect of on-line testing is fair. 7. The operating efficiency of the integrated system is fair. 8. Users are satisfied with the functions of the integrated system. Based on research conclusions drawn thereon, suggestions for tool development, item bank development, and efficiency evaluation are also provided. Keywords: item response theory, item bank, equating, computerized adaptive testing,systems development life cycle

A falta de alternativas ao modelo normal uni/multivariado já é um problema superado pois atualmente é possível encontrar inúmeros trabalhos que introduzem e desenvolvem generalizações da distribuição normal com relação `a assimetria, curtose e/ou multimodalidade (Branco e Arellano-Valle (2004), Genton (2004), Arellano-Valle et al. (2006)). No contexto dos modelos unidimensionais da Teoria da Resposta ao Item (TRI), Bazán (2005) percebeu esta realidade e introduziu uma classe denominada PANA (Probito Assimétrico - Normal Assimétrica) a qual permite modelar possíveis comportamentos assimétricos de um modelo (uma probabilidade) de resposta ao item bem como a especificação de uma distribuição normal assimétrica para os traços latentes (unidimensionais) a qual é utilizada no processo de estimação. Motivado pela necessidade de melhor representar os fenômenos da área psicométrica (Heinen, 1996, p. 105) e da atual disponibilidade de distribuições elípticas assimétricas cujas propriedades são tão convenientes quanto aquelas devidas `a distribuição normal, a proposta do presente trabalho é apresentar uma extensão do modelo K-dimensional de 3 Parâmetros Probito (Kd3PP) com vetores de traços latentes normalmente distribuídos para o caso t-Assimétrico, gerando, assim, o que denominamos modelo Kd3PP-tA. Nossa proposta, portanto, pode ser considerada como uma extensão do trabalho desenvolvido por Bazán (2005) tanto no sentido de extender a distribuição unidimensional assimétrica dos traços latentes para o caso multidimensional quanto no que conscerne em considerar o achatamento (curtose) da distribuição. Nossa proposta também pode ser vista como uma extensão do trabalho de Béguin e Glas (2001) no sentido de desenvolver o método de estimação bayesiana dos modelos multidimensionais da TRI via DAGS (Dados Aumentados com Amostrador de Gibbs) para o caso em que os vetores de traços latentes comportam-se segundo uma distribuição multivariada t-Assimétrica. No desenvolvimento deste trabalho nos deparamos com uma das principais dificuldades encontradas no processo de estimação e inferência dos modelos multidimensionais da TRI que é a falta de identificabilidade e, com a intenção de ampliar e desmistificar nossos conhecimentos sobre um assunto ainda pouco explorado na literatura da TRI, apresentamos um estudo bibliográfico sobre este tema tanto sob o contexto da inferência clássica quanto bayesiana. Com o intuito de identificar situações particulares em que o uso de uma distribuição normal assimétrica para os traços latentes seja de maior relevância para a estimação e inferência dos parâmetros de item, bem como outros parâmetros relacionados à distribuição dos traços latentes, algumas análises sobre conjuntos de dados simulados são desenvolvidas. Como conclusão destas análises, podemos dizer que há uma melhora superficial quando a informação sobre uma possível assimetria na distribuição dos traços latentes não é ignorada. Além disso, os resultados favoreceram a seleção dos modelos que consideram distribuições assimétricas para os traços latentes, principalmente quando são considerados os modelos que possibilitam a estimação dos parâmetros de localização e escala da distribuição dos vetores de traços latentes. Duas principais contribuições que consideramos de ordem prática, são: a análise e a interpretação de testes através da estimação de modelos uni e multidimensionais da TRI que consideram tanto distribuições simétricas quanto assimétricas para os vetores de traços latentes e a disponibilização de uma função escrita em códigos R e C++ para a estimação dos modelos apresentados e desenvolvidos no presente trabalho. / The lack of alternatives to the univariate or multivariate normal model has been already solved because actually it has been possible to find several works that introduce and develop generalizations of the normal distribution in relation to the asymmetry, kurtosis and/or multimodality (Branco e Arellano-Valle (2004), Genton (2004), Arellano-Valle et al. (2006). In the context of unidimensional models of the Item Response Theory (IRT), Baz´an (2005) observed this fact and introduced a class called PANA (Probito Assimétrico - Normal Assimétrica) which allows to take account for asymmetry in the shape of an item response model (probability) and the specification of a skew normal distribution for unidimensional latent traits which is used in the estimation process. Motivated by the need to better represent the phenomenon of psychometric area (Heinen, 1996, p. 105) and the current availability of skew elliptical distributions whose properties are as convenient as those due to normal distribution, the proposal of this work is to provide an extension of multidimensional 3 Parameters Probit model (Kd3PP) where latent traits vectors are normally distributed for the case of Skew-t distribution (Sahu et al., 2003), generating therefore what we call Kd3PP-St model. Our proposal, therefore, can be regarded as an extension of the work of Bazán (2005) in two ways: the first is extending the unidimensional skew normal distribution of latent traits to the multidimensional case and second in the sense to consider the flattening (kurtosis) of this distribution. Our proposal can also be seen as an extension of the work of B´eguin e Glas (2001) in the sense that we develop the Bayesian estimation method of the 3 parameters multidimensional item response model by DAGS (Augmentated Data with Gibbs sampling) for the case where the latent trait vectors behave according to a Skew-t multivariate distribution. In the development of this work we come across one of the main difficulties encountered in the process of estimation and inference of multidimensional IRT models which is the lack of identifiabilitie and, with the intent to demystify and expand our knowledge on a subject still little explored in the literature of the IRT, we present a bibliographical study on this subject both in the context of classical and Bayesian inference. In order to identify particular situations where the use of a skew normal distribution is more relevant to the estimation and inference of item parameters as well as other parameters related to the distribution of latent traits, some analyses on simulated data sets are developed. As results of these analyses, we can say that there is a modest improvement when information about a possible asymmetry in the distribution of latent traits is not ignored. Moreover, the results favored the selection of models that consider asymmetric distributions for latent traits, especially when models that enable the estimation of parameters of location and scale from this distribution are considered. Two main contributions that we consider of pratical interest are: analysis and interpretations of tests using unidimensional and multidimensional IRT models that consider both simetric and skewed distributions for the vectors of latent traits and a function written in R and C++ language program that is made disponible for the estimation of models treated in this work.

A falta de alternativas ao modelo normal uni/multivariado já é um problema superado pois atualmente é possível encontrar inúmeros trabalhos que introduzem e desenvolvem generalizações da distribuição normal com relação `a assimetria, curtose e/ou multimodalidade (Branco e Arellano-Valle (2004), Genton (2004), Arellano-Valle et al. (2006)). No contexto dos modelos unidimensionais da Teoria da Resposta ao Item (TRI), Bazán (2005) percebeu esta realidade e introduziu uma classe denominada PANA (Probito Assimétrico - Normal Assimétrica) a qual permite modelar possíveis comportamentos assimétricos de um modelo (uma probabilidade) de resposta ao item bem como a especificação de uma distribuição normal assimétrica para os traços latentes (unidimensionais) a qual é utilizada no processo de estimação. Motivado pela necessidade de melhor representar os fenômenos da área psicométrica (Heinen, 1996, p. 105) e da atual disponibilidade de distribuições elípticas assimétricas cujas propriedades são tão convenientes quanto aquelas devidas `a distribuição normal, a proposta do presente trabalho é apresentar uma extensão do modelo K-dimensional de 3 Parâmetros Probito (Kd3PP) com vetores de traços latentes normalmente distribuídos para o caso t-Assimétrico, gerando, assim, o que denominamos modelo Kd3PP-tA. Nossa proposta, portanto, pode ser considerada como uma extensão do trabalho desenvolvido por Bazán (2005) tanto no sentido de extender a distribuição unidimensional assimétrica dos traços latentes para o caso multidimensional quanto no que conscerne em considerar o achatamento (curtose) da distribuição. Nossa proposta também pode ser vista como uma extensão do trabalho de Béguin e Glas (2001) no sentido de desenvolver o método de estimação bayesiana dos modelos multidimensionais da TRI via DAGS (Dados Aumentados com Amostrador de Gibbs) para o caso em que os vetores de traços latentes comportam-se segundo uma distribuição multivariada t-Assimétrica. No desenvolvimento deste trabalho nos deparamos com uma das principais dificuldades encontradas no processo de estimação e inferência dos modelos multidimensionais da TRI que é a falta de identificabilidade e, com a intenção de ampliar e desmistificar nossos conhecimentos sobre um assunto ainda pouco explorado na literatura da TRI, apresentamos um estudo bibliográfico sobre este tema tanto sob o contexto da inferência clássica quanto bayesiana. Com o intuito de identificar situações particulares em que o uso de uma distribuição normal assimétrica para os traços latentes seja de maior relevância para a estimação e inferência dos parâmetros de item, bem como outros parâmetros relacionados à distribuição dos traços latentes, algumas análises sobre conjuntos de dados simulados são desenvolvidas. Como conclusão destas análises, podemos dizer que há uma melhora superficial quando a informação sobre uma possível assimetria na distribuição dos traços latentes não é ignorada. Além disso, os resultados favoreceram a seleção dos modelos que consideram distribuições assimétricas para os traços latentes, principalmente quando são considerados os modelos que possibilitam a estimação dos parâmetros de localização e escala da distribuição dos vetores de traços latentes. Duas principais contribuições que consideramos de ordem prática, são: a análise e a interpretação de testes através da estimação de modelos uni e multidimensionais da TRI que consideram tanto distribuições simétricas quanto assimétricas para os vetores de traços latentes e a disponibilização de uma função escrita em códigos R e C++ para a estimação dos modelos apresentados e desenvolvidos no presente trabalho. / The lack of alternatives to the univariate or multivariate normal model has been already solved because actually it has been possible to find several works that introduce and develop generalizations of the normal distribution in relation to the asymmetry, kurtosis and/or multimodality (Branco e Arellano-Valle (2004), Genton (2004), Arellano-Valle et al. (2006). In the context of unidimensional models of the Item Response Theory (IRT), Baz´an (2005) observed this fact and introduced a class called PANA (Probito Assimétrico - Normal Assimétrica) which allows to take account for asymmetry in the shape of an item response model (probability) and the specification of a skew normal distribution for unidimensional latent traits which is used in the estimation process. Motivated by the need to better represent the phenomenon of psychometric area (Heinen, 1996, p. 105) and the current availability of skew elliptical distributions whose properties are as convenient as those due to normal distribution, the proposal of this work is to provide an extension of multidimensional 3 Parameters Probit model (Kd3PP) where latent traits vectors are normally distributed for the case of Skew-t distribution (Sahu et al., 2003), generating therefore what we call Kd3PP-St model. Our proposal, therefore, can be regarded as an extension of the work of Bazán (2005) in two ways: the first is extending the unidimensional skew normal distribution of latent traits to the multidimensional case and second in the sense to consider the flattening (kurtosis) of this distribution. Our proposal can also be seen as an extension of the work of B´eguin e Glas (2001) in the sense that we develop the Bayesian estimation method of the 3 parameters multidimensional item response model by DAGS (Augmentated Data with Gibbs sampling) for the case where the latent trait vectors behave according to a Skew-t multivariate distribution. In the development of this work we come across one of the main difficulties encountered in the process of estimation and inference of multidimensional IRT models which is the lack of identifiabilitie and, with the intent to demystify and expand our knowledge on a subject still little explored in the literature of the IRT, we present a bibliographical study on this subject both in the context of classical and Bayesian inference. In order to identify particular situations where the use of a skew normal distribution is more relevant to the estimation and inference of item parameters as well as other parameters related to the distribution of latent traits, some analyses on simulated data sets are developed. As results of these analyses, we can say that there is a modest improvement when information about a possible asymmetry in the distribution of latent traits is not ignored. Moreover, the results favored the selection of models that consider asymmetric distributions for latent traits, especially when models that enable the estimation of parameters of location and scale from this distribution are considered. Two main contributions that we consider of pratical interest are: analysis and interpretations of tests using unidimensional and multidimensional IRT models that consider both simetric and skewed distributions for the vectors of latent traits and a function written in R and C++ language program that is made disponible for the estimation of models treated in this work.