As use of the public-domain International Personality Item Pool (IPIP) scales has grown significantly over the past decade (Goldberg, Johnson, Eber, Hogan, Ashton, Cloninger, & Gough, 2006) research on the psychometric properties of the items and scales have become increasingly important. This research study examines the IPIP scale constructed to measure the Five Factor Model (FFM) domain of Neuroticism (as measured by the NEO-PI-R) for occurrences of differential functioning at both the item and test level by gender and three age ranges using the DFIT framework (Raju, van der Linden, & Fleer, 1993) This study found six items that displayed differential item functioning by gender and three items that displayed differential item functioning by age. No differential functioning at the test level was found. Items demonstrating DIF and implications for potential scale revision are discussed. / Ph. D.

The need to publish and disseminate data continues to grow. Administrators of large-scale educational assessment should provide examinee microdata in addition to publishing assessment reports. Disclosure avoidance methods are applied to the data to protect examinee privacy before doing so, while attempting to preserve as many item statistical properties as possible. When important properties like differential item functioning are lost due to these disclosure avoidance methods, the microdata can give off misleading messages of effectiveness in measuring the test construct. In this research study, I investigated the preservation of differential item functioning in a large-scale assessment after disclosure avoidance methods have been applied to the data. After applying data swapping to protect the data, I attempted to empirically model and explain the likelihood of preserving various levels of differential item functioning as a function of several factors including the data swapping rate, the reference-to-focal group ratio, the type of item scoring, and the level of DIF prior to data swapping. / Ph. D.

A Monte Carlo simulation study investigated the effect of scoring format, item parameterization, threshold configuration, and prior ability distribution on the accuracy of ability estimation given various IRT models. Item response data on 30 items from 1,000 examinees was simulated using known item parameters and ability estimates. The item response data sets were submitted to seven dichotomous or polytomous IRT models with different item parameterization to estimate examinee ability. The accuracy of the ability estimation for a given IRT model was assessed by the recovery rate and the root mean square errors. The results indicated that polytomous models produced more accurate ability estimates than the dichotomous models, under all combinations of research conditions, as indicated by higher recovery rates and lower root mean square errors. For the item parameterization models, the one-parameter model out-performed the two-parameter and three-parameter models under all research conditions. Among the polytomous models, the partial credit model had more accurate ability estimation than the other three polytomous models. The nominal categories model performed better than the general partial credit model and the multiple-choice model with the multiple-choice model the least accurate. The results further indicated that certain prior ability distributions had an effect on the accuracy of ability estimation; however, no clear order of accuracy among the four prior distribution groups was identified due to an interaction between prior ability distribution and threshold configuration. The recovery rate was lower when the test items had categories with unequal threshold distances, were close at one end of the ability/difficulty continuum, and were administered to a sample of examinees whose population ability distribution was skewed to the same end of the ability continuum.

The purpose of this dissertation is to examine how the method used to a score situational judgment test (SJT) affects the validity of the SJT both in the presence of other predictors and as a single predictor of task performance. To this end, I compared the summed score approach of scoring SJTs with item response theory and multivariate items response theory. Using two samples and three sets of analyses, I found that the method used to score SJTs influences the validity of the test and that IRT and MIRT show promise for increasing SJT validity. However, no individual scoring method produced the highest amount of validity across all sets of analyses. In line with previous research, SJTs added incremental validity in the presence of GMA and personality and, again, the method used to score the SJT affected the incremental validity. A relative weights analysis was performed for each scoring method across all the sets of analyses showing that, depending on the scoring method, SJT score may account for more criterion variance than either GMA or personality. However, it is likely that the samples were influenced by range restriction present in the incumbent samples.

Este trabalho visa auxiliar professores e profissionais da Educação Básica para a elaboração de itens nas avaliações em larga escala. A princípio, é realizado um breve histórico sobre a situação da Educação Básica no país, em meados dos anos 80. Em seguida, são reveladas algumas das medidas planejadas pelos órgãos educacionais na busca por melhorias no cenário educacional brasileiro como, por exemplo, a reestruturação das avaliações em larga escala existentes na década de 90 e a criação de novos exames. O capítulo seguinte apresenta os documentos que são consultados durante este processo de construção dessas avaliações (matrizes curriculares e de referência) com ênfase no Exame Nacional do Ensino Médio (ENEM), pelo fato de ser a avaliação em larga escala de maior abrangência, em nível federal, desde 2009. Os capítulos seguintes revelam a importância do item nas avaliações em larga escala e apresentam alguns modelos elaborados, com base na Matriz de Referência do ENEM. / This work aims to help teachers and professionals of Basic Education to elaborate items in the large scale evaluations. At the outset, a brief history of the situation of Basic Education in the country in the mid-1980s is made. Then, some of the measures planned by the educational agencies are revealed in the search for improvements in the Brazilian educational scenario, such as the restructuring of the large-scale assessments in the 1990s and the creation of new examinations. The following chapter presents the documents that are consulted during this process of construction of these evaluations (curricular and reference matrices) with emphasis on the Exame Nacional do Ensino Médio (ENEM), because it is the large scale federal, since 2009. The following chapters reveal the importance of the item in the large-scale evaluations and present some elaborate models, based on the ENEM Reference Matrix.

O desempenho de estudantes em matemática na prova do ENEM é a discussão central deste trabalho. Com as mudanças no ENEM ocorridas no ano de 2009, a TRI - Teoria de Resposta ao Item - passou a ser utilizada para elaboração e correção da prova, permitindo, assim, mais confiabilidade nos resultados das provas, e, claro, uma resposta mais interessante ao estudantes, para além do aspecto quantitativos de acerto e de erro em questões. O presente trabalho tem o propósito analisar o desempenho de estudantes na prova do ENEM 2012, na cidade de Luis Eduardo Magalhães, (BA) comparando com os resultados desta mesma prova dos participantes de todo o estado da Bahia. Realizou-se uma análise de 10 questões e seus resultados de acertos e erros, sendo possível uma análise, mesmo sem os parâmetros TRI, sobre o desempenho dos estudantes na respectiva prova. Os resultados da pesquisa são o encaminhamento de ações na educação básica voltados ao compromisso de elevar o desempenho dos estudantes de ensino médio que realizam a prova do ENEM, através de programas de estudos e outros meios, na forma de produtos educacionais. A presente pesquisa aponta também o avanço de sua análise com a obtenção dos parâmetros TRI - já que não são de domínio publico e sua obtenção não é de fácil localização e acesso no Ministério da Educação, bem como, programas específicos de softwares, não são de fácil acesso - que contribuiriam muito para melhorar o esclarecimento desta avaliação em todo o Brasil e, por conseguinte, permitir a elevação dos índices de desempenho dos estudantes, sobretudo, em matemática em Luis Eduardo Magalhães(BA). / The performance of students in mathematics in the ENEM test is the central discussion of this work. With the changes in the ENEM in 2009, TRI - Item Response Theory - began to be used for the preparation and correction of the test, allowing, therefore, more reliability in the results of the tests, and, of course, a more interesting response To students, beyond the quantitative aspect of correctness and error in questions. The purpose of this paper is to analyze student performance in the ENEM 2012 test in the city of Luis Eduardo Magalhães (BA), comparing with the results of this same test of the participants from the entire state of Bahia. An analysis of 10 questions and their results of correct answers and errors was made, and it was possible to analyze, even without the TRI parameters, on the students’ performance in the respective test. The results of the research are the referral of actions in basic education aimed at raising the performance of high school students who take the ENEM test, through study programs and other means, in the form of educational products. The present research also indicates the progress of its analysis with the obtaining of the TRI parameters - since they are not of public domain and their obtaining is not of easy location and access in the Ministry of Education, as well as, specific programs of software, are not of Which would greatly contribute to improving the clarification of this evaluation throughout Brazil and, consequently, to allow students to increase their performance, especially in mathematics in Luis Eduardo Magalhães (BA).

Em testes nos quais uma quantidade considerável de indivíduos não dispõe de tempo suciente para responder todos os itens temos o que é chamado de efeito de Speededness. O uso do modelo unidimensional da Teoria da Resposta ao Item (TRI) em testes com speededness pode nos levar a uma série de interpretações errôneas uma vez que nesse modelo é suposto que os respondentes possuem tempo suciente para responder todos os itens. Nesse trabalho, desenvolvemos uma análise Bayesiana do modelo tri-dimensional da TRI proposto por Wollack e Cohen (2005) considerando uma estrutura de dependência entre as distribuições a priori dos traços latentes a qual modelamos com o uso de cópulas. Apresentamos um processo de estimação para o modelo proposto e fazemos um estudo de simulação comparativo com a análise realizada por Bazan et al. (2010) na qual foi utilizada distribuições a priori independentes para os traços latentes. Finalmente, fazemos uma análise de sensibilidade do modelo em estudo e apresentamos uma aplicação levando em conta um conjunto de dados reais proveniente de um subteste do EGRA, chamado de Nonsense Words, realizado no Peru em 2007. Nesse subteste os alunos são avaliados por via oral efetuando a leitura, sequencialmente, de 50 palavras sem sentidos em 60 segundos o que caracteriza a presença do efeito speededness. / In tests where a reasonable amount of individuals does not have enough time to answer all items we observe what is called eect of Speededness. The use of a unidimensional model from Item Response Theory (IRT) in tests with speededness can lead us to erroneous interpretations, since this model assumes that the respondents have enough time to answer all items. In this work, we propose a Bayesian analysis of the three-dimensional item response models (IRT) proposed by Wollack and Cohen et al (2005) considering a dependency structure between the prior distributions of the latent traits which is modeled using Copulas. We propose and develop a MCMC algorithm for the estimation of the model. A simulation study comparing with the analysis in Bazan et al (2010), wherein an independent prior distribution assumption was presented. Finally, we apply our model in a set of real data from EGRA, called Nonsense Words, held in Peru in 2007, where students are evaluated for their performance in reading.

Research has shown that the increased use of computer-based testing has brought about new challenges. With the ease of online test administration, a large number of items are necessary to maintain the item bank and minimise the exposure rate. However, the traditional item development process is time-consuming and costly. Thus, alternative ways of creating items are necessary to improve the item development process. Automatic Item Generation (AIG) is an effective method in generating items rapidly and efficiently. AIG uses algorithms to create questions for testing purposes. However, many of these generators are in the closed form, available only to the selected few. There is a lack of open source, publicly available generators that researchers can utilise to study AIG in greater depth and to generate items for their research. Furthermore, research has indicated that AIG is far from being understood, and more research into its methodology and the psychometric properties of the items created by the generators are needed for it to be used effectively. The studies conducted in this thesis have achieved the following: 1) Five open source item generators were created, and the items were evaluated and validated. 2) Empirical evidence showed that using a weak theory approach to develop item generators was just as credible as using a strong theory approach, even though they are theoretically distinct. 3) The psychometric properties of the generated items were estimated using various IRT models to assess the impact of the template features used to create the items. 4) Joint responses and response time modelling was employed to provide new insights into cognitive processes that go beyond those obtained by typical IRT models. This thesis suggests that AIG provides a tangible solution for improving the item development process for content generation and reducing the procedural cost of generating a large number of items, with the possibility of a unified approach towards test administration (i.e. adaptive item generation). Nonetheless, this thesis focused on rule-based algorithms. The application of other forms of item generation methods and the potential for measuring the intelligence of artificial general intelligence (AGI) is discussed in the final chapter, proposing that the use of AIG techniques create new opportunities as well as challenges for researchers that will redefine the assessment of intelligence.

Em testes nos quais uma quantidade considerável de indivíduos não dispõe de tempo suciente para responder todos os itens temos o que é chamado de efeito de Speededness. O uso do modelo unidimensional da Teoria da Resposta ao Item (TRI) em testes com speededness pode nos levar a uma série de interpretações errôneas uma vez que nesse modelo é suposto que os respondentes possuem tempo suciente para responder todos os itens. Nesse trabalho, desenvolvemos uma análise Bayesiana do modelo tri-dimensional da TRI proposto por Wollack e Cohen (2005) considerando uma estrutura de dependência entre as distribuições a priori dos traços latentes a qual modelamos com o uso de cópulas. Apresentamos um processo de estimação para o modelo proposto e fazemos um estudo de simulação comparativo com a análise realizada por Bazan et al. (2010) na qual foi utilizada distribuições a priori independentes para os traços latentes. Finalmente, fazemos uma análise de sensibilidade do modelo em estudo e apresentamos uma aplicação levando em conta um conjunto de dados reais proveniente de um subteste do EGRA, chamado de Nonsense Words, realizado no Peru em 2007. Nesse subteste os alunos são avaliados por via oral efetuando a leitura, sequencialmente, de 50 palavras sem sentidos em 60 segundos o que caracteriza a presença do efeito speededness. / In tests where a reasonable amount of individuals does not have enough time to answer all items we observe what is called eect of Speededness. The use of a unidimensional model from Item Response Theory (IRT) in tests with speededness can lead us to erroneous interpretations, since this model assumes that the respondents have enough time to answer all items. In this work, we propose a Bayesian analysis of the three-dimensional item response models (IRT) proposed by Wollack and Cohen et al (2005) considering a dependency structure between the prior distributions of the latent traits which is modeled using Copulas. We propose and develop a MCMC algorithm for the estimation of the model. A simulation study comparing with the analysis in Bazan et al (2010), wherein an independent prior distribution assumption was presented. Finally, we apply our model in a set of real data from EGRA, called Nonsense Words, held in Peru in 2007, where students are evaluated for their performance in reading.

Este trabalho visa auxiliar professores e profissionais da Educação Básica para a elaboração de itens nas avaliações em larga escala. A princípio, é realizado um breve histórico sobre a situação da Educação Básica no país, em meados dos anos 80. Em seguida, são reveladas algumas das medidas planejadas pelos órgãos educacionais na busca por melhorias no cenário educacional brasileiro como, por exemplo, a reestruturação das avaliações em larga escala existentes na década de 90 e a criação de novos exames. O capítulo seguinte apresenta os documentos que são consultados durante este processo de construção dessas avaliações (matrizes curriculares e de referência) com ênfase no Exame Nacional do Ensino Médio (ENEM), pelo fato de ser a avaliação em larga escala de maior abrangência, em nível federal, desde 2009. Os capítulos seguintes revelam a importância do item nas avaliações em larga escala e apresentam alguns modelos elaborados, com base na Matriz de Referência do ENEM. / This work aims to help teachers and professionals of Basic Education to elaborate items in the large scale evaluations. At the outset, a brief history of the situation of Basic Education in the country in the mid-1980s is made. Then, some of the measures planned by the educational agencies are revealed in the search for improvements in the Brazilian educational scenario, such as the restructuring of the large-scale assessments in the 1990s and the creation of new examinations. The following chapter presents the documents that are consulted during this process of construction of these evaluations (curricular and reference matrices) with emphasis on the Exame Nacional do Ensino Médio (ENEM), because it is the large scale federal, since 2009. The following chapters reveal the importance of the item in the large-scale evaluations and present some elaborate models, based on the ENEM Reference Matrix.