This study explores the relationship between students’ missing responses on a large-scale assessment and their cognitive skill profiles and characteristics. Data from the 48 multiple-choice items on the 2006 Ontario Secondary School Literacy Test (OSSLT), a high school graduation requirement, were analyzed using the item response theory (IRT) three-parameter logistic model and the Reduced Reparameterized Unified Model, a Cognitive Diagnostic Model. Missing responses were analyzed by item and by student. Item-level analyses examined the relationships among item difficulty, item order, literacy skills targeted by the item, the cognitive skills required by the item, the percent of students not answering the item, and other features of the item. Student-level analyses examined the relationships among students’ missing responses, overall performance, cognitive skill mastery profiles, and characteristics such as gender and home language. Most students answered most items: no item was answered by fewer than 98.8% of the students and 95.5% of students had 0 missing responses, 3.2% had 1 missing response, and only 1.3% had more than 1 missing responses). However, whether students responded to items was related to the student’s characteristics, including gender, whether the student had an individual education plan and language spoken at home, and to the item’s characteristics such as item difficulty and the cognitive skills required to answer the item. Unlike in previous studies of large-scale assessments, the missing response rates were not higher for multiple-choice items appearing later in the timed sections. Instead, the first two items in some sections had higher missing response rates. Examination of the student-level missing response rates, however, showed that when students had high numbers of missing responses, these often represented failures to complete a section of the test. Also, if nonresponse was concentrated in items that required particular skills, the accuracy of the estimates for those skills was lower than for other skills. The results of this study have implications for test designers who seek to improve provincial large-scale assessments, and for teachers who seek to help students improve their cognitive skills and develop test taking strategies.

The purpose of the present study is to investigate the predictive validity of the BUEPE through the use of the three-parameter IRT model&amp / #8217 / s ability estimates. The study made use of the BUEPE September 2000 data which included the responses of 699 students. The predictive validity was established by using the departmental English courses (DEC) passing grades of a total number of 371 students. As for the prerequisite analysis the best fitted model of IRT was determined by first, checking the assumptions of IRT / second, by analyzing the invariance of ability parameters and item parameters and thirdly, by interpreting the chi-square statistics. After the prerequisite analyses, the best fitted model&amp / #8217 / s estimates were correlated with DEC passing grades to investigate the predictive power of BUEPE on DEC passing grades. The findings indicated that the minimal guessing assumption of the one- and two-parameter models was not met. In addition, the chi-square statistics indicated a better fit to the three-parameter model. Therefore, it was concluded that the best fitted model was the three-parameter model. The findings of the predictive validity analyses revealed that the best predictors for DEC passing grades were the three-parameter model ability estimates. The second best predictor was the ability estimates obtained from sixty high information items. In the third place BUEPE total scores and the total scores obtained from sixty high information items followed with nearly the same correlation coefficients. Among the three sub-tests, the reading sub-test was found to be the best predictor of DEC passing grades.

Repeated measures designs are widely used in educational and psychological research to compare the changes exhibited in response to a treatment. Traditionally, measures of change are found by calculating difference scores (subtracting the observed initial score from the final score) for each person. However, problems such as the reliability paradox and the meaning of change scores arise from using simple difference scores to study change. A new item response theory model will be presented that estimates latent change scores instead of difference scores, addresses some of the limitations of using difference scores, and provides a direct comparison of the mean latent changes exhibited by different groups (e.g. females versus males). A simulation-based test was conducted to ascertain the viability of the model and results indicate that parameters of the newly developed model can be estimated accurately. Two sets of analyses were performed on the Early Childhood Longitudinal Study-Kindergarten cohort (ECLS-K) to examine differential growth in math ability between 1) male and female students and 2) Caucasian and African American students from kindergarten through fifth grade.

Submitted by Leonardo Cavalcante (leo.ocavalcante@gmail.com) on 2018-06-11T11:51:35Z No. of bitstreams: 1 Arquivototal.pdf: 3445476 bytes, checksum: 555b1ed01b554f9292c07b7471e3de2f (MD5) / Made available in DSpace on 2018-06-11T11:51:35Z (GMT). No. of bitstreams: 1 Arquivototal.pdf: 3445476 bytes, checksum: 555b1ed01b554f9292c07b7471e3de2f (MD5) Previous issue date: 2017-08-28 / Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES / An adequate selection of suppliers can make a difference in the future of organizations, lowering operating costs, improving product quality and enabling quick responses to customer demands. In the current business context of supply chains, the appropriate choice of suppliers is essential for good management and maintenance and improvement of competitive advantages. Therefore, the objective of this dissertation is to develop a method to evaluate suppliers performance in the context of supply chains, using item response theory (IRT). To achieve this goal, 60 supplier performance aspects, covering seven dimensions (cost, time, quality, flexibility, innovation, reputation/ industry experience and sustainability), were considered in the formulation of a questionnaire with 67 items to evaluate supplier performance. The questionnaire made possible the evaluation of 243 supply links of companies across different sectors and 14 Brazilian States. The evaluation results of the 243 supply links were analyzed using IRT’s Graded Response Model (GRM). GRM establishes a difficulty parameter for each category of the presented items, defining levels of performance in an interpretable scale that indicates the aspects served by each evaluated relationship and informs what aspects the company evaluated still needs to evolve in. Depending on what the client company expects and prioritizes in the performance of its suppliers, it can associate scale levels (and the presence and/or absence of certain aspects) with decisions about what to do in the relationship with the supplier company (deepen relationship, request changes in behavior or end relationship, for example). / Uma seleção adequada de fornecedores pode fazer diferença no futuro das organizações, diminuindo custos operacionais, melhorando a qualidade dos produtos e possibilitando respostas rápidas às demandas dos clientes. No atual contexto empresarial de atuação em cadeias, a escolha apropriada de fornecedores é imprescindível para uma boa gestão e para manutenção e melhoria de vantagens competitivas. Assim, o objetivo desta dissertação é desenvolver um método para avaliar desempenho de fornecedores no contexto de cadeias de suprimentos, utilizando a Teoria de Resposta ao Item (TRI). Para atingir tal objetivo, foram levantados 60 aspectos de desempenho de fornecedores, que contemplam sete dimensões: custo, tempo, qualidade, flexibilidade, inovação, reputação/experiência no setor e sustentabilidade. Com base nestes aspectos, foi elaborado um questionário com 67 itens para avaliação do desempenho de fornecedores, que permitiu avaliar 243 elos de fornecimento de empresas respondentes de diversos setores e 14 Estados brasileiros. Estes resultados foram avaliados utilizando o Modelo de Resposta Gradual da TRI que permite estabelecer um parâmetro de dificuldade para cada categoria dos itens apresentados, definindo níveis de desempenho em uma escala interpretável que indica os aspectos atendidos por cada relacionamento avaliado e em quais aspectos ainda é preciso evoluir. A depender do que a empresa cliente espera e prioriza no desempenho de seus fornecedores, esta pode associar os níveis da escala (e a presença e/ou ausência de certos aspectos) a decisões sobre o que fazer no relacionamento com a empresa fornecedora (estreitar laços, solicitar mudanças no comportamento ou cortar relações, por exemplo).

abstract: The purpose of this study was to investigate the effect of complex structure on dimensionality assessment in compensatory and noncompensatory multidimensional item response models (MIRT) of assessment data using dimensionality assessment procedures based on conditional covariances (i.e., DETECT) and a factor analytical approach (i.e., NOHARM). The DETECT-based methods typically outperformed the NOHARM-based methods in both two- (2D) and three-dimensional (3D) compensatory MIRT conditions. The DETECT-based methods yielded high proportion correct, especially when correlations were .60 or smaller, data exhibited 30% or less complexity, and larger sample size. As the complexity increased and the sample size decreased, the performance typically diminished. As the complexity increased, it also became more difficult to label the resulting sets of items from DETECT in terms of the dimensions. DETECT was consistent in classification of simple items, but less consistent in classification of complex items. Out of the three NOHARM-based methods, χ2G/D and ALR generally outperformed RMSR. χ2G/D was more accurate when N = 500 and complexity levels were 30% or lower. As the number of items increased, ALR performance improved at correlation of .60 and 30% or less complexity. When the data followed a noncompensatory MIRT model, the NOHARM-based methods, specifically χ2G/D and ALR, were the most accurate of all five methods. The marginal proportions for labeling sets of items as dimension-like were typically low, suggesting that the methods generally failed to label two (three) sets of items as dimension-like in 2D (3D) noncompensatory situations. The DETECT-based methods were more consistent in classifying simple items across complexity levels, sample sizes, and correlations. However, as complexity and correlation levels increased the classification rates for all methods decreased. In most conditions, the DETECT-based methods classified complex items equally or more consistent than the NOHARM-based methods. In particular, as complexity, the number of items, and the true dimensionality increased, the DETECT-based methods were notably more consistent than any NOHARM-based method. Despite DETECT's consistency, when data follow a noncompensatory MIRT model, the NOHARM-based method should be preferred over the DETECT-based methods to assess dimensionality due to poor performance of DETECT in identifying the true dimensionality. / Dissertation/Thesis / Ph.D. Educational Psychology 2011

abstract: The current study employs item difficulty modeling procedures to evaluate the feasibility of potential generative item features for nonword repetition. Specifically, the extent to which the manipulated item features affect the theoretical mechanisms that underlie nonword repetition accuracy was estimated. Generative item features were based on the phonological loop component of Baddelely's model of working memory which addresses phonological short-term memory (Baddeley, 2000, 2003; Baddeley & Hitch, 1974). Using researcher developed software, nonwords were generated to adhere to the phonological constraints of Spanish. Thirty-six nonwords were chosen based on the set item features identified by the proposed cognitive processing model. Using a planned missing data design, two-hundred fifteen Spanish-English bilingual children were administered 24 of the 36 generated nonwords. Multiple regression and explanatory item response modeling techniques (e.g., linear logistic test model, LLTM; Fischer, 1973) were used to estimate the impact of item features on item difficulty. The final LLTM included three item radicals and two item incidentals. Results indicated that the LLTM predicted item difficulties were highly correlated with the Rasch item difficulties (r = .89) and accounted for a substantial amount of the variance in item difficulty (R2 = .79). The findings are discussed in terms of validity evidence in support of using the phonological loop component of Baddeley's model (2000) as a cognitive processing model for nonword repetition items and the feasibility of using the proposed radical structure as an item blueprint for the future generation of nonword repetition items. / Dissertation/Thesis / M.A. Educational Psychology 2011

abstract: ABSTRACT This study investigated the possibility of item parameter drift (IPD) in a calculus placement examination administered to approximately 3,000 students at a large university in the United States. A single form of the exam was administered continuously for a period of two years, possibly allowing later examinees to have prior knowledge of specific items on the exam. An analysis of IPD was conducted to explore evidence of possible item exposure. Two assumptions concerning items exposure were made: 1) item recall and item exposure are positively correlated, and 2) item exposure results in the items becoming easier over time. Special consideration was given to two contextual item characteristics: 1) item location within the test, specifically items at the beginning and end of the exam, and 2) the use of an associated diagram. The hypotheses stated that these item characteristics would make the items easier to recall and, therefore, more likely to be exposed, resulting in item drift. BILOG-MG 3 was used to calibrate the items and assess for IPD. No evidence was found to support the hypotheses that the items located at the beginning of the test or with an associated diagram drifted as a result of item exposure. Three items among the last ten on the exam drifted significantly and became easier, consistent with item exposure. However, in this study, the possible effects of item exposure could not be separated from the effects of other potential factors such as speededness, curriculum changes, better test preparation on the part of subsequent examinees, or guessing. / Dissertation/Thesis / M.A. Educational Psychology 2012

O objetivo central da presente tese de doutorado foi a construção e a análise psicométrica de um instrumento de autorrelato para avaliar traços de psicopatia na população geral. Para fundamentar a proposta do instrumento de avalição, foram defendidos três argumentos. O primeiro deles é que embora o problema da medida dos atributos psicológicos ainda se encontre aberto à investigação teórica e empírica, instrumentos psicométricos são recursos adequados para obter informação psicológica. O segundo é que o autorrelato é um método legítimo de avaliação da psicopatia em situações em que não há incentivos explícitos para distorções nas respostas. O terceiro, por sua vez, é que um modelo dimensional é mais coerente com a literatura teórica e empírica da psicopatia do que uma perspectiva taxônica ou categórica. A construção e a análise psicométrica do instrumento contaram com uma coleta piloto com estudantes de graduação (N = 224), uma coleta online com universitários e indivíduos da população geral (N = 1238) e uma segunda coleta com estudantes de graduação (N = 12). Uma combinação de análises fatoriais exploratórias e confirmatórias ordinais e dois modelos de Teoria de Resposta ao Item evidenciaram excelentes propriedades psicométricas para um conjunto de 60 itens. Esses itens se mostraram distribuídos em três escalas (Insociabilidade, Descontrole e Audácia) e 10 subescalas (Tendências Antissociais, Dependência de Recompensas, Baixo Autocontrole, Dominância Social, Déficits Emocionais, Narcisismo Patológico, Exploração Interpessoal, Cinismo, Despreocupação e Intrepidez). As escalas e subescalas construídas se correlacionaram com as variáveis ruminação, Behavioral Inhibition System, Behavioral Approach System, afetos positivos, afetos negativos e com dois instrumentos de avaliação da psicopatia, a Levenson Self-Report Psychopathy scale e a Psychopathy Checklist-Revised. Além disso, construiu-se um sistema quantitativo-qualitativo de interpretação dos escores produzidos pelo instrumento. O trabalho oferece um recurso metodológico gratuito, com excelentes propriedades psicométricas, para suprir às necessidades de pesquisadores e profissionais de avaliação de traços de psicopatia que trabalham com indivíduos da população geral. / The aim of the present doctoral thesis was to develop and analyze the psychometric properties of a self-report instrument of psychopathic traits devised for use with nonforensic and noncriminal populations. In order to provide a theoretical framework for proposing the instrument, I developed three main arguments. First, although the psychological measurement problem remains as an open issue to theoretical an empirical scrutiny, psychometric instruments comprise adequate means for obtaining psychological information. Second, empirical studies support self-report method as a useful, reliable psychological resource for assessing psychopathic traits when individuals have no incentives for faking responses. Third, evidences favor a dimensional model of psychopathy as a more plausible theoretical perspective than a taxonic, categorical model. Three distinct samples were employed for constructing and evaluating the self-report instrument: a pilot sample of undergraduate students (N = 224), a large sample of university students and other individuals from the general population (1,238) and a sample of undergraduate students (N = 12). I analyzed data combining categorical exploratory and confirmatory factor analyses and two Item Response Theory models, which suggested retaining 60 items with excellent psychometric properties. These items comprised three scales (Meanness, Boldness and Disinhibition) and 10 subscales (Antisocial Tendencies, Reward Dependence, Low Self-Control, Social Dominance, Emotional Deficits, Pathological Narcissism, Interpersonal Exploitation, Cynism, Unconcern and Fearlessness). Scales and subscales correlated with rumination, behavioral inhibition, behavioral approach, positive and negative affects and scores on other instruments for assessing psychopathy, namely, Levenson Self-Report Psychopathy scale and Psychopathy Checklist-Revised. Furthermore, I devised a quantitative-qualitative system to help interpreting raw scores on the instrument. The present work offers a free self-report method with excellent psychometric properties to assist Brazilian researchers and professionals in dealing with the assessment of psychopathy among individuals from the general population.

Este trabalho descreve as etapas de transformação de um exame de proficiência em inglês acadêmico, aplicado via lápis-e-papel, com itens de múltipla escolha administrados segundo o método de Medida de Probabilidade Admissível (Shuford Jr et al., 1966), utilizado no programa de pós-graduação do Instituto de Ciências Matemáticas e de Computação da Universidade de São Paulo (ICMC-USP), em um teste adaptativo informatizado (TAI-PI) baseado em um modelo da Teoria de Resposta ao Item (TRI). Apesar do programa aceitar diversos exames que atestam a proficiência em inglês para indivíduos não-nativos de abrangência e reconhecimento internacionais, como o TOEFL (Test of English as a Foreign Language), IELTS (International English Language Testing System) e CPE (Certicate of Proficiency in English), por exemplo, a sua obrigatoriedade é incoerente em universidades públicas do Brasil devido ao custo que varia de 200 a 300 dólares por exame. O software TAI-PI (Teste Adaptativo Informatizado para Proficiência em Inglês), que foi desenvolvido em Java e SQLite, será utilizado para a avaliação da proficiência em inglês dos alunos do programa a partir do segundo semestre de 2013, de forma gratuita. A metodologia estatística implementada foi definida considerando a história e objetivos do exame e adotou o modelo de resposta gradual unidimensional de Samejima (Samejima, 1969), o critério de Kullback-Leibler para seleção de itens, o método de estimação da esperança a posteriori para os traços latentes (Baker, 2001) e a abordagem Shadow test (Van der Linden e Pashley, 2010) para imposição de restrições (de conteúdo e tamanho da prova) na composição do teste de cada indivíduo. Uma descrição da estrutura do exame, dos métodos empregados, dos resultados das aplicações do TAI-PI a alunos de pós-graduação do ICMC e estudos de classificação dos alunos em aprovados e reprovados, são apresentados neste trabalho, evidenciando a boa qualidade da nova proposta adotada e aprimoramento do exame com a utilização dos métodos de TRI e TAI. / This work describes the steps for converting a linear paper-and-pencil English proficiency test for academic purposes, composed with multiple choice items that are administered following the admissible probability measurement procedure (Shuford Jr et al., 1966), adopted by the graduate program of Institute of Mathematical Sciences and Computing of University of São Paulo (ICMCUSP), Brazil, to a computerized adaptive test (TAI-PI) based on an item response theory model (IRT). Despite the Institute recognizes reliable international English-language exams for academic purposes and non-native speakers, as TOEFL (Test of English as a Foreign Language), IELTS (International English Language Testing System) and CPE (Cambridge English: Proficiency), for instance, it is inconsistent that public universities in Brazil require them as certification because of the cost of approximately US$ 200.00 to US$ 300.00 per exam. The software TAI-PI (computerized adaptive test for English proficiency) was implemented in Java language, used SQLite as database engine, and it shall be offered free of charge for English proficiency assessment of the graduate students from October 2013. The statistical methodology employed for TAI-PI construction was defined considering the history and the aims of the evaluation and adopted the Samejima\'s graded response model (Samejima, 1969), the Kullback-Leibler information criterion for item selection, the expected a posteriori Bayesian estimation for latent trait (Baker, 2001) and shadow test approach (Van der Linden e Pashley, 2010) for test constraints (content and size of the test, for example). A description of the test design, the employed statistical methods, study results of a real application of TAI-PI to graduate students are presented in this work and the validation studies of the new methodology for pass/fail classification, highlighting the good quality of the new evaluation system and examination of improvement with the use of the methods of IRT and CAT.

Submitted by Viviane Lima da Cunha (viviane@biblioteca.ufpb.br) on 2017-07-06T11:44:55Z No. of bitstreams: 1 arquivototal.pdf: 3395465 bytes, checksum: 2cec5af03e01034c60cd5bbe8cd30f27 (MD5) / Made available in DSpace on 2017-07-06T11:44:55Z (GMT). No. of bitstreams: 1 arquivototal.pdf: 3395465 bytes, checksum: 2cec5af03e01034c60cd5bbe8cd30f27 (MD5) Previous issue date: 2016-02-19 / Personality disorders are defined as a persistent pattern of internal experience and behaviors that deviate markedly from the individual culture's expectations in a widespread and inflexibly way. Begins in adolescence and early adulthood, is stable over time and leads to injury. The types of personality disorders (PD) according to DSM-V are: Paranoid, Schizoid, Schizotypal, Antisocial, Borderline, Histrionic, Narcissistic, Avoidant, Dependent and Obsessive-Compulsive. The aim of this master thesis is to elaborate, validate and verify the psychometric properties of the items of an instrument to assess personality disorders. To do this we applied the Assessment Instrument of Personality Disorders (AIPD), developed and validated in this study in a sample of 470 people. The investigation verified the one-dimensionality of each item set of disorders through factor analysis for two forms of the instrument (a polytomic and a dichotomous). The factor analysis conducted by the principal-axis method and VARIMAX rotation resulted in a final matrix composed of nine factors that are organized as representative of the following personality disorders: Paranoid, Schizoid, Schizotypal, Antisocial, Borderline, Narcissistic, Avoidant, Dependent and Obsessive-Compulsive. For the Histrionic, there was no satisfactory psychometric items and in line with the literature to interpret this PD. The reliability of the measure in each disorder showed levels above 0.60, an acceptable value considering the low number of items in some factors. Regarding the instrument in dichotomized form, this did not show interpretable factors. The structure pointed to only one factor that explained 20% of the variance (considered low) and 0.955 trust index. The IRT analysis was conducted with polytomous format because it presented the best results. Three factors stood out as the amount of information offered on the latent trait, they are: Narcissistic PD, Dependent PD and Avoidant DP. Factors that have the best parameters of difficulty and discrimination were the Narcissistic and Avoidant, all between median and high. The teta values (ability) estimated from the subjects from the TRI are presented in this investigation as a possible tool for decision making about the presence or absence of a specific PD. The overall results are considered satisfactory and the validated instrument can be used as a tool for research in the evaluation of personality disorders. / Os transtornos da personalidade são definidos como um padrão persistente da experiência interior e de comportamentos que se desviam acentuadamente das expectativas da cultura do indivíduo de forma generalizada e inflexível. Tem início na adolescência ou início da idade adulta, é estável ao longo do tempo e leva a prejuízo. Os tipos de transtornos da personalidade (TP) de acordo com DSM-V são: Paranoide, Esquizoide, Esquizotípica, Antissocial, Borderline, Histriônica, Narcisista, Evitativa, Dependente, Obsessivo-compulsiva. O objetivo da presente dissertação é elaborar, validar e verificar as propriedades psicométricas dos itens de um instrumento para avaliar transtornos da personalidade. Para isso, foram aplicados o Instrumento de Avaliação dos Transtornos da Personalidade (IATP), elaborado e validado neste estudo, em uma amostra de 470 pessoas. A investigação verificou a unidimensionalidade de cada conjunto de itens nos transtornos através de análises fatoriais para duas formas do instrumento (politômica e dicotômica). A Análise fatorial realizada pelo método dos eixos principais fatoriais (PAF) e rotação VARIMAX, resultou em uma matriz final composta por nove fatores que se organizaram como representativos dos seguintes transtornos da personalidade: Paranoide, Esquizoide, Esquizotípica, Antissocial, Borderline, Narcisista, Evitativa, Dependente e Obssessivo-compulsiva. Para o Histriônico, não houve itens psicometricamente satisfatórios e em consonância com a literatura, capazes de interpretar esse TP. A confiabilidade da medida em cada transtorno apresentou índices superiores a 0,60, um valor aceitável pelo baixo número de itens de alguns fatores. Quanto ao instrumento no formato dicotomizado, este não apresentou fatores interpretáveis. A estrutura apontou para apenas um fator com variância explicada de 20%, considerada baixa, e índice de fidedignidade de 0,955. A análise da TRI foi feita com o formato politômico, pois este apresentou os melhores resultados. Três fatores se destacaram quanto a quantidade de informação oferecida sobre o traço latente, são eles: TP Narcisista, TP Dependente e TP Evitativa. Os fatores que obtiveram os melhores parâmetros de dificuldade e discriminação foram o Narcisista e Evitativa, todos entre medianos e altos. O valor do teta (habilidade) dos sujeitos estimados a partir da TRI são apresentados nessa investigação como uma possível ferramenta para tomada de decisão quanto a presença ou ausência de um TP específico. Os resultados de modo geral são considerados satisfatórios e o instrumento validado pode ser utilizado como ferramenta para fins de pesquisa na avaliação dos transtornos da personalidade.