This diploma thesis is focused on proposal and implementation of automatized user testing in StartupJobs.cz project. It describes it from theoretical preparation through implemenation to evaluation of this project after first year. It is divided into two main parts. First one is about description of company which is implementing the testing. It describes reasons and expectations from testing implementation. It also creates a theoretical background by describing Classical Test Theory (CTT), Item Reposnse Theory (IRT) and more others. End of this part is consists of review of relevant literary sources. In second, practical part is the thesis focused on using these theories in praxis. How to use them for company's purposes. It is followed by description of system implementation and main part of the thesis is focused on evaluation of user testing implementation using data collected through the first year. Goal of this thesis is to validate hypothesis which are related to testing implementation (relationship between having tests and success rate of candidates). Main contribution of this work is complex overview on this field from theories to practical implementation which created during first year a lot of unique data presented in this paper.

Orientador: Caio Lucidius Naberezny Azevedo / Dissertação (mestrado) - Universidade Estadual de Campinas, Instituto de Matemática Estatística e Computação Científica / Made available in DSpace on 2018-08-24T12:51:18Z (GMT). No. of bitstreams: 1 Ferreira_EduardoVargas_M.pdf: 8131052 bytes, checksum: f344cd1f11e8d818f3aac90f48396cbc (MD5) Previous issue date: 2014 / Resumo: No presente trabalho propõem-se novos modelos da Teoria de Resposta ao Item para respostas politômicas nominais e ordinais (graduais), via dados aumentados, para grupos múltiplos. Para a modelagem das distribuições dos traços latentes de cada grupo, considera-se normais assimétricas centradas. Tal abordagem, além de acomodar a característica de assimetria aos dados, ajuda a garantir a identificabilidade dos modelos estudados, a qual é tratada tanto sob a ótica frequentista quanto bayesiana. Com relação aos métodos de estimação, desenvolveu-se procedimentos bayesianos através de algoritmos de Monte Carlo via cadeias de Markov (MCMC), utilizando o algoritmo de Gibbs (DAGS), com a verossimilhança aumentada (dados aumentados) e Metropolis-Hastings, considerando a verossimilhança original. As implementações computacionais foram escritas em linguagem C++, integradas ao ambiente computacional, gráfico e estatístico R, viabilizando rotinas gratuitas, de código aberto e alta velocidade no processamento, essenciais à difusão de tais metodologias. Para a seleção de modelos, utilizou-se o critério de informação deviance (DIC), os valores esperados do critério de informação de Akaike (EAIC) e o critério de informação bayesiano (EBIC). Em relação à verificação da qualidade do ajuste de modelos, explorou-se a checagem preditiva a posteriori, que fornece meios concretos de se avaliar a qualidade do instrumento de medida (prova, questionário etc), qualidade do ajuste do modelo de um modo global, além de indícios de violações de suposições específicas. Estudos de simulação, considerando diversas situações de interesse prático, indicam que os modelos e métodos de estimação produzem resultados bastante satisfatórios, com superioridade dos modelos assimétricos com relação ao simétrico (o qual assume simetria das distribuições das variáveis latentes). A análise de um conjunto de dados reais, referente à primeira fase do vestibular da UNICAMP de 2013, ilustra o potencial da tríade: modelagem, métodos de estimação e ferramentas de diagnósticos, desenvolvida neste trabalho / Abstract: In this work, we propose new Item Response Theory models for nominal and ordinal (gradual) polytomous responses through augmented data schemes considering multiple groups. For the distribution of the latent traits of each group, we consider a skew-normal distribution under the centered parametrization. This approach will allow for accommodating a possible skewness of the latent trait distribution, but is also helpful to ensure the identifiability of the models, which is studied under frequentist and Bayesian paradigms. Concerning estimation methods, we developed Bayesian methods through Markov chain Monte Carlo (MCMC) algorithms by using the Gibbs algorithm (DAGS), with augmented likelihood (augmented data) and Metropolis-Hastings algorithms, considering the original likelihood. The computational environment was written in the C++ language and integrated with the R program (a statistical computational and graphical environment), allowing for free, open source and high-speed routines which, in turn, are essential to the dissemination of the developed methodologies. In terms of model selection, we considered the deviance information criterion (DIC), the expected Akaike information criterion (EAIC) and expected Bayesian information criterion (EBIC). Regarding model-fit assessment tools, we explore the posterior predictive model- checking which allows for assessing the quality of measurement, instruments (tests, questionnaires, and others), the model fit in a global sense, besides providing directions toward violations of specific assumptions. Simulation studies, considering different situations of practical interest, indicate that the models and estimation methods produced reasonable results, with outperformance of skew models when compared to symmetric ones (which assumes symmetry of the latent trait distribution). Analysis of a data set which corresponds to the first phase of the 2013 written examination of UNICAMP (State University of Campinas), illustrates the potential of the following triad: modelling; estimation methods; and diagnostic tools developed in this work. / Mestrado / Estatistica / Mestre em Estatística

Testes Adaptativos Computadorizados (TAC) são aqueles que selecionam gradativamente as questões (itens) a serem apresentadas ao indivíduo de acordo com o seu nível de conhecimento (traco latente). Um TAC pode se basear em um modelo da Teoria da Resposta ao Item (TRI) para a estimação do traco latente e escolha do item a ser apresentado em cada passo do teste. Este trabalho apresenta modelos da TRI utilizados em TAC encontrados na literatura e descreve alguns métodos de calibração de itens para a formação e manutenção do banco de questões do teste sob o modelo de Samejima (1969), estimação do traço latente, seleção de itens com restrições utilizando a abordagem Shadow test e critérios de parada normalmente utilizados. Foram realizadas simulações com um banco grande (500 itens) e com um banco pequeno (21 itens) e avaliada a qualidade das estimativas dos traços latente (através do cálculos dos vícios e erros quadráticos médios) de TACs com diferentes números de itens. Foi aplicado o modelo de Samejima às respostas de estudantes do Exame ao proficiência em inglês (EPI) do ICMC - USP, que é aplicado semestralmente no formato lápis e papel, para a formação de um banco de itens e posterior construção de um TAC. Também foi aplicado o modelo às respostas de pacientes clínicos do Hospital das Clínicas da Medicina da USP, cedido pelo doutor Yuang-Pang Wang, ao Inventário de Depressão de Beck (BDI) para os mesmos propósitos. Comparações com a atual metodologia para avaliação da proficiência em língua inglesa do EPI (Medida de Probabilidade Admissível, MPA) e para o diagnóstico de depressão do BDI (critério sugerido por Kendall et al., 1987) foram realizadas demonstrando as vantagens e maior riqueza dos resultados obtidos com a TRI e com os TACs implementados. Adcionalmente foi desenvolvido um programa Same-CAT que armazena bancos de itens e possibilita a criação e aplicação de TACs com restrições, através da abordagem Shadow test / Computerized Adaptive Tests (CAT) are those that select questions (items) gradually to be presented to an individual according to their proficiency (latent trait level). A CAT can be based on an Item Response Theory (IRT) model for estimation of the latent trait and selection of the next item to be presented in each step of the test. This paper presents IRT models used in CATs found in literature and describes some methods of item calibration for creation and maintenance of a test items bank under the Samejima\'s model (Samejima; 1969), estimation of latent trait, item selection with constraints using the Shadow test approach and usuals stopping criteria. Simulations were conducted with a large bank (500 items) and a small bank (21 items) and the quality of the estimatives of latent traits were evaluated (through calculations of mean squared errors and bias) TACs with different item numbers. Samejima\'s model were applied for the responses of students to the English Proficiency Exam (EPE) of ICMC - USP, a test applied twice a year in paper and pencil format, to create an item bank and subsequent construction of a CAT. The model was also applied to the responses of clinical patients from the Hospital das Clnicas - USP, given by Dr. Yuang-Pang Wang, to the Beck Depression Inventory (BDI) for the same purposes. Comparisons using the current methodology to evaluate the English Language Proficiency of EPE (Measure of Probability Allowable, MPA) and the BDI (criterion suggested by Kendall et al., 1987) were performed, and the CATs provided better and richer results. Furthermore a program, Same-CAT, that stores item banks and allows CAT\'s applications with constraints was created

O objetivo deste trabalho é apresentar os conceitos relacionados a Teste Adaptativo Informatizado, ou abreviadamente TAI, para o modelo logístico unidimensional da Teoria de Resposta ao Item. Utilizamos a abordagem bayesiana para a estimação do parâmetro de interesse, chamado de traço latente ou habilidade. Apresentamos os principais algoritmos de seleção de itens em TAI e realizamos estudos de simulação para comparar o desempenho deles. Para comparação, usamos aproximações numéricas para o Erro Quadrático Médio e para o Vício e também calculamos o tempo médio para o TAI selecionar um item. Além disso, apresentamos como instalar e usar a implementação de TAI desenvolvida neste projeto chamada de TAI2U, que foi desenvolvido no VBA-Excel usando uma interface com o R / The main of this work is to introduce the subjects related to Computerized Adaptive Testing, or breafly CAT, for the unidimensional three-parameter logistic model of Item Response Theory. We use bayesian approach to estimate the parameter of interest. We present several item selection algorithms and we perform simulations comparing them. The comparisons are made in terms of the mean square error, bias of the trait estimates, the average time for item selection and the average length of test. Furthermore, we show how to install e use the CAT implementation of this work called built in MIcrosoft Excel - VBA using interface with the statistical package R

Suite aux progrès techniques et méthodologiques dans le secteur de la modélisation, l’apport de ces approches est désormais reconnu par l’ensemble des acteurs de la recherche clinique et pourrait avoir un rôle clé dans la recherche sur les maladies progressives. Parmi celles-ci les études pharmacométriques (PMX) sont rarement utilisées pour répondre aux hypothèses posées dans le cadre d’études dites de confirmation. Parmi les raisons évoquées, les analyses PMX traditionnelles ignorent l'incertitude associée à la structure du modèle lors de la génération d'inférence statistique. Or, ignorer l’étape de sélection du modèle peut aboutir à des intervalles de confiance trop optimistes et à une inflation de l’erreur de type I. Pour y remédier, nous avons étudié l’apport d’approches PMX innovantes dans les études de choix de dose. Le « model averaging » couplée à un test du rapport de « vraisemblance combiné » a montré des résultats prometteurs et tend à promouvoir l’utilisation de la PMX dans les études de choix de dose. Pour les études dites d’apprentissage, les approches de modélisation sont utilisées pour accroitre les connaissances associées aux médicaments, aux mécanismes et aux maladies. Dans cette thèse, les mérites de l’analyse PMX ont été évalués dans le cadre de la maladie de Parkinson. En combinant la théorie des réponses aux items à un modèle longitudinal, l’analyse PMX a permis de caractériser adéquatement la progression de la maladie tout en tenant compte de la nature composite du biomarqueur. Pour conclure, cette thèse propose des méthodes d’analyses PMX innovantes pour faciliter le développement des médicaments et/ou les décisions des autorités réglementaires. / In the mid-1990, model-based approaches were mainly used as supporting tools for drug development. Restricted to the “rescue mode” in situations of drug development failure, the impact of model-based approaches was relatively limited. Nowadays, the merits of these approaches are widely recognised by stakeholders in healthcare and have a crucial role in drug development for progressive diseases. Despite their numerous advantages, model-based approaches present important drawbacks limiting their use in confirmatory trials. Traditional pharmacometric (PMX) analyses relies on model selection, and consequently ignores model structure uncertainty when generating statistical inference. The problem of model selection is potentially leading to over-optimistic confidence intervals and resulting in a type I error inflation. Two projects of this thesis aimed at investigating the value of innovative PMX approaches to address part of these shortcomings in a hypothetical dose-finding study for a progressive disorder. The model averaging approach coupled to a combined likelihood ratio test showed promising results and represents an additional step towards the use of PMX for primary analysis in dose-finding studies. In the learning phase, PMX is a key discipline with applications at every stage of drug development to gain insight into drug, mechanism and disease characteristics with the ultimate goal to aid efficient drug development. In this thesis, the merits of PMX analysis were evaluated, in the context of Parkinson’s disease. An item-response theory longitudinal model was successfully developed to precisely describe the disease progression of Parkinson’s disease patients while acknowledging the composite nature of a patient-reported outcome. To conclude, this thesis enhances the use of PMX to aid efficient drug development and/or regulatory decisions in drug development.

Some basic but desirable employability skills are antecedents of job performance. The Workplace Skills Survey (WSS) is a 48-item situational judgment test (SJT) used to assess non-technical workplace skills for both entry-level and experienced workers. Unfortunately, the psychometric evidence for use of its scores is far from adequate. The purpose of current study was two-fold: (a) to examine the proposed structure of WSS scores using confirmatory factor analysis (CFA), and (b) to explore the WSS item functioning and performance using item response theory (IRT). A sample of 1,018 Jamaican unattached youth completed the WSS instrument as part of a longitudinal study on the efficacy of a youth development program in Jamaica. Three CFA models were tested for the construct validity of WSS scores. Parameter estimations of item difficulty, item discrimination, and examinee’s proficiency estimations were obtained with item response theory (IRT) and plotted in item characteristics curves (ICCs) and item information curves (IICs). Results showed that the WSS performed quite well as a whole and provided precise measurement especially for respondents at latent trait levels of -0.5 and +1.5. However, some modifications of some items were recommended. CFA analyses showed supportive evidence of the one-factor construct model, while the six-factor model and higher-order model were not achieved. Several directions for future research are suggested.

The thesis is focused on creating a test battery for the sense of rhythm assessment for university students in the field of Physical Education and sport. Music-sports practice lacks such tests and so far it is forced to use only tests designed for the music field. The structure of the new tests corresponds to the characteristics of the sense of rhythm from the point of view of music psychology. The tests are divided into three sections, which are Perceptual Skills and Activities (items 1 to 18), Reproductive Skills and Activities (items 19 to 27) and Production Skills and Activities (item 28). The Classical test theory (factor analysis) and Item response theory (two-parametric model) were used for the statistical processing of results. The methods also included the calculation of the reliability and validity of the test. The expected refusal of the proposed hypothesis was confirmed in both the classical test theory and item response theory. The exception was model 4, whose fit indexes, especially TLI = 0.537, showed a lack of evidence for the refusal of the hypothesis rather than for a perfect fit of the model with the data. The aim of the research was to compile and test models whose conformity with the data would be the best. The most favorable conformity was for model No. 1 and No. 5. Model 1...

The ability to monitor client change in psychotherapy over time is vital to quality assurance in service delivery as well as the continuing improvement of psychotherapy research. Unfortunately, there is not currently a comprehensive, affordable, and easily utilized outcome measure for psychotherapy specifically normed and standardized for use in psychology training clinics. The current study took the first steps in creating such an outcome measure. Following development of an item bank, factor analysis and item-response theory analyses were applied to data gathered from a stratified sample of university (n = 101) and community (n = 261) participants. The factor structure did not support a phase model conceptualization, but did reveal a structure consistent with the theoretical framework of the research domain criteria (RDoC). Suggestions for next steps in the measure development process are provided and implications discussed.

Context: Medical management for head and neck cancer (HNC) often includes neck dissection surgery, a side effect of which is shoulder dysfunction. There is no consensus for which patient-reported outcome measure (PRO) is most appropriate to quantify shoulder dysfunction in this population. Objective: The aims of this research study were to: (1) use Rasch methodologies to assess construct validity and overall appropriateness of test score interpretation of Disability of the Arm, Shoulder and Hand (DASH), QuickDASH, Shoulder Pain and Disability Index (SPADI) and Neck Dissection Impairment Index (NDII) in the HNC population; (2) determine appropriateness of use of University of Washington Quality of Life (UW-QoL) shoulder subscale as a screening tool for shoulder impairment; (3) recommend a new PRO, or combination of PROs, that more accurately portrays the construct of shoulder dysfunction in the HNC population. Design: One hundred and eight-two individuals who had received a neck dissection procedure within the past 2 weeks to 18 months completed the PROs. Rasch methodologies were utilized to address the primary aim of the study through consideration of scale dimensionality [principal components analysis, item and person fit, differential item functioning (DIF)], scale hierarchy (gaps/redundancies, floor/ceiling effects, coverage of ability levels), response scale structure, and reliability (person and item reliability and separation statistics). The secondary aim was addressed through correlational analysis of the UW-QoL (shoulder subscale), DASH, QuickDASH, SPADI and NDII. Results: The DASH did not meet criteria for unidimensionality, and was deemed inappropriate for utilization in this sample. The QuickDASH, SPADI and NDII were all determined to be unidimensional. All scales had varying issues with person and item misfit, DIF, coverage of ability levels, gaps/redundancies, and optimal rating scale requirements. The NDII meets most requirements. All measures were found to meet thresholds for person and item separation and reliability statistics. The third aim of this study was not addressed because the NDII was determined to be appropriate for this population. Conclusions: Rasch analysis indicates the NDII is the most appropriate measure studied for this population. The QuickDASH and SPADI are recommended with reservation. The DASH and the UW-QoL (shoulder subscale) are not recommended.

No description available.