No description available.

Als Konsequenz der besonderen Aufmerksamkeit nach der Veröffentlichung der Ergebnisse der ersten PISA-Studie, wurden auf unterschiedlichen Ebenen des deutschen Schulsystems evaluatorische Prozesse implementiert. In Deutschland fußen solche Messungen von Erträgen des Bildungssystems auf von der Konferenz aller Kultusminister (KMK) beschlossenen Bildungsstandards, die als Kompetenzstufenmodelle für die Messung operationalisiert wurden. Mit den Vergleichsarbeiten etablierte sich in Deutschland ein Verfahren, das die Ergebnisse solcher Messungen primär schulischen Rezipienten zur Verfügung stellt. Damit sich der intendierte Nutzen von Unterrichtsentwicklung realisiert, muss Validität im Sinne Kanes vorausgesetzt werden. Die vorliegende Arbeit untersucht dazu drei Aspekte des Validitätsarguments. Dies sind zum Ersten mit der Nutzungsperspektive eng verbundene psychometrische Anforderungen an das Instrument. Das vierte Kapitel untersucht dazu methodische Aspekte, welche als grundlegend für die Validität angesehen werden. Es werden Gewissheiten formuliert und in Frage gestellt, die für die Testkonstruktion, aber insbesondere für die Weiterarbeit mit den Testergebnissen in der Schule essentiell sind. Der Validität von Testwertinterpretationen über mehr als einen Messzeitpunkt widmet sich das fünfte Kapitel. Wegen der Art und Weise der Verknüpfung der Messskalen haben die Ergebnisse letztlich aber Relevanz für die Interpretation jeder Einzelmessung. Der dritte Aspekt von Nutzungsvoraussetzung wird im wohl meistzitierten Nutzungsmodell für die Vergleichsarbeiten von Helmke als technische Übermittlung der Rezeption untergeordnet und ist ein Element konsequenzieller Validität. Der Abruf von Rückmeldungen ist so trivial wie grundlegend und doch findet sich dazu im deutschsprachigen Raum nur eine einzige Veröffentlichung mit relevanten Daten. Die vorliegende Untersuchung ermöglicht erstmals einen dezidierten Einblick in diesen Aspekt der Nutzung von Vergleichsarbeiten. / As a consequence of the special attention following the release of the results of the first PISA-Study, evaluative processes were implemented at multiple different levels of the German educational system. In Germany, these empirical measurement of results are based on educational standards determined by the Standing Conference of the Ministers of Education and Cultural Affairs of the Länder in Germany, which were operationalized for the measurements as competence level models. With the Vergleichsarbeiten, a process was established in Germany that continues these measurements of competency outcomes, but whose results are primarily made available to recipients in schools. To realize the intended use of data-based lesson as well as school development, validity in Kane’s sense is required. For that reason, this paper examines three aspects of the validity argument. On one hand, these are psychometric requirements for the instrument, which are highly dependend on the intended use. In this regard, the fourth chapter examines methodical aspects seen as fundamental for validity. Certainties are defined and questioned, that are essential for test construction, but also especially for working with the test results in lesson development. The fifth chapter is concerned with the validity of test result interpretations spanning multiple measurement times, but because of the way the scales of measurement are connected, the results end up being relevant to the interpretation of singular measurements as well. The third aspect is an element of consequentional validity which, as \emph{technical transmission}, is subordinated to reception, in the most cited usage model for the Vergleichsarbeiten by Helmke. The download of Feedback is as trivial as it is fundamental, yet (at least in the German-speaking world) only a single publication with relevant data can be found. This examination allows, for the first time, a dedicated look into this aspect of the Vergleicharbeiten’s usage.

[pt] Nesta tese, trabalhou-se com a modelagem Rasch visando a apresentar alternativas mais práticas e de melhor qualidade em termos de medida, para dois cenários distintos. O primeiro está relacionado ao fato de que medir conhecimento é algo muito complexo e de difícil entendimento para profissionais que não são da área da psicometria. Por meio de experimentos envolvendo modelos da família Rasch, apresentamos a aplicabilidade e as potencialidades dessa modelagem para atender a novas demandas de avaliação em larga escala no Brasil. O segundo cenário relaciona-se à busca de medir, de modo o mais imparcial possível, itens de produção escrita, em que a nota recebida pelos alunos é influenciada pela subjetividade dos corretores, ou seja, corretores lenientes beneficiam alunos e corretores severos penalizam alunos. Diante desses dois cenários, esta tese tem os seguintes objetivos: (i) trazer para o âmbito das avaliações realizadas no Brasil uma modelagem matemática mais simples que aquela atualmente adotada, visando uma melhor comunicação com os professores, e; (ii) a possibilidade de operar não apenas com itens de múltipla escolha, corrigidos de forma automática, mas também com itens de produção escrita, em que a subjetividade dos corretores (severidade) é controlada pelo modelo psicométrico, gerando medidas de melhor qualidade. Para isso, utilizou-se a modelagem Rasch com multifacetas, abordando, por meio de casos práticos, as vantagens dessa modelagem em relação a outras metodologias atualmente adotadas no país. Assim, para a alcançarmos o primeiro objetivo, confrontamos a modelagem Rasch com multifacetas com a modelagem de três parâmetros logísticos em um estudo de efeito contexto em testes compostos por diferentes modelos de cadernos e com mais de uma disciplina avaliada por caderno e, para o segundo, comparamos as medidas de proficiência através da Rasch com multifacetas com as notas médias das duplas correções dadas pelos corretores aos alunos em testes do tipo redação. A partir dos resultados encontrados, concluímos que a Rasch com multifacetas pode ser utilizada de forma alternativa ou concomitante com as avaliações que utilizam a modelagem de três parâmetros logísticos, produzindo resultados mais rápidos e de entendimento mais fácil por parte dos professores e que, no caso de redações, as proficiências obtidas pela Rasch com multifacetas apresentaram medidas com melhores indicadores de fidedignidade e validade, quando comparadas com as medidas de notas via Teoria Clássica do Teste, sendo, portanto, uma alternativa mais viável para esse tipo de avaliação. Conclui-se essa tese apresentando situações de empregabilidade das metodologias estudadas. / [en] In this thesis, we worked with Rasch modeling, aiming to present more practical alternatives and better quality in terms of measurement, for two different scenarios. The first one is related to the fact that measuring knowledge is something very complex and difficult to understand for professionals who are not in the psychometrics area. Through experiments involving the Rasch family models, we present the applicability and the potentiality of this model to adequately comply with the new demands of the large-scale evaluation in Brazil. The second scenario is related to the search of measuring, in the most impartial way possible, written production items which grade received by the subjectivity of the raters (severity), that is, lenient raters benefit students and severe raters penalize them. In view of these two scenarios, this thesis has the following objectives: (i) to bring to the scope of the evaluations carried out in Brazil a simpler mathematical modeling than the currently adopted, aiming at a better communication with the teachers; and (ii) the possibility of operating not only with multiple choice items, corrected automatically, but also with written production items, in which the subjectivity of the raters (severity) is controlled by the psychometric model, generating better quality measures. For this, Many-Facet Rasch Measurement was used, approaching, through practical cases, the advantages of this modeling in relation to other methodologies currently adopted in the country. Thus, in order to reach the first objective, we confronted Many-Facet Rasch Measurement with the modeling of three logistic parameters in a study of context effect in tests composed by different models of test books and with more than one discipline evaluated by test book and, for the second one, we compared the measures of proficiency through the Many-Facet Rasch Measurement with the average scores of the double corrections given by the raters to the students in tests of the essay type. From the results found, we conclude that the Many-Facet Rasch Measurement can be used in an alternative or concomitant way with the evaluations that use the three logistic parameters model, producing faster results and easier to understand by the teachers and that, in the case of essays, the measures of proficiency obtained by Many-Facet Rasch Measurement presented measures with better reliability and validity indicators, when compared to the grading measures through the Classical Theory of Testing, being, therefore, a more viable alternative for this type of evaluation. This thesis concludes with situations of usability of the methodologies studied.

The Rasch sampler is an efficient algorithm to sample binary matrices with given marginal sums. It is a Markov chain Monte Carlo (MCMC) algorithm. The program can handle matrices of up to 1024 rows and 64 columns. A special option allows to sample square matrices with given marginals and fixed main diagonal, a problem prominent in social network analysis. In all cases the stationary distribution is uniform. The user has control on the serial dependency. (authors' abstract)

This study explores, from a test validity perspective, the extent to which the two variables of topic and background knowledge of topic have an effect on spoken performance in language proficiency interviews. It is argued that in assessment contexts where topics are randomly assigned to test takers, it is necessary to demonstrate that topics of tasks and the level of background knowledge that test takers brings to these topics do not exert an undue influence on test results. Otherwise, a validity threat may be introduced to the test. Data were collected from 82 Farsi speakers of English who performed on ten different topics, across three task types. Participants’ background knowledge of topics was elicited using self- report questionnaires while C-tests were used as a measure of general English language proficiency. Four raters assigned scores to spoken performances using rating scales. Semi- structured interviews were carried out with raters, upon completion of the rating process. A mixed- methods strategy of inquiry was adopted where findings from the quantitative analyses of score data (using Multi-Faceted Rasch Measurement, multiple regression and descriptive statistics) were synthesised with the results of the qualitative analyses of rater interviews and test takers’ content of speech in addressing the foci of the study. The study’s main findings showed that the topics used in the study exhibited difficulty measures which were statistically distinct i.e. topics, within a given task type, could not be considered parallel. However, the size of the differences in topic difficulties was too small to have a large practical effect on scores. Participants’ different levels of background knowledge were shown to have a consistent, systematic and statistically significant effect on performance with low levels of background knowledge posing the highest level of challenge for test takers and vice versa. Nevertheless, these statistically significant differences in background knowledge levels failed to translate into practically significant differences, as the size of the differences were too small to have a large impact on performance scores. Results indicated that, compared to general language proficiency which accounted for approximately 60% of the variance in spoken performance scores, background knowledge only explained about 1-3% of the variance. Qualitative analyses of data suggested lack of background knowledge to be associated with topic abandonment, disengagement from topic-related questions, and fewer opportunities for test takers to elaborate on topics. It was also associated with negative affective influence on test takers, particularly lower proficiency individuals. Taken together, the findings have theoretical, methodological and practical implications for second language speaking performance assessment.

The need to recruit quality professors into research-1 institutions continues to be a priority among American universities (Solem & Foote, 2004), but the ability to retain them is another story. The tenure process is quite demanding in such institutions where much emphasis is often placed on high quality research and publishing in reputable peer-reviewed journals (Boyer, 1992; DeFleur, 2007). Some scholars have identified faculty-to-faculty mentoring as one of many things institutions can promote to motivate beginning faculty to persist and complete the tenure process (Boice, 1991). On the other, others feel such activities are not necessary (Selby & Calhoun, 1998). Faculty-to-faculty mentoring could be beneficial, but institutions should have a tool to assess whether faculty members desire to receive mentoring. The literature review revealed that fields in social sciences lack well-developed quantitative data collection tool for such purposes. This study was, therefore, carried out to develop an instrument to fill that gap, especially in the era of data-driven decision making models. The new provides a foundation for other researchers to build on. The Rasch-Andrich Rating Scale model was applied in the item investigation and calibration. Response rate was 45.57% and the majority of respondents agreed that faculty-to-faculty mentoring is beneficial to faculty members who are on the tenure track.

Tarptautinis švietimo tyrimas PISA (Programme for International Student Assessment) tiria penkiolikmečių mokinių skaitymo gebėjimus, matematinį ir gamtamokslį raštingumą. Taip pat tyrimo metu renka informaciją apie mokinį, jo pomegius, informacinius ir komunikacinių technologijų išmanymą ir naudojimą, namų aplinką, mokyklą kurioje jis mokosi ir pan. Informacija surinkta tyrimo metu yra vertinga švietimo tyrėjams, sprendimų priėmėjams, švietimiečiams ir pan. Šio darbo tikslas – taikant hierarchinė regresiją rasti mokinių matematinio raštingumo rezultatų prikausomybę nuo mokymo(si) aplinkos veiksnių. Kadangi mokiniai yra pirmojo lygmens objektai, o mokyklos – antrojo lygmens, todėl analizuojant duomenis reikia atsižvelgti į duomenų hierarchiškumą. Besąlyginio hierarchinio modelio analizė atskleidė, kad 32 % Lietuvos penkiolikmečių matematinio raštingumo rezultatų skirtumų lemia mokyklos. Taikant hierarchinį tiesinį modeliavimą, sudarytas geriausias hierarchinis tiesinis modelis su atitinkamais kintamaisiais bei užrašyta lygtis, kuri gali būti naudojama Lietuvos penkiolikmečių matematinio raštingumo rezultatų prognozavimui. / International education survey PISA (Programme for International Student Assessment) examine 15-yers-old reading, mathematical and science literacy. A lot of information about student‘s, theirs hobbies, knowledge and use of information and comunication technology, home and school environment and etc. is collected during PISA survey. The collected information is valuable source of information for researchers, policy makers, educators and ect. The main purpose of this work, “Analysis of Lithuanian Students Literacy Using International Survey PISA 2009 Data“, is using hierarchical regression to find the students' mathematical literacy dependency of training and learning environment factors. The students are first-level objects and schools - at the second level, so data analysis should take into account the data hierarchy. Unconditional hierarchical model analysis revealed that 32 % of Lithuanian 15-years-old students’ mathematical literacy outcomes determined by the school. Best hierarchical linear model with relevant variables was done using hierarchical linear modelling. Also a prognostic equation of Lithuaninan 15-years-old students matematic literacy was created.

South African organisations face the challenge of creating organisations that will engage employees in ways that allow for the optimisation of their strengths. This can be achieved by following a strength-based approach (SBA). An SBA aims to achieve optimisation of human functioning, where talents and strengths are the focus and weaknesses are understood and managed. Although previous research suggests that an SBA has positive influences on individual and organisational outcomes, no instrument exists within the South African context that measures employees’ perception of the extent to which they believe their organisation makes use of their strengths and talents. Recently, a new scale was developed to address this need. However, no studies have been done to see how well the items of this scale function. The objectives of this research were to 1) conceptualise an SBA according to literature, 2) determine whether the items in the SBA scale are unidimensional, 3) to determine the internal validity and reliability of the new SBA scale, and 4) make recommendations for future research. A cross-sectional quantitative research design was used whereby online and hardcopy versions of the questionnaire were distributed to participants. A sample of 699 participants was collected from the Gauteng and North-West provinces. Rasch analysis was used to determine the reliability and validity of the items. Acceptable item reliability was found. Both the item and person separation indices were acceptable. Mean infit and outfit indices for both person and item were acceptable. The seven-point frequency-based Likert scale worked satisfactorily overall, although categories 0, 1, and 2 were under-utilised. Finally, the infit and outfit statistics for all eight items functioned satisfactorily, except for one item. Recommendations are made for practice as well as for future research. / Thesis (MA (Industrial Psychology))--North-West University, Potchefstroom Campus, 2012

There is a distinct lack of research into the concept of acute mental health crisis. Without investigating the concept of crisis itself, it is not possible to appreciate the attributes of crisis so that it can be measured. This has hampered the development of good psychometric tools for crisis. The aim of this research was to develop the first standardised, valid and reliable measure for the assessment of people presenting to Crisis Resolution and Home Treatment (CRHT) teams. This research utilised qualitative and quantitative research techniques to develop a crisis measure starting with a comprehensive investigation into the concept of acute mental health crisis to identify an item pool and clinically credible item rating scale. A prototype crisis measure was developed and piloted in two NHS CRHTs and data collected. This data was analysed to identify the key areas of crisis assessment (the subscales), a flexible rating scale and scoring system creating a measure named the Crisis Risk and Adaptive Functioning Tool (CRAFT). The CRAFT provides patient crisis profiles highlighting areas of strength, resilience, weakness and vulnerability. A thorough investigation of crisis was completed with CRHT staff and patients through interviews and focus groups. An initial 143 item pool and clinically credible item rating scale were identified and developed into a prototype pilot crisis measure. This measure utilised a flexible rating approach encapsulating both risk and protective factors believed to reflect clinical practice. The measure was piloted and the data analysed to assess the structure of the crisis measure’s item pool using the statistical techniques of Principal Component Analysis (PCA) and Rasch analysis. These analyses resulted in a 66 item measure with 8 unidimensional subscales including; 1) Crisis Recovery Indicators, 2) Adaptive Decision Making, 3) Risk of Harm to Self, 4) Mediating Factors, 5) Daily Structure, 6) Risk of Harm to Others, 7) Feelings and Affect, and 8) Basic Needs. The total variance explained by these 8 subscales was 67.6% with excellent internal reliability as indicated by a Cronbach’s alpha coefficient of 0.98 (p<0.001) and temporal reliability indicated by Spearman’s correlation of 0.971 (p<0.001, one tailed). This suggests that this measure has a strong internal structure and provides stable outcomes over time at both the subscale and global overall measurement levels. Receiver Operator Characteristic curve analysis supported the identification of cut-offs to indicate low, moderate and high levels of crisis and were shown to have good levels of sensitivity and specificity for the crude discrimination between individuals who require CRHT treatment and individuals who do not require CRHT treatment (sensitivity 0.89 and specificity 0.73) and for accurately discriminating between the basic treatment levels of low, moderate and high (sensitivity 0.80; specificity 0.69). One of the great advantages of utilising the Rasch model is that it supports the identification of key characteristics from an item pool. Application of the subscales and the overall measure to the Rasch model identified items that were most representative of underlying constructs and risk, highlighting items of essential essence for assessing crisis in the context of community treatment. These items may act as useful clinical and risk indicators for community assessment. After considering the evidence from the PCA and Rasch analysis for the underpinning construct, the measure was named the Crisis Risk and Adaptive Functioning Tool (CRAFT) to encapsulate both the risk and adaptive functioning (coping and management) aspects measured by the tool. There are a number of clinical implications resulting from the development of CRAFT for the assessment of crisis. This research clearly identifies 8 key areas for crisis assessment and the specific items that describe them. This promises to be a powerful clinical tool as it clarifies the main areas of concern and importance for crisis assessment and provides mental health professionals with a means of assessing and monitoring patients experiencing crisis. In addition to the clinical benefits offered by the CRAFT, it provides an approach to assessing and monitoring crisis to support further research in the area of acute mental health crisis. This research offers significant steps towards the development of a quality measure for crisis assessment. However, it is acknowledged that the process of measurement development is never complete. It simply evolves over time with the aim of coming closer to the valued direction.

This study investigated the effects of test length (10, 20 and 30 items), scoring schema (proportion of dichotomous ad polytomous scoring) and item analysis model (IRT and Rasch) on the ability estimates, test information levels and optimization criteria of mixed item format tests. Polytomous item responses to 30 items for 1000 examinees were simulated using the generalized partial-credit model and SAS software. Portions of the data were re-coded dichotomously over 11 structured proportions to create 33 sets of test responses including mixed item format tests. MULTILOG software was used to calculate the examinee ability estimates, standard errors, item and test information, reliability and fit indices. A comparison of IRT and Rasch item analysis procedures was made using SPSS software across ability estimates and standard errors of ability estimates using a 3 x 11 x 2 fixed factorial ANOVA. Effect sizes and power were reported for each procedure. Scheffe post hoc procedures were conducted on significant factos. Test information was analyzed and compared across the range of ability levels for all 66-design combinations. The results indicated that both test length and the proportion of items scored polytomously had a significant impact on the amount of test information produced by mixed item format tests. Generally, tests with 100% of the items scored polytomously produced the highest overall information. This seemed to be especially true for examinees with lower ability estimates. Optimality comparisons were made between IRT and Rasch procedures based on standard error rates for the ability estimates, marginal reliabilities and fit indices (-2LL). The only significant differences reported involved the standard error rates for both the IRT and Rasch procedures. This result must be viewed in light of the fact that the effect size reported was negligible. Optimality was found to be highest when longer tests and higher proportions of polytomous scoring were applied. Some indications were given that IRT procedures may produce slightly improved results in gathering available test information. Overall, significant differences were not found between the IRT and Rasch procedures when analyzing the mixed item format tests. Further research should be conducted in the areas of test difficulty, examinee test scores, and automated partial-credit scoring along with a comparison to other traditional psychometric measures and how they address challenges related to the mixed item format tests.