No description available.

The reporting of scores on educational tests is at times misunderstood, misinterpreted, and potentially confusing to examinees and other stakeholders who may need to interpret test scores. In reporting test results to examinees, there is a need for clarity in the message communicated. As pressure rises for students to demonstrate performance at a certain level, the communication of scores to the public needs to be examined. Although public school student testing often is placed in the spotlight, this study examines score reporting in teacher certification, which may not have the same complexities of student test score reporting, but does have the equally critical need to effectively communicate scoring information. The purpose of this study was to create multiple teacher certification examinee score reports based on findings in the literature on educational test score reporting, as well as marketing and design principles, and to conduct interviews and focus groups to gather feedback on the comprehension and preferences in interpreting the designed score reports and results. Different approaches for reporting test scores were used to design the score reporting materials for a hypothetical teacher certification testing examinee who had not passed. Educators and educational testing professionals were convened and interviewed to review the score reports and offer feedback, suggestions and discussion. The findings are covered in great detail. Using the findings, a final model score report was designed, which was then reviewed with doctoral students in educational measurement. Through this process, some clear patterns and differences arose. Overall, there was a desire on the educator and doctoral student end to provide as much information as possible, where supported by sound measurement principles. The reporting of raw performance information, as well as accommodating comprehension styles by providing performance information in contextual, statistical and visual ways were requested. Upon addressing these requests, two areas that may not have full clarity and direction remained: The process of converting raw score performance to a scaled score (participants wanted more information on this process), and information provided that could address candidate weak areas, directing examinees to materials that could improve their studies, understanding, and examination performance.

Validity is a crucial part of all forms of measurement, and especially in instruments that are high-stakes to the test takers. The aim of this thesis was to examine theory and validity evidence for a recently revised large-scale instrument used for selection to higher education in Sweden, the Swedish Scholastic Assessment Test (SweSAT), as well as identify threats to its validity. Previous versions of the SweSAT have been intensely studied but when it was revised in 2011, further research was needed to strengthen the validity arguments for the test. The validity approach suggested in the most recent version of the Standards for education and psychological testing, in which the theoretical basis and five sources of validity evidence are the key aspects of validity, was adopted in this thesis. The four studies that are presented in this thesis focus on different aspects of the SweSAT, including theory, score reporting, item functioning and linking of test forms. These studies examine validity evidence from four of the five sources of validity: evidence based on test content, response processes, internal structure and consequences of testing. The results from the thesis as a whole show that there is validity evidence that supports some of the validity arguments for the intended interpretations and uses of SweSAT scores, and that there are potential threats to validity that require further attention. Empirical evidence supports the two-dimensional structure of the construct scholastic proficiency, but the construct requires a more thorough definition in order to better examine validity evidence based on content and consequences for test takers. Section scores provide more information about test takers' strengths and weaknesses than what is already provided by the total score and can therefore be reported, but subtest scores do not provide additional information and should not be reported. All four quantitative subtests, as well as the Swedish reading comprehension subtest, are essentially free of differential item functioning (DIF) but there is moderate DIF that could be bias in two of the four verbal subtests. Finally, the equating procedure, although it appears to be appropriate, needs to be examined further in order to determine whether it is the best practice available or not for the SweSAT. Some of the results in this thesis are specific to the SweSAT because only SweSAT data was used but the design of the studies and the methods that were applied serve as practical examples of validating a test and are therefore likely useful to different populations of people involved in test development, test use and psychometric research. Suggestions for further research include: (1) a study to create a more clear and elaborate definition of the construct, scholastic proficiency; (2) a large and empirically focused study of subscore value in the SweSAT using repeat test takers and applying Haberman’s method along with recently proposed effect size measures; (3) a cross-validation DIF-study using more recently administered test forms; (4) a study that examines the causes for the recurring score differences between women and men on the SweSAT; and (5) a study that re-examines the best practice for equating the current version of the SweSAT, using simulated data in addition to empirical data.