This study compared three methods of selecting the best discriminating test items and the resultant test reliability of mastery/nonmastery classifications. These three methods were (a) the agreement approach, (b) the phi coefficient approach, and (c) the random selection approach. Test responses from 1,836 students on a 50-item physical science test were used, from which 90 distinct data sets were generated for analysis. These 90 data sets contained 10 replications of the combination of three different sample sizes (75, 150, and 300) and three different numbers of test items (15, 25, and 35). The results of this study indicated that the agreement approach was an appropriate method to be used for selecting criterion-referenced test items at the classroom level, while the phi coefficient approach was an appropriate method to be used at the district and/or state levels. The random selection method did not have similar characteristics in selecting test items and produced the lowest reliabilities, when compared with the agreement and the phi coefficient approaches.

Educators who use criterion-referenced measurement to ascertain the current level of performance of an examinee in order that the examinee may be classified as either a master or a nonmaster need to know the accuracy and consistency of their decisions regarding assignment of mastery states. This study examined the sampling distribution characteristics of two reliability indices that use the squared-error agreement function: Livingston's k^2(X,Tx) and Brennan and Kane's M(C). The sampling distribution characteristics of five indices that use the threshold agreement function were also examined: Subkoviak's Pc. Huynh's p and k. and Swaminathan's p and k. These seven methods of calculating reliability were also compared under varying conditions of sample size, test length, and criterion or cutoff score. Computer-generated data provided randomly parallel test forms for N = 2000 cases. From this, 1000 samples were drawn, with replacement, and each of the seven reliability indices was calculated. Descriptive statistics were collected for each sample set and examined for distribution characteristics. In addition, the mean value for each index was compared to the population parameter value of consistent mastery/nonmastery classifications. The results indicated that the sampling distribution characteristics of all seven reliability indices approach normal characteristics with increased sample size. The results also indicated that Huynh's p was the most accurate estimate of the population parameter, with the smallest degree of negative bias. Swaminathan's p was the next best estimate of the population parameter, but it has the disadvantage of requiring two test administrations, while Huynh's p index only requires one administration.

No description available.

published_or_final_version / Education / Master / Master of Education

The problem of this study was to examine the equivalency of two different types of criterion-referenced item-selection techniques on simulated data as item pools varied in degrees of item difficulty. A pretest-posttest design was employed in which pass-fail scores were randomly generated for item pools of twenty-five items. From the item pools, the two techniques determined which items were to be used to make up twelve-item criterion-referenced tests. The twenty-five items also were rank ordered according to the discrimination power of the two techniques.

The promise of increased student achievement through educational reform is delivered still-born if teachers do not know how to implement complex instructional practices and sophisticated analysis of student performance. Metacognitive awareness is crucial to the adoption and application of proven educational initiatives. Teachers who successfully implement criterion-referenced assessment instruction, scoring rubrics, transfer to their students the metacognitive knowledge and skills of how to learn. This study is predicated on the research assumptions that metacognition and its attendant skills are critical to the successful implementation of scoring rubrics. A researcher-developed instrument, the Scoring Rubrics Inventory (SRI) and the Metacognitive Awareness Inventory (MAI) were distributed to core-subject teachers from three large public schools in Southwest Louisiana. From a population of sixty-eight (N=68) voluntary participants, eighteen teacher-participants self-reported as high implementers of scoring rubrics, thirty-nine as mid-level implementers and eleven as low-level implementers. From this population, twelve subjects were randomly selected (four high, four mid-level, and four lowlevel implementers) by an outside rater for double-blind observations and interviews. Pearson Product Moment correlations of the SRI and the MAI revealed five significant pairings using an alpha level of .05. The statistical results, coupled with the observation and interview findings from the sample-subjects established the consistency and stability of the Scoring Rubrics Inventory. Further, the totality of the results reported here support the research hypothesis of the study: H1: There is a significant correlation between the metacognitive awareness of secondary school core-subject teachers and the successful implementation of scoring rubrics. The results of the study indicated that secondary school core-subject teachers who successfully implement scoring rubrics possess a metacognitive awareness that transcends professional development training. The findings also suggested that teacher-participants who do not implement scoring rubrics either cannot or lack commitment to the innovation. Implications for teacher educators and school leaders indicated the need to: identify those persons who require additional professional development training; include operational strategies and modeling of successful implementation during training; and maintain a consistent training program in scoring rubrics. Recommendations for future research were offered.

Student Number: 9710860W School of Education Doctor of Philosophy / The study investigates the extent to which primary school teachers understood and implemented the requirements of the continuous assessment programme that has been introduced in primary schools in Swaziland. By focussing on teaching and learning activities that occurred during eight teachers’ lessons within the Salesian-Ekutsimuleni zone in Manzini, and what they expressed as intentions and justifications for these activities, it tries to clarify, in particular, the relationship between their assessment strategies and the broader educational principles promoted by the programme. Classroom observations and stimulated interviews were used to capture, respectively, data on what teachers did and principles that informed their behaviour. Codes abstracted from classroom observations and grounded on the views expressed by teachers indicate that they followed slavishly what was contained in the curriculum support materials with which they had been provided. Even though they used assessment strategies promoted by the Continual Assessment (CA) programme, their assessment strategies prioritized knowledge-retention rather than the cognitive development advocated by the programme and, in a specific sense, implied by lesson objectives they had to fulfil. The conclusion is that teachers could not translate the rhetoric of the CA programme into relevant professional judgement, decisions and practices without exposure to meaningful development programmes. Key Words Swaziland, Continuous Assessment, Criterion - Referenced Assessment, Mastery learning, Fordism, Post-Fordism

In recent years, Sweden has adopted a criterion-referenced grading system, where the grade outcome is used for several purposes, but foremost for educational evaluation on student- and school levels as well as for selection to higher education. This thesis investigates the consequences of using criterion-referenced measurement for both educational evaluation and selection purposes. The thesis comprises an introduction and four papers that empirically investigate school grades and grading practices in Swedish upper secondary schools. The first paper investigates the effect of school competition on the school grades. The analysis focuses on how students in schools with and without competition are ranked, based on their grades and SweSAT scores. The results show that schools that are exposed to competition tend to grade their students higher than other schools. This effect is found to be related to the use of grades as quality indicators for the schools, which means that schools that compete for their students tend to be more lenient, hence inflating the grades. The second paper investigates grade averages over a six-year period, starting with the first cohort who graduated from upper secondary school with a GPA based on criterion-referenced grades. The results show that grades have increased every year since the new grading system was introduced, which cannot be explained by improved performances, selection effects or strategic course choices. The conclusion is that the increasing pressure for high grading has led to grade inflation over time. The third paper investigates if grading practices are related to school size. The study is based on a similar model as paper I, but with data from graduates over a six-year period, and with school size as the main focus. The results show small but significant size effects, suggesting that the smallest schools (<300 students) are higher grading than other schools, and that the largest schools (>1000 students) are lower grading than other schools. This is assumed to be an effect of varying assessment practices, in combination with external and internal pressure for high grading. The fourth and final paper investigates if grading practices differ among upper secondary programmes, and how the course compositions in the programmes affect how students are ranked in the process of selection to higher education. The results show that students in vocationally oriented programmes are higher graded than other students, and also favoured by their programmes’ course compositions, which have a positive effect on their competitive strength in the selection to higher education. In the introductory part of the thesis, these results are discussed from the perspective of a theoretical framework, with special attention to validity issues in a broad perspective. The conclusion is that the criterion-referenced grades, both in terms of being used for educational evaluation, and as an instrument for selection to higher education, are wanting both in reliability and in validity. This is related to the conflicting purposes of the instruments, in combination with few control mechanisms, which affects how grades are interpreted and used, hence leading to consequences for students, schools and society in general.

The purpose of this study was to assess the criterion validity of the Indiana Basic Competency Skills Test (IBCST) by exploring the relationships between scores obtained on the IBCST and (a) student gender, (b) teacher-assigned letter grades, (c) scores obtained on the Otis-Lennon School Ability Test (OLSAT), and (a) scores obtained on the Stanford Achievement Test (SAT). The subjects were 300 third grade students enrolled in a small mid-Indiana school system. Data collected included gender, age, IBCST scores, OLSAT scores, SAT scores, and teacher-assigned letter grades in reading and mathematics. An alpha level of .01 was used in each statistical analysis. Gender differences were investigated by comparisons of the relative IBCST pass/fail (p/f) frequencies of males and females and boys' and girls' correct answers on the IBCST Reading and Math tests. Neither the chi square analysis of p/f frequencies nor the multivariate analysis of variance of the IBCST scores disclosed significant gender differences. Therefore, subsequent correlational analyses were done with pooled data.The relationship of teacher-assigned letter grades to IBCST p/f levels was studied with nonparametric and parametric statistical techniques. The 2x3 chi squares computed between IBCST performance and letter grades in reading and math were significant. The analyses of variance of the data yielded similar results. Teacher grades were related to IBCST performance.Multiple regression analyses were used to study the relationships between the IBCST and OLSAT performances. Significant multiple R-squares of approximately .30 were obtained in each analysis. Scholastic aptitude was related to IBCST performance.Canonical correlation analyses were used to explore the relationships between the reading and mathematics sections of the IBCST and SAT. Both analyses yielded a single significant, meaningful canonical correlation coefficient. The canonical variable loadings suggested that the IBCST Reading and Math composites, as well as the SAT composites, were expressions of general achievement. Thus, levels of achievement on the criterion referenced IBCST and the norm referenced SAT were related. The results of the study support the criterion validity of the IBCST with traditional methods of assessment as criteria. / Department of Educational Psychology

The demand for innovative end users of information technology is increasing along with the proliferation of computer equipment within the workplace. This has resulted in increasing demands being made upon educational institutions responsible for the education of computer end users. The demands placed upon the teachers are particularly high. Large class groups and limited physical resources make the task especially difficult. One of the most time consuming, yet important, tasks is that of student evaluation. To effectively assess the practical work of information technology students requires intensive study of the storage media upon which the students'efforts have been saved. The purpose of this study was to assess the suitability of criterion-referenced testing techniques applied to the evaluation of end user computing students. Objective questions were administered to the students using Question Mark, a computer-managed test delivery system which enabled quick and efficient management of scoring and data manipulation for empirical analysis. The study was limited to the classroom situation and the assessment of primary spreadsheet skills. In order to operate within these boundaries, empirical techniques were used which enabled the timeous analysis of the students' test results. The findings of this study proved to be encouraging. Computer-mediated criterion-referenced testing techniques were found to be sufficiently reliable for classroom practice when used to assess primary spreadsheet skills. The validation of the assessment technique proved to be problematic because of the constraints imposed by normal classroom practice as well as the lack of an established methodology for evaluating spreadsheet skills. However, sufficient evidence was obtained to warrant further research aimed at assessing the use of computer-mediated criterion-referenced tests to evaluate information technology end user learning in situations beyond the boundaries of the classroom, such as a national certification examination.