Spelling suggestions: "subject:"educationization -- tests anda remeasurements"" "subject:"educationization -- tests anda aremeasurements""
81 |
Separating the Signal From the Noise: An Examination of Student and Teacher Scores Based on Student Learning Objectives (SLOs) in One StateBuckley, Katie Hills 18 June 2015 (has links)
Despite the prevalence of student learning objectives (SLOs) in teacher evaluation systems throughout the United States, research on the validity of student and teacher SLO scores used for high-stakes decisions is lacking. For this reason, this dissertation is comprised of two chapters that examine student and teacher-level SLO performance data from select districts in one Race to the Top state. In Chapter 1, I describe the quality of student assessment data and the comparability of student scores across alternative growth targets. I find that in the first year of implementation, assessments from half of the courses in the sample contained indicators of poor data quality, including anomalous score distributions and small to negative correlations between student prescores and postscores. However, in the second year of implementation, when student SLO performance is incorporated into final teacher evaluation scores, far fewer assessments contained anomalous score distributions, and there is no evidence to suggest manipulation of student scores. In addition to the assessments, the choice of student growth target does have an impact on the comparability of student and teacher scores across districts and years.
Chapter 2 describes the validity and reliability of teacher SLO scores. I find that while teacher SLO scores are moderately stable across courses, they are not stable over time, likely due to changes made to the assessments and targets used to determine student SLO scores. Further, for teachers with both SLO scores and an alternative metric of performance based on student growth, the two metrics do not converge. Finally, teachers in courses with higher average student prescores and lower proportions of students with disabilities have slightly higher SLO scores. In general, results on teacher SLO scores were similar to those found with value-added based metrics of teacher performance. Findings from both chapters suggest that improvement in the quality of the assessments administered as well as greater consistency in the growth targets assigned to students, both within districts over time and across districts, will improve the validity of student and teacher SLO scores in this state. / Quantitative Policy Analysis in Education
|
82 |
A Three-Study Examination of Test-Based Accountability MetricsYee, Darrick 20 June 2017 (has links)
Recent state and federal policy initiatives have led to the development of a multitude of statistics intended to measure school performance. Of these, statistics constructed from student test scores number among both the most widely-used and most controversial. In many cases, researchers and policymakers alike are not fully aware of the ways in which these statistics may lead to unjustified inferences regarding school effectiveness. A substantial amount of recent research has attempted to remedy this, although much remains unknown.
This thesis seeks to contribute to these research efforts via three papers, each examining how a commonly-employed accountability statistic may be influenced by factors unrelated to student proficiency or school effectiveness. The first paper demonstrates how the discrete nature of test scores leads to biased estimates of changes in the percentage of “proficient” students between any two given years and examines estimators that provide better recovery of this parameter. The second paper makes use of a state-wide natural experiment to show that a change in testing program, from paper-and-pencil to computer-adaptive, may cause apparent changes in achievement gaps even when relative student proficiencies have remained constant. The third paper examines “growth-based” accountability metrics based on vertically-scaled assessments, showing that certain types of metrics based on gain scores can be modeled via nonlinear transformations of the underlying vertical scale. It then makes use of this result to investigate the potential magnitude of impacts of such transformations on growth-based school accountability ratings.
|
83 |
The effect of test length, IRT model, type of aberrance, and level of aberrance on the distribution and effectiveness of three appropriateness indices.Noonan, Brian W. January 1990 (has links)
There were two basic purposes for this study. The first purpose was to investigate the characteristics of the distributions of Lz, ECIZ4, and W3 for non-aberrant response patterns in combinations of test lengths (40 items and 80 items) and IRT model (the 2PLM and the 3PLM). The second purpose was to investigate the effectiveness of the three indices in twenty-four combinations of two test lengths, two IRT models, two types of aberrance, and three levels of aberrance. In order to investigate the distributions of appropriateness indices in non-aberrant response patterns, data were generated by computer to simulate various measurement conditions. Item parameters were generated within specified ranges to produce similar tests for the two test lengths and two IRT models. Simulated examinees were generated from the normal (0,1) distribution. Two thousand non-aberrant, response vectors were generated for each of four conditions, test length by IRT model. The three appropriateness indices, Lz, ECIZ4, and W3 were calculated for each examinee. This procedure was replicated fifty times for each of the four combinations of test length and IRT model. Of the three indices, ECIZ4 produced the most stable distributions over replications. To examine the effect of test length and IRT model on characteristics of the distributions of the indices, the mean, standard deviation, skewness, and kurtosis were computed for each index in each of the combinations of test length and IRT model over fifty replications. There were no significant effects for either test length or IRT model on the means of the three indices. Based on skewness and kurtosis, the distributions of ECIZ4 most closely approximated normality, while the distribution of W3 was least normal. To establish false positive rates, the tails of the distributions of each index were then examined at P$\sb $, P$\sb $, P$\sb $, and P$\sb{25}$ for each of the four conditions. Of the three indices ECIZ4 seemed least affected and W3 most affected by test length, IRT model, and the interaction of test length and IRT model. To investigate the effectiveness of the indices, aberrant response patterns were generated for the twenty-four combinations of the four variables (2 test lengths x 2 models x 2 types of aberrance x 3 levels of aberrance). Four thousand simulated examinees were generated for each of the twenty-four combinations and each index was computed for each examinee for each of the twenty-four combinations. The detection rates of the indices were then computed and compared for each index for each of the twenty-four conditions. Overall, the 80 item test produced somewhat better detection rates than the 40 item test and the 2PLM better rates than the 3PLM. Spuriously low scores tended to produce slightly higher detection rates than spuriously high scores under most conditions. Higher levels of aberrance tended to produce higher detection rates although for some conditions there was little difference between 15% and 30% aberrance. Lz and ECIZ4 tended to produce better detection rates than W3; however, no detection rates seemed to be as high as those reported in previous research. (Abstract shortened by UMI.)
|
84 |
A Monte Carlo comparison of the Type I error rates of the likelihood ratio chi-square test statistic and Hotelling's two-sample T2 on testing the differences between group means.Boulet, John R. January 1990 (has links)
The present paper demonstrates how Structural Equation Modelling (SEM) can be used to formulate a test of the difference in means between groups on a number of dependent variables. A Monte Carlo study compared the Type I error rates of the Likelihood Ratio (LR) Chi-square ($\chi\sp2$) statistic (SEM test criterion) and Hotelling's two-sample T$\sp2$ statistic (MANOVA test criterion) in detecting differences in means between two independent samples. Seventy-two conditions pertaining to average sample size ((n$\sb1$ + n$\sb2$)/2), extent of inequality of sample sizes (n$\sb1$:n$\sb2$), number of variables (p), and degree of inequality of variance-covariance matrices ($\Sigma\sb1$:$\Sigma\sb2$) were modelled. Empirical sampling distributions of the LR $\chi\sp2$ statistic and Hotelling's T$\sp2$ statistic consisted fo 2000 samples drawn from multivariate normal parent populations. The actual proportion of values that exceeded the nominal levels are presented. The results indicated that, in terms of maintaining Type I error rates that were close to the nominal levels, the LR $\chi\sp2$ statistic and Hotelling's T$\sp2$ statistic were comparable when $\Sigma\sb1$ = $\Sigma\sb2$ and (n$\sb1$ + n$\sb2$)/2:p was relatively large (i.e., 30:1). However, when $\Sigma\sb1$ = $\Sigma\sb2$ and (n$\sb1$ + n$\sb2$)/2:p was small (i.e., 10:1) Hotelling's T$\sp2$ statistic was preferred. When $\Sigma\sb{1} \not=\Sigma\sb2$ the LR $\chi\sp2$ statistic provided more appropriate Type I error rates under all of the simulated conditions. The results are related to earlier findings, and implications for the appropriate use of the SEM method of testing for group mean differences are noted.
|
85 |
Problem-solving characteristics of relative novices and experts within an intermediate range of expertise in linear kinematics.Blais, Christine Lorraine. January 1990 (has links)
Within the context of this study, expertise is used to describe the range of skills (a continuum) which lies between those of novice and expert. Although some of these expert-novice differences have been identified, what is less understood is how an individual becomes an expert: the transition from novice to expert. As the study tests a specific hypothesis and seeks information related to a specific objective, it has both confirmatory and exploratory components. The independent variables were context, level of expertise and Problem Type and the dependent variables were solution time and solution patterns. There were two categories of context (familiar and unfamiliar), two levels of expertise (novice and expert) and two Problem Types (simultaneous and successive movement). Solution time was analyzed within a confirmatory framework and solution patterns within an exploratory framework. An information-processing approach to problem-solving was used. From 108 university students an inventory of contexts was compiled to produce familiar and unfamiliar isomorphic problems. The level of expertise of a second group of 57 subjects was based on educational background and produced Concept Map. From this process, two intermediate groups of subjects were identified as relative experts or novices. Each subject was presented with eight isomorphic problems, four in familiar and four in unfamiliar contexts, were presented to each subject. The subjects were presented with one of two Problem Types reflecting Simultaneous or Successive movements as defined by Piaget. The problem solutions were recorded using the technique developed by Ericsson and Simon (1984), were divided into 5-second intervals, and then evaluated using a Coding Grid developed for this study. Thus, the data submitted for analysis was based on a total of 224 problems. While the subjects in this study did represent two distinct levels of expertise, they did not evidence those characteristics associated with the extremes of the expert-novice continuum. There were no significant differences between experts and novices in their problem solution times, but the relative expert subjects did demonstrate some of the 'traditional' expert traits. In particular, experts evidenced an improved ability to recognize key information and, thereby, improve the accuracy of their performance. The more expert problem solver also used more conceptual, as distinct from computational, types of strategy. Overall, while there were no significant differences in solution time, many expert-novice distinctions arose when examining the processes whereby these solutions were achieved. In particular, while the experts tended to show an analytical approach, the novices were more speculative.
|
86 |
Étude des commentaires rétroactifs par les pairs dans le contexte de l'évaluation formative.Malara, Eric. January 2002 (has links)
Les courants de l'heure en éducation misent de plus en plus sur des modalités pédagogiques qui responsabilisent davantage les étudiants dans la prise en charge de leurs apprentissages. L'évaluation par les pairs constitue une de ces modalités pédagogiques car, parmi ses nombreux avantages, elle encourage un véritable échange d'idées. De nombreuses recherches ont validé l'approche de l'évaluation par les pairs avec des notes chiffrées ou des lettres. Les étudiants-maîtres pourtant dans leur carrière d'enseignant ne donneront pas uniquement des notes mais aussi des commentaires. Les mécanismes d'application et la nature de cette forme d'évaluation sont encore à explorer.
Cette recherche vise à examiner de manière plus approfondie les caractéristiques spécifiques des commentaires rétroactifs fournis par les pairs dans le contexte d'une évaluation formative de l'enseignement d'une leçon simulée. Plus spécifiquement, la question de recherche retenue dans cette étude est la suivante: dans un contexte d'évaluation formative pratiquée par les pairs, quelle sera la nature des commentaires des étudiants-maîtres en termes de pertinence, de précision, de direction et de nombre? (Abstract shortened by UMI.)
|
87 |
A Monte Carlo study of the effects of four factors on the effectiveness of the LZ and ECIZ4 appropriateness indices.Cole, Gary. January 1994 (has links)
While a test score may be valid for a group there may sometimes be reason to suspect its validity for an individual. Unusual examinee response patterns may indicate that the test may be invalid for the individual and quantitative measures called appropriateness indices have been developed to detect these unusual patterns. For a number of reasons, Lz and ECIZ4 have so far proven to be two of the most useful of these indices. There were three purposes for this study. The first purpose was to investigate the effects of four variables on the cutoff values of the indices: the range of the distribution of the b parameter (Diff), the level of the a parameter (Disc), IRT model (Model), and sample size used to estimate item parameters (Sampsiz). The second purpose was to investigate the effects of these same variables on the detection rates for response vectors that were made spuriously high(i.e. high aberrance) and for response vectors that were made spuriously low (i.e. low aberrance). The third purpose was to determine the extent to which detection rates obtained by using cutoff values from the standard normal distribution were similar to those obtained by using cutoff values obtained by simulating non-aberrant response vectors. Two levels were set for each of the four variables. For Diff, a broad and a narrow range of the b parameter was used. For Disc, a high and low level for the a parameter of the test items was used. For Model, the 2PL and 3PL models were used. For Sampsiz, a sample size of 1000 and 2500 was used to estimate item parameters. For each of the 16 combinations of these variables, non-aberrant as well as aberrant response vectors were simulated for a 60 item test. For the aberrant response vectors, both high and low aberrance was created by modifying ten of the test items. Detection rates were obtained at the.01,.05, and.10 false-positive rates using cutoff values based on the distribution of the non-aberrant response vectors and using cutoff values based on a standard normal distribution. The simulation of each combination of conditions was replicated 90 times. (Abstract shortened by UMI.)
|
88 |
The robustness of validity and efficiency of the one-sample t test in the presence of normal contamination.Jennings, Martha. January 1994 (has links)
The performance of parametric tests given data which are essentially normal but contain outliers is largely unknown. In this Monte Carlo study the robustness of validity and efficiency for the one-sample location problem are investigated. The Type I error rate and power of the one-sample t test given a normal underlying population are compared with the performance of this test given a systematic range of outlier contamination in the underlying population. Sample sizes of 8, 16, 32, 64, and 128 are included in the design. The robustness of validity results are explored using three sets of regression models. The first set of models is constructed using the parameters of the contamination model and is intended to inform the social science methodologist. The second set of models is constructed using skewness and kurtosis values. A third set of models is developed using an index of contamination proposed by Zumbo (1993). This set of models has practical relevance to the data analyst confronted with outlier contaminated data. Robustness of efficiency results are expressed using both power curves and a proposed fairly stringent criterion for power. In general, the results indicate that the one-sample t test demonstrates fairly stringent robustness of validity for all the symmetric contamination explored. When contamination is asymmetric the Type I error rate becomes inflated as the proportion of contamination increases. If robustness of validity is intact, power is not greatly affected when medium or large effect sizes are examined. This is not necessarily true for small effect sizes and the problems are further exacerbated when sample sizes are also small.
|
89 |
Étude des qualités métrologiques d'un instrument d'observation des comportements déviants en classe d'éducation physique.Audette, Sylvain. January 1994 (has links)
Les buts de l'etude sont de proposer une demarche conceptuelle appropriee a la mesure du niveau de discipline en classe et de verifier les qualites metrologiques d'un unstrument d'observation des comportements deviants en classe d'education physique. Le terme "comportement deviant" designe une transgression a une regle en classe. L'instrument elabore par Kennedy (1980) utilise l'observation directe des comportments deviants suivis d'une reaction de l'enseignant et fait l'objet de la presente etude. Dans un premier temps, l'instrument est traduit et adapte en francais et dans un deuxieme temps est soumis a des analyses de fidelite et de validite. Apres une periode d'entrai nement intensive aupres des deux codeurs, 20 groupes-classes ont participe a l'etude. L'analyse des resultats demontre une bonne fidelite de l'instrument en ce qui concerne les 22 categories de comportements, les cinq sous-categories et le total pour la seance. Cependant, la fidelite calculee avec le pourcentage d'accord est faible si on tient compte des erreurs sur le nombre d'eleves impliques. L'indice de validite concourante (0,86) calcule entre la frequence de comportements deviants et une evaluation par l'enseignant par seance s'avere tres bon. (Abstract shortened by UMI.)
|
90 |
The influence of sample size, effect size, and percentage of DIF items on the performance of the Mantel-Haenszel and logistic regression DIF identification procedures.Kennedy, Michael. January 1994 (has links)
The frequent use of standardized tests for admission, advancement, and accreditation has increased public awareness of measurement issues, in particular, test and item bias. The logistic regression (LR) and Mantel-Haenszel (MH) procedures are relatively new methods of detecting item bias or differential item functioning (DIF) in tests. In only a few studies has the performance of these two procedures been compared. In the present study, sample size, effect size, and percentage of DIF items in the test were manipulated in order to compare detection rates of uniform DIF by the LR and MH procedures. Simulated data, with known amounts of DIF, were used to evaluate the effects of these variables on DIF detection rates. In detecting uniform DIF, the LR procedure had a slight advantage over the MH procedure at the cost of increased false positive rates. P-value difference was definitely a more accurate measure of the amount of DIF than b value difference. (Abstract shortened by UMI.)
|
Page generated in 0.1706 seconds