This study was undertaken to evaluate the impact of modeling decisions made by those charged with implementing teacher evaluation systems that incorporate student achievement data; such choices include how growth is to be modeled, whether student characteristics are to be controlled for, how many years of data are to be used, and which test subject is to be selected. Using a three-cohort longitudinal data set from a school district in which reading and mathematics test scores from a vertically-scaled assessment allowed determination of growth in grades three, four, and five, estimated teacher effects were derived from five value-added models, and the resulting rank orderings of the teachers were examined. The models compared were a covariate adjustment model that conditioned on prior achievement only, a covariate adjustment model that conditioned on certain student characteristics as well as prior achievement, a gain score model, the growth model underlying the vertically-scaled assessment, and student growth percentiles. Teacher rank orderings derived under the five models were highly consistent with one another using either one or three classroom years of test scores. Only when the movement of teachers between quartiles was examined did a difference in performance between some models emerge. The high degree of consistency between the two covariate adjustment models suggested that control for student-level characteristics was unnecessary. Using three years of test scores rather than one led to a small decrease in between-model correlations and a small increase in teacher movement between quartiles. Comparison of teacher value-added based on reading scores versus mathematics scores gave mixed results, with between-model correlations in mathematics being slightly higher than those for reading but with reading showing greater consistency in quartile movement between cohorts. The year-to-year change in teacher rank orderings was very striking, as low, and even negative, correlations emerged between years. Movement of teachers between quartiles from one year to the next was far greater than that observed when comparing the modeling conditions. Using a teacher rating scheme in which groups of teachers were distinguished from average effectiveness if they appeared in the extremes of the rankings, nearly half of teachers changed ratings from one year to the next. Such low inter-temporal stability of teacher value-added is a significant result that should be considered by all stakeholders in teacher evaluation.

Thesis advisor: Henry I. Braun / With teachers being largely held accountable for student learning outcomes, it is of critical importance to identify effective and ineffective teachers through the development and implementation of a successful teacher evaluation system. Addressing the call to explore indicators of teacher effectiveness and enhance the traditional methods and practices of teacher evaluation, this study extends current efforts investigating different approaches to measuring teacher effectiveness through exploration of the relationship between two indicators of teacher effectiveness: the value-added estimates based on student test performance and growth, and the quality of student learning objectives (SLO) developed by teachers. It uses data from a large school district in North Carolina, comprising student achievement outcomes in mathematics and reading across five grades and three years. Different hierarchical linear models are employed to obtain teachers' VAM estimates with regression adjustments for prior years of achievement, student background characteristics, and teacher level covariate adjusted for each set of models. Weighted Least Squares (WLS) analysis, logistic regression, and point-biserial analysis are used to examine the variations in the relationships among teachers' VAM estimates, SLO quality and SLO attainment status across years and grades. The HLM results revealed fluctuations in teachers' VAM rankings obtained at different stages of the model sequence that caused the correlations with SLO quality to vary as well. The WLS results indicated that the correlations between VAM and SLO quality also varied across years and grades. Further data analysis revealed generally weak associations between SLO quality and attainment status, as well as those between teachers' VAM estimates and whether their SLOs were achieved. / Thesis (PhD) — Boston College, 2015. / Submitted to: Boston College. Lynch School of Education. / Discipline: Educational Research, Measurement and Evaluation.

The purpose of this study was to examine the potential for using propensity score-based matching methods to estimate teacher contributions to student learning. Value-added models are increasingly used in teacher accountability systems in the United States in spite of ongoing qualms about the validity of teacher quality estimates resulting from those models. Using a large national dataset, teacher effects were estimated for 435 teachers using both value-added and propensity score-based approaches. The two approaches resulted in teacher effect estimates that were moderately correlated, with propensity score-based estimates more highly correlating with the value-added estimates as the matching ratio was increased. For many teachers' students, finding a set of matched control students was impossible unless the set of matching variables was reduced. Results suggest that many teachers have classroom compositions that are unusual, making evaluation of the teachers' impacts on student outcomes problematic. It was also found that, while value-added estimates were relatively insensitive to covariate inclusion choices or method of effect estimation, propensity score-based estimates were somewhat sensitive. Propensity score-based teacher effect estimates offer promise both for better accounting for classroom composition and student background variables and for indicating when a teacher's context is unique with respect to those variables, making the teacher's impact challenging to evaluate.