Thesis advisor: Henry I. Braun / Addressing the call to provide hard evidence on undergraduate student outcomes and make comparisons across higher education institutions for accountability purposes, this study extends current efforts in measuring higher education outcomes and explores the differences in three value-added methodologies. Using the CIRP freshman and senior survey data from 2002 and 2006, this study examines noncognitive higher education outcomes with a focus on civic engagement. The three value-added methodologies examined are: an OLS-based cross-sectional method, an HLM-based cross-sectional method, and an HLM-based longitudinal method. Rather than seek to establish which methodology is superior, this study intends to provide empirical evidence concerning the similarities and differences in estimating institutional effectiveness with regard to civic engagement. First, several student-level and institution-level covariates were found to be associated with a measure of civic engagement in the senior year after adjusting for their level of civic engagement as freshmen. The model comparison further revealed some advantages in the HLM-based longitudinal method over the other two methods, such as providing a more accurate institutional value-added estimate and the ability to account for a relatively large percent of the total variance in the civic engagement measure when using the same covariates. Next, among all pairs of model comparisons, results from the two HLM-based methods agreed the most (r=.80). However, institutional rankings fluctuate dramatically, even when comparing institutions within small peer groups. Finally, the findings highlighted great divergences among different value-added methodologies in identifying institutions that perform significantly differently from the average for accountability purposes. / Thesis (PhD) — Boston College, 2012. / Submitted to: Boston College. Lynch School of Education. / Discipline: Educational Research, Measurement, and Evaluation.

Value-added models are complex statistical formulas that aim to isolate the effect a teacher has on student learning. States and districts across the nation are adopting laws and policies that will evaluate teachers, in part, using the results provided by value-added models. In many states and districts, these evaluations will be used to inform high-stakes decisions about teacher salary and retention. However, value-added models are imperfect tools for assessing teacher effectiveness, and many scholars have argued that they are not appropriate for use in high-stakes decisions. This Article provides a brief history of the use of value-added models in public education and summarizes the major criticisms of using value-added models. In this context, the Article analyzes and evaluates the extent to which substantive due process claims brought by teachers adversely affected by the results of value-added models will be successful. The Article concludes that while the system as a whole is rationally related to the objective of improving the overall effectiveness of the teaching workforce, in certain cases, individual teachers will be able to successfully claim that the results of their value-added model led to a termination that was arbitrary and capricious. Finally, the paper offers some recommendations to states and school districts on how to implement an evaluation system using value-added models to avoid substantive due process violations. / text

abstract: Teacher evaluation policies have recently shifted in the United States. For the first time in history, many states, districts, and administrators are now required to evaluate teachers by methods that are up to 50% based on their "value-added," as demonstrated at the classroom-level by growth on student achievement data over time. Other related instruments and methods, such as classroom observations and rubrics, have also become common practices in teacher evaluation systems. Such methods are consistent with the neoliberal discourse that has dominated the social and political sphere for the past three decades. Employing a discourse analytic approach that called upon a governmentality framework, the author used a complementary approach to understand how contemporary teacher evaluation polices, practices, and instruments work to discursively (re)define teachers and teacher quality in terms of their market value. For the first part of the analysis, the author collected and analyzed documents and field notes related to the teacher evaluation system at one urban middle school. The analysis included official policy documents, official White House speeches and press releases, evaluation system promotional materials, evaluator training materials, and the like. For the second part of the analysis, she interviewed teachers and their evaluators at the local middle school in order to understand how the participants had embodied the market-based discourse to define themselves as teachers and qualify their practice, quality, and worth accordingly. The findings of the study suggest that teacher evaluation policies, practices, and instruments make possible a variety of techniques, such as numericization, hierarchical surveillance, normalizing judgments, and audit, in order to first make teachers objects of knowledge and then act upon that knowledge to manage teachers' conduct. The author also found that teachers and their evaluators have taken up this discourse in order to think about and act upon themselves as responsibilized subjects. Ultimately, the author argues that while much of the attention related to teacher evaluations has focused on the instruments used to measure the construct of teacher quality, that teacher evaluation instruments work in a mutually constitutive ways to discursively shape the construct of teacher quality. / Dissertation/Thesis / Doctoral Dissertation Educational Leadership and Policy Studies 2014

abstract: This work presents two complementary studies that propose heuristic methods to capture characteristics of data using the ensemble learning method of random forest. The first study is motivated by the problem in education of determining teacher effectiveness in student achievement. Value-added models (VAMs), constructed as linear mixed models, use students’ test scores as outcome variables and teachers’ contributions as random effects to ascribe changes in student performance to the teachers who have taught them. The VAMs teacher score is the empirical best linear unbiased predictor (EBLUP). This approach is limited by the adequacy of the assumed model specification with respect to the unknown underlying model. In that regard, this study proposes alternative ways to rank teacher effects that are not dependent on a given model by introducing two variable importance measures (VIMs), the node-proportion and the covariate-proportion. These VIMs are novel because they take into account the final configuration of the terminal nodes in the constitutive trees in a random forest. In a simulation study, under a variety of conditions, true rankings of teacher effects are compared with estimated rankings obtained using three sources: the newly proposed VIMs, existing VIMs, and EBLUPs from the assumed linear model specification. The newly proposed VIMs outperform all others in various scenarios where the model was misspecified. The second study develops two novel interaction measures. These measures could be used within but are not restricted to the VAM framework. The distribution-based measure is constructed to identify interactions in a general setting where a model specification is not assumed in advance. In turn, the mean-based measure is built to estimate interactions when the model specification is assumed to be linear. Both measures are unique in their construction; they take into account not only the outcome values, but also the internal structure of the trees in a random forest. In a separate simulation study, under a variety of conditions, the proposed measures are found to identify and estimate second-order interactions. / Dissertation/Thesis / Ph.D. Statistics 2013

abstract: Over the past 20 years in the United States (U.S.), teachers have seen a marked shift in how teacher evaluation policies govern the evaluation of their performance. Spurred by federal mandates, teachers have been increasingly held accountable for their students’ academic achievement, most notably through the use of value-added models (VAMs)—a statistically complex tool that aims to isolate and then quantify the effect of teachers on their students’ achievement. This increased focus on accountability ultimately resulted in numerous lawsuits across the U.S. where teachers protested what they felt were unfair evaluations informed by invalid, unreliable, and biased measures—most notably VAMs. While New Mexico’s teacher evaluation system was labeled as a “gold standard” due to its purported ability to objectively and accurately differentiate between effective and ineffective teachers, in 2015, teachers filed suit contesting the fairness and accuracy of their evaluations. Amrein-Beardsley and Geiger’s (revise and resubmit) initial analyses of the state’s teacher evaluation data revealed that the four individual measures comprising teachers’ overall evaluation scores showed evidence of bias, and specifically, teachers who taught in schools with different student body compositions (e.g., special education students, poorer students, gifted students) had significantly different scores than their peers. The purpose of this study was to expand upon these prior analyses by investigating whether those conclusions still held true when controlling for a variety of confounding factors at the school, class, and teacher levels, as such covariates were not included in prior analyses. Results from multiple linear regression analyses indicated that, overall, the measures used to inform New Mexico teachers’ overall evaluation scores still showed evidence of bias by school-level student demographic factors, with VAMs potentially being the most susceptible and classroom observations being the least. This study is especially unique given the juxtaposition of such a highly touted evaluation system also being one where teachers contested its constitutionality. Study findings are important for all education stakeholders to consider, especially as teacher evaluation systems and related policies continue to be transformed. / Dissertation/Thesis / Doctoral Dissertation Educational Policy and Evaluation 2020

Thesis advisor: David Scanlon / There is limited research on the relationship between socially just teaching practices and student achievement. While successful teaching is often defined through test scores, good teaching encompasses the moral elements of teaching (Fenstermacher & Richardson, 2005). This study, building on the work of Mitescu, Cochran-Smith, Pedulla, Cannady, and Jong (2011), is a secondary analysis examining the relationship between socially just teaching practices and student achievement. A subsample of 4th and 5th grade English/language arts (ELA) teachers (n=107) and students (n=2587) was taken from the Measures of Effective Teaching Longitudinal Database (Bill and Melinda Gates Foundation, 2013). Classroom videos were coded using the Teaching for Social Justice Observation Scale (TSJOS) of the Reformed Teaching Observation Protocol-Plus (RTOP+) (Mitescu et al.) to measure socially just teaching practices. Unadjusted linear regression analyses indicated a positive significant correlation between teachers’ mean TSJOS score and the class averages on standardized state ELA exams and the class average on an assessment of higher-order thinking skills. This relationship was also found when the same analysis was conducted on 4th grade classrooms as well as 5th grade classrooms. A hierarchical multiple linear regression found a positive significant relationship between TSJOS scores and student achievement after accounting for location, teacher, and student predictor variables. The relationship between socially just teaching practices and student achievement for subgroups of students is discussed. The study analyzed the significance and magnitude of the relationship between socially just teaching practices after two widely used classroom observation protocols, the Framework for Teaching (FfT) and the Protocol for Language Arts Observation Scale (PLATO), were entered into the model. Teacher mean TSJOS scores were found to explain a significant and unique proportion of the variation in state assessment scores after accounting for average FfT ELA observation scores and teacher average PLATO observation scores, separately. This study adds to the literature on the connection between socially just teaching practices and student achievement, in that it provides compelling evidence that socially just teaching practices are not only related to the good, or moral, side of teaching, but also have a positive and significant relationship with increased student achievement for all students. / Thesis (PhD) — Boston College, 2017. / Submitted to: Boston College. Lynch School of Education. / Discipline: Teacher Education, Special Education, Curriculum and Instruction.

abstract: This study examined the intended and unintended consequences associated with the Education Value-Added Assessment System (EVAAS) as perceived and experienced by teachers in the Houston Independent School District (HISD). To evaluate teacher effectiveness, HISD is using EVAAS for high-stakes consequences more than any other district or state in the country. A large-scale electronic survey was used to investigate the model's reliability and validity; to determine whether teachers used the EVAAS data in formative ways as intended; to gather teachers' opinions on EVAAS's claimed benefits and statements; and to understand the unintended consequences that occurred as a result of EVAAS use in HISD. Mixed methods data collection and analyses were used to present the findings in user-friendly ways, particularly when using the words and experiences of the teachers themselves. Results revealed that the reliability of the EVAAS model produced split and inconsistent results among teacher participants, and teachers indicated that students biased the EVAAS results. The majority of teachers did not report similar EVAAS and principal observation scores, reducing the criterion-related validity of both measures of teacher quality. Teachers revealed discrepancies in the distribution of EVAAS reports, the awareness of trainings offered, and among principals' understanding of EVAAS across the district. This resulted in an underwhelming number of teachers who reportedly used EVAAS data for formative purposes. Teachers disagreed with EVAAS marketing claims, implying the majority did not believe EVAAS worked as intended and promoted. Additionally, many unintended consequences associated with the high-stakes use of EVAAS emerged through teachers' responses, which revealed among others that teachers felt heightened pressure and competition, which reduced morale and collaboration, and encouraged cheating or teaching to the test in attempt to raise EVAAS scores. This study is one of the first to investigate how the EVAAS model works in practice and provides a glimpse of whether value-added models might produce desired outcomes and encourage best teacher practices. This is information of which policymakers, researchers, and districts should be aware and consider when implementing the EVAAS, or any value-added model for teacher evaluation, as many of the reported issues are not specific to the EVAAS model. / Dissertation/Thesis / Ph.D. Educational Leadership and Policy Studies 2012

Submitted by Gabriela Moriconi (gabi.moriconi@gmail.com) on 2012-04-12T17:31:57Z No. of bitstreams: 1 Tese_versão_para_impressão.pdf: 319623 bytes, checksum: 2442d3de6ad04399d9e4a9579d3017cd (MD5) / Approved for entry into archive by Gisele Isaura Hannickel (gisele.hannickel@fgv.br) on 2012-04-12T18:21:23Z (GMT) No. of bitstreams: 1 Tese_versão_para_impressão.pdf: 319623 bytes, checksum: 2442d3de6ad04399d9e4a9579d3017cd (MD5) / Made available in DSpace on 2012-04-12T18:22:22Z (GMT). No. of bitstreams: 1 Tese_versão_para_impressão.pdf: 319623 bytes, checksum: 2442d3de6ad04399d9e4a9579d3017cd (MD5) Previous issue date: 2012-03-21 / Given the importante of teachers to the educational process and availability of data for estimating measures of teacher effectiveness, understood as the ability to produce results in terms of student learning, this study sought to apply a value-added model to estimate and analyse the effectiveness of a sample of teachers who taught fourth grade in São Paulo’s municipal schools. We obtained evidence that variation in teacher effectiveness explains about 9% of the variation in students scores from the analysed sample, less than the variation in student background variables, aorund 15%, but more than variation in school variables, around 5%. For estimating teacher individual effects, results indicate that an increase of one standard-deviation in teacher effects would lead to a minimum increase of 0,062 standard deviation on student scores on Portuguese and 0,049 standard deviation on Mathematics, which represent 2,79 points in Portuguese and 2,20 points in Mathematics. In the analysis of factors associated to the teacher effects were found positive associations between the following variables and teacher effects: time devoted to teaching job out of the school, how often the teacher asks homework, and use of Learning and Support Workbooks. The analysis of reliability and stability of estimated teacher effect measurements indicate that this data have a limited capacity to support recommendations regarding personnel policies, but allow to identify that about 13% of the teachers had different effects from the estimated average. Given this distinction, these teachers are the ideal target to future research on their teaching practices, especially on the factors for which we observed a positive association with the estimated teacher effects. / Dadas a importância dos professores para o processo educacional e a disponibilidade de dados para a estimação de medidas de eficácia dos professores, entendida como a capacidade em produzir resultados em termos da aprendizagem dos alunos, este trabalho buscou aplicar um modelo de valor agregado para estimar e analisar a eficácia de uma amostra dos professores que lecionaram para a 4ª série da rede municipal de ensino de São Paulo no ano de 2010. Obtivemos evidências de que a variação na eficácia dos professores explicaria cerca de 9% da variação nas notas dos alunos da amostra analisada, menos do que a variação nas variáveis de background dos alunos, em torno de 15%, mas mais do que a variação nas variáveis de escola, em torno de 5%. Na estimação dos efeitos professor individuais, os resultados indicam que um aumento de um desvio-padrão nos efeitos professor levaria a um aumento de, no mínimo, 0,062 desvio-padrão na proficiência dos alunos em Língua Portuguesa e 0,049 desvio-padrão em Matemática, que correspondem a, 2,79 pontos em Língua Portuguesa e 2,20 pontos em Matemática. Na análise dos fatores associados aos efeitos professor estimados, foram encontradas associações positivas entre as seguintes variáveis e os efeitos professor: o tempo dedicado ao trabalho pedagógico fora da escola, a freqüência com que o professor passa lição de casa e o uso dos Cadernos de Apoio e Aprendizagem. As análises acerca da confiabilidade e da estabilidade das medidas de efeito professor estimadas indicam que os dados utilizados têm uma capacidade limitada para subsidiar recomendações em relação às políticas de pessoal, porém permitem identificar que cerca de 13% dos professores tiveram efeitos estimados distintos da média. Dada essa distinção, esses professores são o público ideal para pesquisas futuras sobre suas práticas docentes, em especial sobre os fatores para os quais foi observada uma associação positiva com os efeitos professor estimados.

Orientadores: Luiz Carlos de Freitas, Dalton Francisco de Andrade / Tese (doutorado) - Universidade Estadual de Campinas, Faculdade de Educação / Made available in DSpace on 2018-08-24T22:44:15Z (GMT). No. of bitstreams: 1 Dalben_Adilson_D.pdf: 5011742 bytes, checksum: e9c6413b4e6fb98c276dcbdecd13440b (MD5) Previous issue date: 2014 / Resumo: Esta pesquisa é um estudo sobre a eficácia e equidade escolar que tem ganhado atenção especial nos países que usam as avaliações em larga escala a serviço da gestão do sistema educativo. No Brasil, que desde a década de 1990 colocou a avaliação educacional como recurso central em suas políticas educacionais, mas coletando dados seccionais, que são muito frágeis para essa finalidade. Essa fragilidade decorre da alta associação que os fatores extraescolares, sobretudo o nível socioeconômico do aluno, têm sobre as medidas de proficiência. Diante disso, foram usados dados longitudinais e a análise foi feita por meio de modelos lineares hierárquicos. Esta pesquisa teve como objetivo principal desenvolver um modelo estatístico capaz de identificar tais fatores para a realidade brasileira, considerando que a aprendizagem é um processo complexo, isto é, ela é influenciada simultaneamente por múltiplos fatores. Foram desenvolvidos modelos de valor agregado que não só identificam tais variáveis, como também caracterizam sua influência em alunos com distintas proficiências no início de cada período de escolarização. A base de dados utilizada nesses modelos foi fornecida pelo Projeto GERES, que, no período de 2005 a 2008, coletou dados dos mesmos alunos de 1ª a 4ª séries de uma amostra de 312 escolas em cinco grandes cidades brasileiras. Foram medidas as proficiências em Leitura e Matemática de 35.538 alunos e coletadas informações de contexto desses alunos, seus familiares, professores, diretores e escola. Após a redução do grande número de informações disponibilizadas pelo Projeto GERES, feita por meio da Análise Fatorial Exploratória (AFE), as variáveis resultantes foram reorganizadas em três arquivos usados para análise em modelos lineares hierárquicos de três níveis. Os resultados encontrados evidenciam uma significativa instabilidade nos efeitos que as variáveis têm sobre a proficiência, tanto em leitura quanto em matemática. Ao final da pesquisa, são encontrados alguns fatores que influenciam positivamente e negativamente a proficiência em Leitura e Matemática e outros que afetam especificamente cada uma dessas áreas, indicando que podem colaborar para o aumento da eficácia e da equidade das escolas. No entanto, constatam-se também algumas variáveis que têm comportamentos incoerentes com o esperado e outras com comportamentos opostos nas duas áreas. Assim, dos achados das pesquisas, comprova-se que, com base nos dados utilizados, procedimentos metodológicos e modelos estatísticos adotados, os modelos de valor agregado melhoram a confiabilidade das análises em comparação aos modelos que usam dados seccionais, mas ainda são inviáveis como ferramentas para a gestão do sistema educativo, sobretudo para o uso meritocrático de seus resultados. Dessa forma, esta pesquisa corrobora os achados de outras realizadas no âmbito internacional e permite afirmar que a qualidade da modelagem estatística depende da qualidade dos dados que busca modelar, podendo gerar distorções, estabelecer relações inesperadas ou levar a conclusões equivocadas. Em contrapartida, trata-se de recursos que podem ser usados no sistema educativo, fornecendo dados importantes para a orientação das políticas públicas numa perspectiva de avaliação formativa, com vistas ao melhoramento da qualidade de ensino oferecido pelas escolas e à melhor formação dos profissionais docentes e não-docentes que nelas trabalham / Abstract: This research is a study on school effectiveness and equality in Brazil, adding up to a number of other researches that have drawn special attention in countries that use large-scale evaluations at the service of the education system management. In the Brazil has regarded the educational evaluation as a central resource in national education policies, but using cross-sectional data, which are far more fragile for such purpose. This fragility has derived from the great influence that extra-school factors, particularly the students¿ socioeconomic status, exerts on proficiency measures. Longitudinal data was used in the analyses with hierarchical linear models. The main objective of this research was to develop a statistical model to identify such factors in the Brazilian reality, considering that learning is a complex process, i.e. it is simultaneously influenced by multiple factors. Value-added models were developed not only to identify such variables, but also to characterize their influence on students showing different proficiencies at the beginning of every school term. The data base used in those models was provided by the GERES Project, which collected data of the same students from the 1st to the 4th grade from a sample of 312 schools in five Brazilian cities from 2005 to 2008. Proficiencies of 35,538 students were measured, and information about these students¿ context, family, teachers, principals and school were gathered. After the reduction of the great amount of information made available by the GERES Project by means of Exploratory Factor Analysis (EFA), the resulting variables were reorganized in three files used for analysis in three-level hierarchical linear models. The results evidenced significant instability in the effects that the variables have on proficiency both in Reading and in Mathematics. At the end of the research, some factors that influence Reading and Mathematics proficiency either positively or negatively, as well as other factors that specifically affect one of those areas, were found, thus indicating that they may contribute to increased school effectiveness and equality. However, some variables whose behavior was inconsistent with the one expected, and others with opposite behaviors in the two areas were also found. Therefore, from the research findings, based on the data used, the methodological procedures and the statistical models adopted, it has been evidenced that value-added models improve the analysis reliability in comparison with models that use cross-sectional data, but they are still impracticable as tools for education system management, particularly for meritocratic use of their results. Hence, this research has corroborated the findings of other studies carried out over the world and has enabled us to state that the quality of the statistical modeling depends on the quality of data that it attempts to model, and it may generate distortions, establish unexpected relationships or lead to misleading conclusions. On the other hand, these resources may be used in the education system by providing important data for guiding public policies in a educative evaluation perspective, aiming at improving the quality of teaching offered by schools, teachers and other professionals that work in the school setting / Doutorado / Ensino e Práticas Culturais / Doutor em Educação