This applied research in education was undertaken within the context of a school mentoring programme, where my role as a mentor researcher is directed at promoting change in teacher pedagogy consistent with junior-high school educational reforms in Israel. The purpose of this study has been twofold: 1) to confront conflicting issues that exist between the need for change in teacher pedagogy and the resistance felt by many of them toward ‘never ending’ new reforms and: 2). to investigate the impact of the mentoring model (MM) on learning processes that foster teacher empowerment. Assessment of empowerment, as both a process and a product, drew on the teachers' metacognitive development, growing sense of satisfaction and self-efficacy as mediators of their pupils' thinking/learning skills. This study presents a unique approach to teacher empowerment through its theoretical and methodological perspectives. Socio-cultural perspectives serve as an over-arching framework through which various theoretical perspectives for learning and development may be integrated. Action research and discourse analysis were found to be compatible with the researcher’s philosophical approach, whereby educators engage in a collaborative learning process that promotes shared visions and goals. Promoting the characteristics of a learning organisation within the school shed light on ways that can provide teachers with a nurturing environment within the complex dynamics of the school. The detailed account and interpretation of the multi-level reciprocal interactions that occur between teachers, mentor and the school organisation presented in this study is especially significant for understanding multidimensional developmental processes. It illustrates the evolution of inventive methodological tools (such as skills rubrics and discourse analysis techniques), which assume to provide new perspectives for fostering the teachers’ trust in their own judgement when mediating higher order thinking skills. These findings are of particular relevance as contemporary research indicates that teachers often experience difficulties in practicing metacognitive pedagogy.

Thesis advisor: Henry Braun / The Next Generation Science Standards propose a multidimensional model of science learning, comprised of Core Disciplinary Ideas, Science and Engineering Practices, and Crosscutting Concepts (NGSS Lead States, 2013). Accordingly, there is a need for student assessment aligned with the new standards. Creating assessments that validly and reliably measure multidimensional science ability is a challenge for the measurement community (Pellegrino, et al., 2014). Multidimensional assessment tasks may need to go beyond typical item designs of standalone multiple-choice and short-answer items. Furthermore, scoring and modeling of student performance should account for the multidimensionality of the construct. This research contributes to knowledge about best practices for multidimensional science assessment by exploring three areas of interest: 1) item design, 2) scoring rubrics, and 3) measurement models. This study investigated multidimensional scaffolding and response format by comparing alternative item designs on an elementary assessment of matter. Item variations had a different number of item prompts and/or response formats. Observations about student cognition and performance were collected during cognitive interviews and a pilot test. Items were scored using a holistic rubric and a multidimensional rubric, and interrater agreement was examined. Assessment data was scaled with multidimensional scores and holistic scores, using unidimensional and multidimensional Rasch models, and model-data fit was compared. Results showed that scaffolding is associated with more thorough responses, especially among low ability students. Students tended to utilize different cognitive processes to respond to selected-response items and constructed-response items, and were more likely to respond to selected-response arguments. Interrater agreement was highest when the structure of the item aligned with the structure of the scoring rubric. Holistic scores provided similar reliability and precision as multidimensional scores, but item and person fit was poorer. Multidimensional subscales had lower reliability, less precise student estimates than the unidimensional model, and interdimensional correlations were high. However, the multidimensional rubric and model provide nuanced information about student performance and better fit to the response data. Recommendations about optimal combinations of scaffolding, rubric, and measurement models are made for teachers, policymakers, and researchers. / Thesis (PhD) — Boston College, 2018. / Submitted to: Boston College. Lynch School of Education. / Discipline: Educational Research, Measurement and Evaluation.

The purpose of this qualitative study was to extend the understanding of the characteristics of a classroom environment that impact students’ engagement in academics and therefore has the potential to positively impact student achievement scores. Data were collected through content analysis to analyze for reoccurring themes to assess how the characteristics of the classroom environment impact student’s achievement. Ten classrooms within the Kingsport City District were observed and analyzed for this study. Six research questions guided this study, and qualitative data were analyzed for reoccurring themes. Findings from this study suggest that implementing certain characteristics in to the classroom environment can positively impact students’ academic success. The development and construction of classroom environments should include such characteristics as positive discipline, well laid out and organized classrooms, accountable talks, collaborative groups, positive teacher student interaction, and learning targets. As a result of this research a recommendation for practice is that districts support the development of classrooms that would positively impact student’s achievement.

Process skills such as problem solving, critical thinking, information processing, teamwork, and communication are important for student success in their coursework and eventually the workplace, but these skills are not always explicitly taught or assessed in undergraduate courses. These skills should be assessed in order to identify areas for student improvement and because assessment practices can provide clear goals to students. However, my analysis of the current literature suggests that instructors do not have the tools necessary to effectively assess and provide feedback on these skills, particularly in science, technology, engineering, and mathematics (STEM) undergraduate courses. To meet this need of assessing and providing feedback to students, rubrics and other instructional resources have been developed to assess process skills as part of the Enhancing Learning by Improving Process Skills in STEM (ELIPSS) Project. Surveys and interview data indicated that the rubrics were practical for instructors to use to provide feedback to students, represented all relevant aspects of the skills, measured the processes that students used when completing tasks, and could be reliably used by multiple raters. During rubric development, the resources were propagated to the STEM instructor community, and the effectiveness of the propagation methods were examined. The highest rates of adoption resulted from hearing about the rubrics from a colleague or attending a presentation about the rubrics. Additionally, running the ELIPSS workshops and creating the ELIPSS website that people found from searching the internet each led to moderate adoption rates. These results support the idea that a multifaceted propagation strategy may be most effective for researchers who are developing assessment tools. When studying the ways in which STEM instructors were implementing the ELIPSS rubrics, it was found that the instructors each developed different strategies that suited their intended learning outcomes and instructional environments by assessing and providing feedback to students in a variety of ways. Instructors with different class sizes, course levels, online course management systems, and access to teaching assistants all adapted the rubric implementation strategy to fit their unique classroom environments. Multiple instructors reported that they were better able to articulate professional skill expectations to their students through the use of the rubrics. Additionally, they were more aware of how their students interacted with one another in groups after using the interaction rubrics. These results indicate that ELIPSS rubrics can encourage more reflective practice in undergraduate instructors by providing them with more information about their students that can be used to modify their teaching methods. Further work was done to examine how students developed process skills in a first-year chemistry laboratory course. Students in a first-year chemistry laboratory course used the ELIPSS rubrics to assess their own process skills, and they were also assessed by a teaching assistant. Additionally, students reported their understanding of process skills and their perceived improvements over the course of the semester. The results suggest that students understand interpersonal process skills such as teamwork and communication better than they understand cognitive process skills such as critical thinking and information processing. While the evidence further suggests that students improved their process skills, and students reported that they improved their process skills, the students showed inconsistent abilities to self-assess and provide justification for their assessment using rubrics.

The overarching purpose of this study was to describe educators’ beliefs about the evaluation of student writing. The inquiry was guided by the following research questions: (a) what are the differences in the ways in which educators approach evaluating student writing? (b) how do educators evaluate the effectiveness of their evaluation methods for judging the quality of students’ writing samples? and (c) what factors impact the evaluation decisions of educators? The following variables were considered: public and private school settings, evaluation methods, and educators’ beliefs about evaluating writing. In order to gain perspective of the current status of the methods utilized by educators in their evaluation of and response to student writing, it is helpful to observe them during the teaching of writing and to talk with them about their process for evaluating samples of student writing. A mixed methods approach was undertaken during this study and included the collection of questionnaire responses, educator interviews, a classroom observation, and the collection of student writing samples. Interesting points in the findings included the noticeable absence of the notions of validity and reliability in the decision-making process of educators, the apparent impact of educators’ self-efficacies on their selection of evaluation methods, and a focus by educators on writing factors perceived as impacting readability. Implications and future directions for research are discussed.

Esta investigación examinó el uso de rúbricas de ensamblaje, describiendo su evolución a partir de rúbricas de piezas, y estudió cómo afectan a la autoevaluación de los estudiantes. También se valoró la evaluación de los estudiantes por los instructores, encontrando que, mientras que las rúbricas de ensamblaje fueron parcialmente comprendidas y utilizadas de manera eficiente por los estudiantes, éstas fueron usadas con más éxito por los instructores. En esta investigación se han abordado estrategias diseñadas para mejorar la comunicación de la intención de diseño en modelos CAD, acrecentando así su calidad, con directrices dirigidas a evaluar su eficiencia. Es evidente que se necesitan métricas dirigidas hacia la instrucción de la intención de diseño, ya que la intención de diseño transferida a través de modelos CAD puede realizarse en tres etapas con criterios contradictorios que deben ser equilibrados para llegar a la mejor estrategia de modelado. La investigación ha incluido el desarrollo de un método de validación que demuestra que las rúbricas son dispositivos útiles para garantizar una comunicación consistente de la intención de diseño, y son fundamentales no sólo para evaluar, sino también para comunicar las expectativas del instructor. En esta investigación se examinó cómo definir claramente las cualidades de la intención de diseño para permitir una más fácil evaluación de un ensamblaje CAD. Para todas las dimensiones de la rúbrica, se encontró más concordancia y correlación entre instructores que entre instructores y estudiantes. Existe una correlación moderada/fuerte entre los instructores para las dimensiones de la validez, completitud, concisión y claridad, mientras que existe una ligera correlación para las dimensiones de consistencia e intención del diseño. En segundo lugar, las rúbricas también pueden ser descritas como estáticas o dinámicas. Las rúbricas estáticas, existen sólo en papel, no proporcionan retroalimentación inmediata al educando. Las rúbricas dinámicas realizan cálculos que proporcionan observaciones de evaluación inmediatas al usuario. Además, pueden adaptarse a situaciones específicas dependiendo de la capacidad del usuario. Las rúbricas electrónicas son ideales para rúbricas dinámicas, y permiten el uso y desarrollo de rúbricas adaptativas y adaptables, como se describe a continuación. En tercer lugar, las rúbricas deben ser adaptables lo que debería hacerlas fácilmente comprensibles y fáciles de usar, y adaptativas. Las rúbricas de evaluación se usan cuando un experto determina el progreso pedagógico de un educando, mientras que las rúbricas formativas son empleadas por los propios estudiantes, para trazar su progreso e identificar las deficiencias escolares para las que necesitan apoyo. Las rúbricas se deben refinar y mejorar de forma continuada, en un proceso iterativo y colaborativo, hasta que se alcance un acuerdo satisfactorio, tanto entre evaluadores como entre evaluadores y alumnos. Por ello, se desarrollaron mapas de aserciones que ilustran el modo en que la estrategia de expansión-contracción adapta las rúbricas al progreso del aprendiz de CAD, a la vez que ayudan a comprender las diferentes dimensiones de la rúbrica. Basándose en los experimentos con las rúbricas de ensamblajes, es evidente que las pequeñas diferencias entre los instructores sugieren que la rúbrica de ensamblajes propuesta es lo suficientemente sofisticada como para proporcionar una evaluación acumulativa imparcial del desempeño del alumno. En consecuencia, se puede afirmar con confianza que los evaluadores pueden usarse indistintamente sin sacrificar la precisión. Sin embargo, la rúbrica de ensamblaje posee una eficacia finita para producir una autoevaluación formativa de las habilidades de ensamblaje CAD para nuevos alumnos. / This research examined the use of assembly rubrics, described how they evolved from parts rubrics, and studied how they affect student self-evaluation. Instructor assessment of students was also evaluated, finding that while the assembly rubrics were partially understood and effectively used by the students, they were more successfully utilized by the instructors. Strategies designed to improve design intent communication in CAD models, in order to enhance their quality, with guidelines targeted to evaluate efficiency, have been addressed with this research. It is apparent that metrics directed toward the instruction of design intent are needed, since design intent transferred through CAD models can be performed at three stages with competing tradeoffs that must be balanced to arrive at the best modeling strategy. Research included the development of a validation approach that reflects that rubrics are valuable devices to expedite consistent design intent communication, and are vital not only for evaluation, but also for the communication of instructor expectations. This research examined how to clearly define qualities of design intent to enable easier CAD assembly assessment. It has been found that there is more inter-rater agreement and correlation between instructors than between instructors and students, for all rubric dimensions. There is strong to moderate correlation between instructors for the dimensions of validity, completeness, conciseness, and clarity, while slight correlation exists for the dimensions of consistency and design intent. Secondly, rubrics can also be described as being either static or dynamic. Static rubrics, existing in paper form only, do not provide immediate feedback to the learner. Dynamic rubrics perform calculations that provide immediate evaluative observations to the user. Besides, they can be independently adapted to specific situations depending on the capability of the user. Electronic rubrics are ideally suited for dynamic rubrics, and permit the use and development of both adaptable and adaptive rubrics, as described next. Thirdly, rubrics need to be adaptable which should make them easily understood and user-friendly, and adaptive (rubric can change itself, depending on the usage pattern). Evaluative rubrics are used when an expert determines the pedagogical progress of a learner, while formative rubrics are employed by the learners themselves, in order to chart their progress and identify scholastic deficiencies that are in need of remediation. Rubrics must be continually refined and improved, in an iterative, collaborative process, until satisfactory agreement is attained, both between raters, but also between raters and learners. Thus, assertions maps were developed, illustrating how the expand-contract strategy adapts the rubrics to CAD trainee progress, while assisting the understanding of the different rubric dimensions. Based on the assembly rubric experiments, it is apparent that the small differences between instructors suggests that the proposed assemblies rubric is sufficiently sophisticated to furnish an unbiased accumulative assessment of student performance. Accordingly, it can be confidently stated that raters can be used interchangeably without sacrificing accuracy. However, the assembly rubric possesses finite efficacy to produce formative self-evaluation of CAD assembly skills for new learners. / Aquesta investigació examinà l'ús de rúbriques de acoblament, descrivint la seua evolució a partir de rúbriques de peces, i estudià cóm afecten a la autoavaluació dels estudiants. També es va valorar la avaluació dels estudiants per els instructors, trobant que, mentre que les rúbriques de acoblament van ser parcialment compreses i fetes servir de manera eficient per els estudiants, van ser usades amb mes èxit per els instructors. En aquesta investigació s'han abordat estrategues dissenyades per a millorar la comunicació de la intenció de disseny en models CAD, creixentat així la seua qualitat, amb directrius dirigides a avaluar la seua eficiència. Es evident que es necessiten mètriques dirigides cap a la instrucció de la intenció de disseny, ja que la intenció de disseny transferida a través de models CAD pot realitzar-se en tres etapes amb criteris contradictoris que deuen ser equilibrats per a arribar a la millor estratègia de modelatge. La investigació ha inclòs el desenvolupament de un mètode de validació que demostra que las rúbriques son dispositius útils per a garantir una comunicació consistent de la intenció de disseny, i son fonamentals no només per a avaluar, però també per a comunicar les expectatives de l'instructor. En aquesta investigació s'examinà com definir clarament les qualitats de la intenció de disseny per a permetre una mes fàcil avaluació de un acoblament CAD. Per a totes les dimensions de la rúbrica, es va trobar mes concordança i correlació entre instructors que no pas entre instructors i estudiants. Existeix una correlació moderada/fort entre els instructors per a les dimensions de la validesa, completesa, concisió i claredat, mentre que existeix una lleugera correlació per a les dimensions de consistència i intenció del disseny. En segon lloc, les rúbriques també poden ser descrites com estàtiques o dinàmiques. Les rúbriques estàtiques, existeixen sòls en paper, no proporcionen retroalimentació immediata a l'educand. Les rúbriques dinàmiques realitzen càlculs que proporcionen observacions de avaluació immediates al usuari. A mes, poden adaptar-se a situacions específiques dependent de la capacitat de l'usuari. Les rúbriques electròniques son ideals per a rúbriques dinàmiques, i permeten l'ús i desenvolupament de rúbriques adaptatives i adaptables, como es descriu a continuació. En tercer lloc, les rúbriques deuen ser adaptables, el que deuria ferles fàcilment comprensibles i fàcils d'usar, i adaptatives. Les rúbriques d'avaluació se usen quant un expert determina el progrés pedagògic de un educand, mentre que les rúbriques formatives son fetes servir per els propis estudiants, per a traçar el seu progrés i identificar les deficiències escolars para a les que necessiten suport. Les rúbriques es deuen refinar i millorar de forma continuada, en un procés iteratiu i col·laboratori, fins que se arriba a un acord satisfactori, tant entre avaluadors como entre avaluadors i alumnes. Por això, es desenvoluparen mapes de assercions que il·lustren el mode en que la estratègia de expansió-contracció adapta les rúbriques al progres del aprenent de CAD, a la vegada que ajuden a comprendre les diferents dimensions de la rúbrica. Basant-se en els experiments amb les rúbriques de acoblaments, es evident que les xicotetes diferencies entre els instructors suggereixen que la rúbrica de acoblaments proposta es lo suficientment sofisticada com per a proporcionar una avaluació acumulativa imparcial del acompliment de l'alumne. En conseqüència, es pot afirmar amb confiança que els avaluadors poden usar-se indistintament sense sacrificar la precisió. No obstant, la rúbrica de acoblaments posseeix una eficàcia finita per a produir una autoavaluació formativa de les habilitats de acoblament CAD per a nous alumnes. / Otey, JM. (2017). A Contribution to Conveying Quality Criteria in Mechanical CAD Models and Assemblies through Rubrics and Comprehensive Design Intent Quantification [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/94627 / TESIS

The purpose of this study is to compare features of read-alouds from fourth grade classrooms that have been identified as very high-performing using contrasting rubrics for classroom observations, including an English/Language Arts-specific rubric, a general rubric designed for 4th-8th grade classrooms, and a general rubric designed for all subjects and grades. We draw on analytic tools from conversation analysis (Sacks, 1992) and positioning theory (Davies & Harré, 1990) to conduct a micro-analysis of instructional interactions during read-alouds in order to identify similarities and differences among read-alouds from exemplary classrooms across these three different rubric rating systems. In doing so we discuss the implications of each rubric as a guide for shaping specific instructional practices.

No description available.

No description available.

This thesis concerns a longitudinal study of the practice of Live Peer Assessment on two University courses in Computer Science. By Live Peer Assessment I mean a practice of whole-class collective marking using electronic devices of student artefacts demonstrated in a class or lecture theatre with instantaneous aggregated results displayed on screen immediately after each grading decision. This is radically different from historical peer-assessment in universities which has primarily been asynchronous process of marking of students' work by small subsets of the cohort (e.g. 1 student artefact is marked by < 3 fellow students). Live Peer Assessment takes place in public, is marked by (as far as practically possible) the whole cohort, and results are instantaneous. This study observes this practice, first on a level 4 course in E-Media Design where students' main assignment is a multimedia CV (or resume) and secondly on a level 7 course in Multimedia Specification Design and Production where students produce a multimedia information artefact in both prototype and final versions. In both cases, students learned about these assignments from reviewing works done by previous students in Live Peer Evaluation events where they were asked to collectively publicly mark those works according to the same rubrics that the tutors would be using. In this level 4 course, this was used to help students get a better understanding of the marks criteria. In the level 7 course, this goal was also pursued, but was also used for the peer marking of students' own work. Among the major findings of this study are: • In the level 4 course student attainment in the final assessment improved on average by 13% over 4 iterations of the course, with very marked increase among students in the lower percentiles • The effectiveness of Live Peer Assessment in improving student work comes from o Raising the profile of the marking rubric o Establishing a repertoire of example work o Modelling the 'noticing' of salient features (of quality or defect) enabling students to self-monitor more effectively • In the major accepted measure of peer-assessment reliability (correlation between student awarded marks and tutor awarded marks) Live Peer Assessment is superior to traditional peer assessment. That is to say, students mark more like tutors when using Live Peer Assessment • In the second major measure (effect-size) which calculates if students are more strict or generous than tutors, (where the ideal would be no difference), Live Peer Assessment is broadly comparable with traditional peer assessment but this is susceptible to the conditions under which it takes place • The reason for the better greater alignment of student and tutor marks comes from the training sessions but also from the public nature of the marking where individuals can compare their marking practice with that of the rest of the class on a criterion by criterion basis • New measures proposed in this thesis to measure the health of peer assessment events comprise: Krippendorf's Alpha, Magin's Reciprocity Matrix, the median pairwise tutor student marks correlation, the Skewness and Kurtosis of the distribution of pairwise tutor student marking correlations • Recommendations for practice comprise that: o summative peer assessment should not take place under conditions of anonymity but that very light conditions of marking competence should be enforced on student markers (e.g. > 0.2 correlation between individual student marking and that of tutors) o That rubrics can be more suggestive and colloquial in the conditions of Live Peer Assessment because the marking criteria can be instantiated in specific examples of student attainment and therefore the criteria may be less legalistically drafted because a more holistic understanding of quality can be communicated.