Grades serve as one of the primary indicators of student learning, directing subsequent actions for students, instructors, and administrators, alike. Therefore, grade validity—that is, the extent to which grades communicate a meaningful and credible representation of what they purport to measure—is of utmost importance. However, a grade cannot be valid if one cannot trust that it will consistently and reliably result in the same value, regardless of who makes a measure or when they make it. Unfortunately, such reliability becomes increasingly challenging to achieve with larger class sizes, especially when utilizing multiple evaluators, as is often the case with mandatory introductory courses at large universities. Reliability suffers further when evaluating open-ended tasks, as are prevalent in authentic, high-quality engineering coursework.<div><br></div><div>This study explores grading reliability in the context of a large, multi-section engineering course. Recognizing the number of people involved and the plethora of activities that affect grading outcomes, the study adopts a systems approach to conduct a human reliability analysis using the Functional Resonance Analysis Method. Through this method, a collection of data sources, including course materials and observational interviews with undergraduate teaching assistant graders, are synthesized to produce a general model for how actions vary and affect subsequent actions within the system under study. Using a course assignment and student responses, the model shows how differences in contextual variables affect expected actions within the system. Next, the model is applied to each of the observational interviews with undergraduate teaching assistants to demonstrate how these actions occur in practice and to compare graders to one another and with expected behaviors. These results are further related to the agreement in system outcomes, or grades, assigned by each grader to guide analysis of how actions within the system affect its outcome.<br></div><div><br></div><div>The results of this study connect and elaborate upon previous models of grader cognition by analyzing the phenomenon in engineering, a previously unexplored context. The model presented can be easily generalized and adapted to smaller systems with fewer actors to understand sources of variability and potential threats to outcome reliability. The analysis of observed outcome instantiations guides a set of recommendations for minimizing grading variability.<br></div>

Looking into and Evaluation Quallity, Efficiency and Effects of Further Education A thesis centers upon description and comparison of methods that gain a feedback in the field of the quality of further education. It considers both institutions providing further education and institutions/organisations (e.g. schools, offices of public administration) representing source of participants of further education courses. A theoretical part the thesis compares well known methods reporting on the quality and effectivity of further education in a view of both a provider and a customer. A research aims for an analysis of gaining a feedback in the field of further education from 50 providing institutions and 340 institutions/organizations in a role of a customer. Those 340 institutions/organizations include 32 offices of public administrative and another 308 publicly figuring organizations (e.g. schools). This part of the thesis also presents positives and negatives of chosen methods. A conclusion of the thesis summarizes findings and provides recommendations for both surveyed groups.

The desire to produce a learning environment which promotes student motivation, collaboration, and higher order thinking is common within the higher education system of today. Such learning environments also have the ability to address challenges’ Mechanical Engineering Technology (MET) students face entering the workforce. Through the vertical and horizontal integration of courses, this research presents how a scaffolded learning environment with a centralized theme of energy can increase motivation and conceptual retention within students. The integration of courses allows students to systematically translate their competency of concepts between energy based courses through experiential learning. The goal of this work is to develop a competency based learning model where students earn a professionally recognizable credential. The credential is earned through demonstrating their mastery of industry desired skills at a level that goes above and beyond the stock curriculum. The result is a more continuous curriculum that enhances multi-disciplinary problem solving while better preparing MET students for the workforce.

Increasing enrollment of students and declining availability of qualified and experienced faculty are leading to increased assessment loads of the existing faculty. Moreover, the assessment techniques are changing drastically due to the ever-increasing demand of new knowledge and abilities from the students. The tools offered by information technology can now be effectively used in enhancing the productivity of a teacher. This thesis proposes a mechanism for creating both summative and formative assessment instruments for a course in an engineering program. The assessment instruments will vary widely in nature depending on the subject. With increasing prevalence of digital devices in all walks of life a first level knowledge of digital systems is considered necessary for all engineers especially under electrical and computer engineering curricula. The first level course ‘Basics of Digital Systems’ is chosen for developing a framework of computer assisted evaluation. Creation of assessment instruments is best done in the context of an instructional system design (ISD) model. ADDIE, a generic model is chosen for the study. Bloom’s classification of levels of cognition, Vincenti’s categorization of engineering knowledge, and ‘Gronlund 2-level’ method for writing the learning objectives are integrated to create a ‘Bloom-Vincenti–Gronlund’(BVG)framework for preparing the learning objectives/assessment instruments. Developing tools for evaluation of performance of students in the assessment tests requires consideration of many issues: analysis of problems and their solution methods, errors normally committed by students, grading preferences of the instructor and feedback to students. A set of tools are developed that are able to evaluate the truth tables, state tables, excitation tables, timing diagram and VHDL codes. The developed tools are validated. The submission of the assignment and the integration of all the tools for evaluation will be more effective if they can be integrated in a learning management system (LMS). ‘MOODLE’, an open source LMS, is identified for the integration of the tools. The developed tools execute the files submitted by the students, evaluate them, and provide feedback to the students. In summary, the thesis addressed some key issues related to “assessment and evaluation of students’ performance” and proposed an integrated computer assisted system for the evaluation of students’ performance in the course ‘Basics of Digital Systems’.

<div>Service-learning is an organized volunteer activity in which learners serve the community while utilizing and enhancing their own skills, thus benefiting both the learners and the community. Studies have shown that students gain various benefits from participating in a service-learning activity, especially in their academic skills and civic growth through continued reflections (Eyler, Giles, & Braxton, 1997; Eyler & Giles, 1999; Billig, 2000; Grassi et al., 2004; Steinberg, Bringle, & Williams, 2010), often increasing their motivation to learn the related subject (Steinberg et al., 2010). Service-learning has been implemented in foreign language courses in the United States, especially Spanish (Barreneche & Ramos-Flores, 2013). However, service-learning literature on Japanese as a foreign language is limited.</div><div>The researcher founded a service-learning program in the Japanese language. In the program, the university students enrolled in intermediate- or higher-level Japanese courses help Japanese children with their schoolwork as volunteer tutors. The researcher conducted a qualitative case study on four of the student-tutors to examine the program's potential benefits to maintain and enhance the student-tutors' various motivations toward learning Japanese. The Volunteer Functions Inventory (VFI) (Clary, Snyder, & Ridge, 1998) was used as an analysis scheme, which reports six most commonly found functions, or varying motivations, for participating in a volunteer activity. The student-tutors indicated five out of the six VFI functions, showing a connection between their service-learning experience and their personal growth. They built strong connections with the Japanese community and kept their motivation to improve their Japanese skills to better help the children. It is hoped that the present research will contribute to providing an example of Japanese service-learning in the U.S.</div>

<p>Internationalization of higher education is a collaborative responsibility academic and non-academic programs share to facilitate the integration of various student populations within the broader culture of the university. My dissertation project links First Year Writing (FYW) classes of domestic and international students to promote and evaluate their intercultural competence development. My research questions explore the use of reflective writing as a genre for formative assessment in the writing classroom and investigate the data it provides about students’ continuous learning. My research methodology combines qualitative analysis of reflective writing and quantitative analysis of intercultural competence development. Participants come from four sections of FYW courses spanning two semesters – Spring 2016 and Fall 2017. I collected reflective writing data from four embedded reflective journals and a final reflective essay assigned to students in each section. Using a grounded scheme, I applied thematic coding analysis of reflective writing and traced frequencies of codes. I also mapped students’ reflections onto the Developmental Model of Intercultural Sensitivity (DMIS; Bennett, 1993). Results from both coding methods contextualize and interpret students’ development in both intercultural competence and writing skills. I also share pedagogical, assessment, and administrative implications for more effective teaching of reflective writing and better continuous assessment of intercultural competence skills within the context of the linked course model curriculum. </p> <p> </p>

<p>Content about learners with gifts and talents is not necessarily a part of most teacher education programs. Without high quality training and professional development opportunities, preservice and inservice teachers are left with no tools to identify and serve the students with gifts and talents. However, adding more content is not enough. The successful translation of training and professional development into effective practice depends on understanding teacher motivation, debunking misconceptions, building adequate knowledge base, and building teacher identity. I adopted several theoretical perspectives in this study: teacher identity formation (Gardner & Kaplan, 2018), Teacher Efficacy (Tschannen-Moran et al., 1998), Teacher Goal Orientations (Butler, 2007), beliefs about gifted learners and gifted education (Gagne & Nadeau, 1991; McCoach & Siegle, 2007), desire to teach (Watt & Richardson, 2007). My participants were 236 preservice teachers who desire to teach learners with gifts and talents and inservice teachers in gifted education.</p><p>The objectives of this mixed-methods investigation were: (1) identifying the differences between preservice and inservice teachers in measures of identity, beliefs, motivation, and desire to teach learners with gifts and talents, (2) modeling the structural relationships among dimensions of identity, motivation, beliefs, and desire to teach, and (3) understanding how participants experiences and perceptions inform their identity, motivation, and belief systems. I used a combination of Multivariate Analysis of Variance (MANOVA), Structural Equation Modeling (SEM) and qualitative thematic analysis to answer my research questions.</p><p>Findings revealed inservice teachers (n=155) have high levels of relational goals, instructional efficacy, positive beliefs, and teacher identity, while preservice teachers (n=81) have high levels of intrinsic motivation and social value for gifted education. SEM showed that teacher identity, mastery goals, influenced positive beliefs; teacher identity was influenced by efficacy, mastery and relational goals. The strongest predictors of desire to teach learners with gifts and talents were teacher identity, teacher efficacy, and relational goals. Qualitative findings indicated that self-perceptions as gifted played a meaningful role in participants deciding to become teachers, understanding the needs of gifted learners, and advocating for gifted education.</p>

<p>Innovations in technology are changing not only everyday life for many individuals around the world but are also influencing the expansion of online learning opportunities at an accelerated rate (Collins & Halverson, 2<a></a>018; Mah, 2016). Online learning platforms allow for scalability, flexibility, greater global access, and innovative and new ways to deliver education (Goodman, Melkers, & Pallais, 2019; Kizilcec et al., 2019). Enrollments in online learning programs and opportunities have seen significant growth in recent years (Seaman, Allen, & Seaman, 2018; U.S. Department of Education, 2018) with continued and steady growth expected into the future. The ubiquity and newness of new online learning formats present a challenge in linking research and practice. Through three separate academic papers, the following dissertation discusses and considers key questions and topics with regards to the use of digital badges and Massive Open Online Courses (MOOCs), two types of emerging online innovations and platforms, and aspects of their efficacy. The three papers respectively 1) identify and discuss the theoretical and empirical foundations digital badges use in specific learners groups by reviewing current literature; 2) highlight the application of a use case in which digital badges have been implemented as a means to offer training; and 3) explore the perceptions of MOOC instructors toward quality learning in their courses in a case study. Conclusions are drawn and solutions as well as potential future directions for research and practice of discussed. </p>

<p>In an era of widespread automation—from grocery store self-checkout machines to self-driving cars—it is not outrageous to wonder: can teachers be automated? And more specifically, can automated computer teachers instruct students how to write? Automated computer programs have long been used in summative writing evaluation efforts, such as scoring standardized essay exams, ranking placement essays, or facilitating programmatic outcomes assessments. However, new claims about automated writing evaluation’s (AWE) formative educational potential mark a significant shift. My project questions the effectiveness of using AWE technology for formative educational efforts such as improving and teaching writing. Taken seriously, these efforts portend a future embrace of semi, or even fully, automated writing classes, an unprecedented development in writing pedagogy.</p><p>Supported by a summer-long grant from the Purdue Research Foundation, I conducted a small-<i>n </i>quasi-experiment to test claims by online college tutoring site Chegg.com that its EasyBib Plus AWE tool can improve both writing and writers. The experiment involved four college English instructors reading pairs of essays comprising one AWE-treated and untreated version per pair. Using a comparative judgment model, a rubric-free method of writing assessment based on Thurstone’s law, raters read and designated one of each pair “better.” Across four raters and 160 essays, I found that AWE-treated essays were designated better only 30% of the time (95% confidence interval: 20-40%), a statistically significant difference from the null hypothesis of 50%. The results suggest that Chegg’s EasyBib Plus tool offers no discernible improvement to student writing, and potentially even worsens it.</p><p>Finally, I analyze Chegg’s recent partnership with the Purdue Writing Lab and Online Writing Lab (OWL). The Purdue-Chegg partnership offers a useful test case for anticipating the effects of higher education’s embrace of automated educational technology going forward. Drawing on the history of writing assessment and the results of the experiment, I argue against using AWE for formative writing instruction. In an era of growing automation, I maintain that a human-centered pedagogy remains one of the most durable, important, effective, and transformative ingredients of a quality education.</p>

<p>External representations (ERs) used in science education are multimodal ensembles consisting of design elements to convey educational meanings to the audience. As an example of a dynamic ER, an animation presenting its content features (i.e., scientific concepts) via varying the feature’s depiction over time. A production team invited the dissertation author to inspect their creation of a biochemistry animation about the role of the actin cytoskeleton in cell motility and the animation’s implication on learning. To address this, the author developed a four-step methodology entitled the Multimodal Variation Analysis of Dynamic External Representations (MVADER) that deconstructs the animation’s content and design to inspect how each content feature is conveyed via the animation’s design elements.</p><p><br></p><p> </p><p>This dissertation research investigated the actin animation’s educational value and the MVADER’s utility in animation evaluation. The research design was guided by descriptive case study methodology and an integrated framework consisting of the variation theory, multimodal analysis, and visual analytics. As stated above, the animation was analyzed using MVADER. The development of the actin animation and the content features the production team members intended to convey via the animation were studied by analyzing the communication records between the members, observing the team meetings, and interviewing the members individually. Furthermore, students’ learning experiences from watching the animation were examined via semi-structured interviews coupled with post- storyboarding. Moreover, the instructions of MVADER and its applications in studying the actin animation were reviewed to determine the MVADER’s usefulness as an animation evaluation tool.</p><p><br></p><p> </p><p>Findings of this research indicate that the three educators in the production team intended the actin animation to convey forty-three content features to the undergraduate biology students. At least 50% of the student who participated in this thesis learned thirty-five of these forty-three (> 80%) features. Evidence suggests that the animation’s effectiveness to convey its features was associated with the features’ depiction time, the number of identified design elements applied to depict the features, and the features’ variation of depiction over time.</p><p><br></p><p>Additionally, one-third of the student participants made similar mistakes regarding two content features after watching the actin animation: the F-actin elongation and the F-actin crosslink structure in lamellipodia. The analysis reveals the animation’s potential design flaws that might have contributed to these common misconceptions. Furthermore, two disruptors to the creation process and the educational value of the actin animation were identified: the vagueness of the learning goals and the designer’s placement of the animation’s beauty over its reach to the learning goals. The vagueness of the learning goals hampered the narration scripting process. On the other hand, the designer’s prioritization of the animation’s aesthetic led to the inclusion of a “beauty shot” in the animation that caused students’ confusion.</p><p><br></p><p> </p><p>MVADER was used to examine the content, design, and their relationships in the actin animation at multiple aspects and granularities. The result of MVADER was compared with the students’ learning outcomes from watching the animation to identify the characteristics of content’s depiction that were constructive and disruptive to learning. These findings led to several practical recommendations to teach using the actin animation and create educational ERs.</p><p><br></p><p> </p><p>To conclude, this dissertation discloses the connections between the creation process, the content and design, and the educational implication of a biochemistry animation. It also introduces MVADER as a novel ER analysis tool to the education research and visualization communities. MVADER can be applied in various formats of static and dynamic ERs and beyond the disciplines of biology and chemistry.</p>