The research reported in this thesis investigated issues relating to the use of computer-assisted assessment in Higher Education through the design, implementation and evaluation of a computer-adaptive test (CAT) for the assessment of and provision of feedback to Computer Science undergraduates. The CAT developed for this research unobtrusively monitors the performance of students during a test, and then employs this information to adapt the sequence and level of difficulty of the questions to individual students. The information about each student performance obtained through the CAT is subsequently employed for the automated generation of feedback that is tailored to each individual student. In the first phase of the research, a total of twelve empirical studies were carried out in order to investigate issues related to the adaptive algorithm, stakeholders’ attitude, and validity and reliability of the approach. The CAT approach was found to be valid and reliable, and also effective at tailoring the level of difficulty of the test to the ability of individual students. The two main groups of stakeholders, students and academic staff, both exhibited a positive attitude towards the CAT approach and the user interface. The second phase of the research was concerned with the design, implementation and evaluation of an automated feedback prototype based on the CAT approach. Five empirical studies were conducted in order to assess stakeholders’ attitude towards the automated feedback, and its effectiveness at providing feedback on performance. It was found that both groups of stakeholders exhibited a positive attitude towards the feedback approach. Furthermore, it was found that the approach was effective at identifying the strengths and weaknesses of individual students, and at supporting the adaptive selection of learning resources that meet their educational needs. This work discusses the implications of the use of the CAT approach in Higher Education assessment. In addition, it demonstrates the ways in which the adaptive test generated by the CAT approach can be used to provide students with tailored feedback that is timely and useful.

We explore how to automatically detect specific phrases in audio from noisy, multi-speaker videos using deep neural networks. Specifically, we focus on classroom observation videos that contain a few adult teachers and several small children (< 5 years old). At any point in these videos, multiple people may be talking, shouting, crying, or singing simultaneously. Our goal is to recognize polite speech phrases such as "Good job", "Thank you", "Please", and "You're welcome", as the occurrence of such speech is one of the behavioral markers used in classroom observation coding via the Classroom Assessment Scoring System (CLASS) protocol. Commercial speech recognition services such as Google Cloud Speech are impractical because of data privacy concerns. Therefore, we train and test our own custom models using a combination of publicly available classroom videos from YouTube, as well as a private dataset of real classroom observation videos collected by our colleagues at the University of Virginia. We also crowdsource an additional 1152 recordings of polite speech phrases to augment our training dataset. Our contributions are the following: (1) we design a crowdsourcing task for efficiently labeling speech events in classroom videos, (2) we develop a neural network-based architecture for speech recognition, robust to noise and overlapping speech, and (3) we explore methods to synthesize new and authentic audio data, both to increase the training set size and reduce the class imbalance. Finally, using our trained polite speech detector, (4) we investigate the relationship between polite speech and CLASS scores and enable teachers to visualize their use of polite language.

The demand for new competent programmers is increasing with the ever-growing dependency on technology. The workload for teachers with more and more students creates the need for more automated tools for feedback and grading. While some tools exist that alleviate this to some degree, machine learning presents an interesting avenue for techniques and tools to do this more efficiently. Logical errors are common occurrences within novice code, and therefore a model that could detect these would alleviate the workload for the teachers and be a boon to students. This study aims to explore the performance of the machine learning model code2seq in detecting logical errors. This is explored through an empirical experiment where a data-set consisting of real-world Java code that is modified to contain one specific logical error is used to train, validate and test the code2seq model. The performance of the model is measured using the metrics: accuracy, precision, recall and F1-score. The results of this study show promise for the application of the code2seq model in detecting logical errors and have the potential for real-world use in classrooms.

I denna studie undersöktes värdet av automatisk bedömning av programmeringsuppgifter inom högre utbildning ur studenternas perspektiv. Studien utgick från tidigare litteratur och studier om lärande av programmering för att föreslå en utvecklad pedagogik i användningen av automatbedömning. För att konkretisera automatiserad bedömning så undersökte studien hur det automatiserade bedömningsverktyget Kattis används inom ramen för en datalogikurs på Kungliga Tekniska Högskolan. För att förstå studenternas upplevelse av Kattis genomfördes kvalitativa intervjuer med studenter som läst eller läser datalogikursen. Därefter genomfördes en tematisk analys av intervjuerna för att identifiera de centrala delarna i hur studenter använder och upplever Kattis. Dessa delar analyserades sedan utifrån tidigare studier för identifiera vad som bör ändras för att förbättra verktyget ur ett pedagogiskt perspektiv. Däribland identifierades behovet av att informera studenterna och lärare om användning av Kattis och avläsning av Kattis återkoppling. Utifrån denna analys samt diskussion med lärare som använder Kattis i sin undervisning utformades en arbetsguide med metoder och information för lärare och studenter. / This study examines the value of automated assessment of programming assignments in higher education from a student perspective. The study used earlier literature and studies about learning programming and pedagogy to suggest a developed pedagogy in the use of automated assessment. To concretize automated assessment the study explored how the automated assessment tool Kattis have been used in a course in computer science at KTH, Royal Institute of Technology. To understand students’ experience of Kattis, qualitative interviews was conducted withstudents who are taking or have taken the course in computer science. The interviews were analyzed using thematic analysis to identify central aspects of how students use and experience Kattis. These aspects were then analyzed based on earlier scientific studies to identify what should be changed in the use of the tool from a pedagogical perspective. Among this there is a need to inform students and teachers about the use of Kattis and how to read the feedback from Kattis. Based on this analysis and discussion with teachers who use Kattis in their courses, a work guide for teachers was designed with methods and information for teachers and students.