OpenDSA is a collection of open source tutorials for teaching data structures and algorithms. It was created with the goals of visualizing complex, abstract topics; increasing the amount of practice material available to students; and providing immediate feedback and incremental assessment. In this thesis, I first describe aspects of the OpenDSA architecture relevant to collecting user interaction data. I then present an analysis of the interaction log data gathered from three classes during Spring 2013. The analysis focuses on determining the time distribution of student activity, determining the time required for assignment completion, and exploring \credit-seeking" behaviors and behavior related to non-required exercises. We identified clusters of students based on when they completed exercises, verified the reliability of estimated time requirements for exercises, provided evidence that a majority of students do not read the text, discovered a measurement that could be used to identify exercises that require additional development, and found evidence that students complete exercises after obtaining credit. Furthermore, we determined that slideshow usage was fairly high (even when credit was not ordered), and skipping to the end of slideshows was more common when credit was offered but also occurred when it was not. / Master of Science

Automatic assessment tools are increasingly utilized in undergraduate programming courses to evaluate software solutions, streamlining the grading process for both students and professors. In spite of their benefits of speed and convenience, such online systems for providing instant feedback have the tendency to draw attention to performance-based outcomes while failing to reliably recognize the effort and hard work a student puts into a solution. For the many struggling students who are new to programming, this type of objective feedback can be discouraging and may decrease their motivation to stay engaged towards success. To address this issue, this paper explores strategies for more effectively recognizing student progress on programming assignments and identifying small tasks for students to complete that will steer them in the right direction. Further, this paper will discuss a gamification approach for adding ``Daily Missions" to Web-CAT, the Web-based Center for Automated Testing. From an evaluation of results gathered from student experiences with this latest version of Web-CAT at Virginia Tech, this paper will highlight the valuable potential gamification has in boosting student engagement in computer science learning. / Master of Science / In college computer science courses today, many students submit their assignments and receive instant feedback through online websites. Although they streamline the grading process for both students and professors, these systems have the tendency to draw attention to performance-based outcomes while failing to reliably recognize the effort and hard work a student puts into a solution. For the many struggling students who are new to programming, this type of objective feedback can be discouraging and may decrease their motivation to stay engaged towards success. To address this issue, this paper explores strategies for more effectively recognizing student progress on programming assignments and identifying small tasks for students to complete that will steer them in the right direction. One strategy in particular is called gamification, which refers to giving game-like attributes to a non-game system. This paper will discuss a gamification approach for adding “Daily Missions” to WebCAT, the Web-based Center for Automated Testing. From an evaluation of results gathered from student experiences with this latest version of Web-CAT at Virginia Tech, this paper will highlight the valuable potential gamification has in boosting student engagement in computer science learning.

L’objectif de cette thèse est de modéliser l’interlangue prosodique des apprenants francophones de l’anglais afin de pouvoir élaborer des critères utilisables pour une évaluation semi-automatique de leur niveau prosodique. Le domaine évaluatif requiert la plus grande rigueur dans la mise en place de ses critères pour aboutir à la validité, la fiabilité, la faisabilité et l’équité maximales, alors que la prosodie anglaise de la langue cible se caractérise par son extrême variabilité. Aussi, peu d’études se sont engagées dans l’évaluation de la prosodie, qui représente une réelle gageure. Pour relever ce défi, une stratégie particulière a été mise en place pour élaborer une méthodologie permettant d’atteindre l’objectif fixé, en lecture.L’approche choisie repose sur la symbiose permanente qu’entretient la prosodie avec le monde dans lequel évolue le locuteur. Cette méthodologie, ou « profilage », est destinée à sélectionner par inclusion ou exclusion les éléments analysés tant au niveau perceptif qu’acoustique. Le profilage des réalisations sur l’axe syntagmatique permet de sélectionner les locuteurs natifs servant de modèles, et celui basé sur le phénomène d’emphase rend possible un ciblage de leurs réalisations les plus pertinentes à modéliser sur l’axe paradigmatique. Conformément à cette méthodologie d’investigation nouvelle et aux résultats perceptifs et acoustiques concordants pour la langue cible, les réalisations des apprenants francophones du corpus Longdale-Charliphonia sont analysés acoustiquement. Le classement automatique à partir des variables prosodiques (acoustiques et perceptives) est confronté à celui d’experts évaluant par perception classique.Les travaux de cette thèse aboutissent essentiellement à : Une modélisation de la prosodie anglaise non native par grilles évaluatives critériées s’appuyant sur critères distinctifs natifs et non natifs issus de variables temporelles (vitesse d’élocution avec ou sans pauses), de registre et de mélodie, ainsi que de rythme, À partir de ces variables, une évaluation semi-automatisée de 15 apprenants représentatifs du corpus par classement et notation, une correspondance des résultats de l’évaluation traditionnelle avec celle semi-automatique évoluant entre 56,83% et 59,74% dans une catégorisation des apprenants en 3 niveaux de maîtrise, en fonction du profilage d’experts évaluateurs. / The aim of our study is to modelise the prosodic interlanguage of Francophone learners of English in order to provide useful criteria for a semi-automatic assessment of their prosodic level in English. Learner assessment is a field that requires to be very rigorous and fair when setting up criteria that ensure validity, reliability, feasibility and equality, whereas English prosody is highly variable. Hence, few studies have carried out research in assessing prosody because it represents a real challenge. To address this issue, a specific strategy has been devised to elaborate a methodology that would ensure assessing a reading task successfully.The approach relies upon the constant symbiosis between prosody and a speaker’s subjective response to their environment. Our methodology, also known as « profiling », first aims at selecting relevant native perceived and acoustic prosodic features that will optimize assessment criteria by using their degree of emphasis and creating speakers’ prosodic profiles. Then, using the Longdale-Charliphonia corpus, the learner's productions are analysed acoustically. The automatic classification of the learners based on acoustic or perception prosodic variables is then submitted to expert aural assessment which assesses the learner evaluation criteria.This study achieves: A modelisation of non-native English prosody based on assessment grids that rely upon features of both native and non-native speakers of English, namely, speech rate – with or without the inclusion of pauses, register, melody and rhythm,A semi-automatic evaluation of 15 representative learners based on the above modelisation – ranking and marking,A comparison of the semi-automatic results with those of experts' auditory assessment; correspondence between the two varies from 56.83% to 59.74% when categorising the learners into three prosodic proficiency groups.

Research in team science suggests strategies for addressing difficulties that groups face when working together. This dissertation examines how student teams work in project based learning (PBL) environments, with the goal of creating strategies and technology to improve collaboration. The challenge of working in such a group is that the members frequently come from different backgrounds and thus have different ideas on how to accomplish a project. In these groups, teamwork and production of successful solutions depends on whether members consider each other’s dissimilar perspectives. However, the lack of a shared history means that members may have difficulty in taking the time to share and build knowledge collectively. The ultimate goal of my research is to design strategies and technology to improve the inner workings of PBL groups so that they will learn from each other and produce successful outcomes in collaborative settings. The field of computer supported collaborative learning has made much progress on designing, implementing, and evaluating environments that support project based learning. However, most existing research concerns students rather than instructors. Therefore, in my initial research, I explore the needs of the instructors in conducting student assessments (studies one, two). These studies identify five different group processes that are of importance from the instructors’ perspective. My subsequent research focuses on one of them, namely the process of knowledge co-construction, which is a process that instructors have significant difficulty in assessing. In order to support the assessment of the knowledge co-construction process, my research has progressed along two axes: (a) identifying conditions that support the knowledge co-construction process and its relationship to learning and knowledge transfer (studies three, four, and five), and (b) automatically monitoring the knowledge co-construction process using natural language processing and machine learning (studies six ~ nine). Studies five and eight look at a specific type of knowledge co-construction process called the idea co-construction process (ICC). ICC is the process of taking up, transforming, or otherwise building on an idea expressed earlier in a conversation. I argue that ICC is essential for groups to function well in terms of knowledge sharing and perspective taking.

Nearly seven hundreds of students each year apply for informatics' studies at University of economics, Prague (UEP). At the beginning, each one must complete two courses related to programming - Introduction to Programming and Fundamentals of Software Engineering. Both courses produce a lot of outputs -- simpler home works or more complex semestral projects -- which must be assessed by the teachers. Especially assessment of early home works is laboring and routine which makes it perfect candidate for automation. The goal of this work is to suggest the best suitable way how to implement this automation at UEP. Since mass courses make troubles universities all around the world there is already a lot of existing solutions how to automate evaluation of programming courses. First two chapters summarize the effort to find the most interesting ones and present them to the Czech audience. Because existing solutions are really sophisticated but does not provide white box tests support needed at UEP the rest of this thesis is not about creating own new automatic assessment system but about creating a middleware which can handle the problem mentioned. The middleware called Duckapter makes usage of Java reflection API easier and is downloadable from project website http://code.google.com/p/duckapter. Using them, the teachers can provide white box tests inside the black box (unit) tests which are supported by the most of the current automatic assessment systems.

Automated short-answer marking cannot 'guarantee' 100% agreement between the marks generated by a software system and the marks produced separately by a human. This problem has prevented automated marking systems from being used in high-stake short-answer marking. Given this limitation, can an automated short-answer marking system have any practical application? This thesis describes how an automated short-answer marking system, called IndusMarker, can be effectively used to improve learning and teaching.The design and evaluation of IndusMarker are also presented in the thesis. IndusMarker is designed for factual answers where there is a clear criterion for answers being right or wrong. The system is based on structure matching, i.e. matching a pre-specified structure, developed via a purpose-built structure editor, with the content of the student's answer text. An examiner specifies the required structure of an answer in a simple purpose-designed language called Question Answer Markup Language (QAML). The structure editor ensures that users construct correct required structures (with respect to QAML's syntax and informal semantics) in a form that is suitable for accurate automated marking.

O solfejo, em seu conceito mais amplo, é uma técnica usual no processo de ensinoaprendizagem musical, o qual envolve a realização vocal de melodias, considerando as alturas e as durações dos sons musicais registrados em partitura, devidamente associado à marcação de compassos por intermédio de gestos que definem a estrutura métrica e o andamento da peça musical. Este trabalho apresenta uma abordagem audiovisual para avaliação automática dessa prática de estudo pertinente à leitura e à estruturação musicais. O sistema proposto é dividido em três partes. A primeira efetua o reconhecimento visual dos gestos de marcação de compassos realizados pela mão, por intermédio de um classificador probabilístico. Um processo de alinhamento temporal garante o reconhecimento dos padrões de movimento mesmo em casos com variação de andamento, permitindo também a avaliação da precisão rítmica do aluno, quando comparado com um referencial metronômico. A segunda parte deste sistema obtém a transcrição melódica do canto a partir da análise do respectivo sinal de áudio. Os fragmentos melódicos detectados são agrupados e mapeados em relação às notas da partitura do exercício de solfejo, permitindo uma avaliação direta nota-a-nota da performance do canto. Por fim, a terceira parte do sistema proposto faz a integração entre o gesto de marcação de compassos e a transcrição melódica. Nesse caso, o gesto atua como um metrônomo, controlando o fluxo temporal. Assim, a avaliação nota-a-nota do solfejo pode ser empregada também em casos onde exista grande variação no andamento da peça. Tanto o processo de avaliação do gesto de marcação de compassos quanto a avaliação do canto são obtidos por intermédio de um classificador Bayesiano gerado a partir de avaliações reais, feitas por especialistas em música. Dessa forma, o sistema desenvolvido efetua o mapeamento advindo da opinião de especialistas humanos em um sistema de avaliação automática de solfejo executado por máquina, que é capaz de identificar as notas musicais cantadas pelo aluno em cada instante métrico determinado, devidamente conduzido pelo gesto, sem a necessidade de sincronização por um metrônomo ou manutenção de um andamento fixo. / Solfége is a general technique used in the music learning process, which involves the vocal performance of melodies, regarding the time and duration of musical sounds as specified in the music score, properly associated the meter-mimicking performed by the hand movement. This thesis presents an audiovisual approach for automatic assessment of this relevant musical study practice. The proposed system is divided into three parts. First, a probabilistic classifier recognizes the musical metric patterns drawn by the hand movement. A time alignment process assures the proper recognition of the movement patterns even in cases where there are changes in the musical tempo. Also, this process allows to estimate the accuracy of the rhythmic performance. In the second part of this work, audio analysis is applied to achieve the melodic transcription of the sung notes. The detected melodic fragments are then grouped and mapped into single notes, which are connected to their related notes on the music score of the solfège exercise. This mapping procedure allows the direct assessment (note by note) of the singing performance, even if there are slight discrepancies between the transcribed notes and the music score. Finally, the last part of the proposed system combines the gesture of meter-mimicking (video information) with the melodic transcription (audio information), where the hand movement works as a metronome, controlling the time flow (tempo) of the musical piece. Thus, the meter-mimicking is used to align the music score (ground truth) with the sung melody, allowing the assessment even in time dynamic scenarios. Both meter-mimicking and sung notes are evaluated by a set of Bayesian classifiers that were generated from real evaluations done by experts listeners. In this way, the developed model performs the mapping of the point of view from human experts into an automatic system which is able to make the solfege assessment, regarding the pitch, onset and duration of the music notes, without the need for external synchronization with a metronome or the maintenance of a fixed tempo.

O solfejo, em seu conceito mais amplo, é uma técnica usual no processo de ensinoaprendizagem musical, o qual envolve a realização vocal de melodias, considerando as alturas e as durações dos sons musicais registrados em partitura, devidamente associado à marcação de compassos por intermédio de gestos que definem a estrutura métrica e o andamento da peça musical. Este trabalho apresenta uma abordagem audiovisual para avaliação automática dessa prática de estudo pertinente à leitura e à estruturação musicais. O sistema proposto é dividido em três partes. A primeira efetua o reconhecimento visual dos gestos de marcação de compassos realizados pela mão, por intermédio de um classificador probabilístico. Um processo de alinhamento temporal garante o reconhecimento dos padrões de movimento mesmo em casos com variação de andamento, permitindo também a avaliação da precisão rítmica do aluno, quando comparado com um referencial metronômico. A segunda parte deste sistema obtém a transcrição melódica do canto a partir da análise do respectivo sinal de áudio. Os fragmentos melódicos detectados são agrupados e mapeados em relação às notas da partitura do exercício de solfejo, permitindo uma avaliação direta nota-a-nota da performance do canto. Por fim, a terceira parte do sistema proposto faz a integração entre o gesto de marcação de compassos e a transcrição melódica. Nesse caso, o gesto atua como um metrônomo, controlando o fluxo temporal. Assim, a avaliação nota-a-nota do solfejo pode ser empregada também em casos onde exista grande variação no andamento da peça. Tanto o processo de avaliação do gesto de marcação de compassos quanto a avaliação do canto são obtidos por intermédio de um classificador Bayesiano gerado a partir de avaliações reais, feitas por especialistas em música. Dessa forma, o sistema desenvolvido efetua o mapeamento advindo da opinião de especialistas humanos em um sistema de avaliação automática de solfejo executado por máquina, que é capaz de identificar as notas musicais cantadas pelo aluno em cada instante métrico determinado, devidamente conduzido pelo gesto, sem a necessidade de sincronização por um metrônomo ou manutenção de um andamento fixo. / Solfége is a general technique used in the music learning process, which involves the vocal performance of melodies, regarding the time and duration of musical sounds as specified in the music score, properly associated the meter-mimicking performed by the hand movement. This thesis presents an audiovisual approach for automatic assessment of this relevant musical study practice. The proposed system is divided into three parts. First, a probabilistic classifier recognizes the musical metric patterns drawn by the hand movement. A time alignment process assures the proper recognition of the movement patterns even in cases where there are changes in the musical tempo. Also, this process allows to estimate the accuracy of the rhythmic performance. In the second part of this work, audio analysis is applied to achieve the melodic transcription of the sung notes. The detected melodic fragments are then grouped and mapped into single notes, which are connected to their related notes on the music score of the solfège exercise. This mapping procedure allows the direct assessment (note by note) of the singing performance, even if there are slight discrepancies between the transcribed notes and the music score. Finally, the last part of the proposed system combines the gesture of meter-mimicking (video information) with the melodic transcription (audio information), where the hand movement works as a metronome, controlling the time flow (tempo) of the musical piece. Thus, the meter-mimicking is used to align the music score (ground truth) with the sung melody, allowing the assessment even in time dynamic scenarios. Both meter-mimicking and sung notes are evaluated by a set of Bayesian classifiers that were generated from real evaluations done by experts listeners. In this way, the developed model performs the mapping of the point of view from human experts into an automatic system which is able to make the solfege assessment, regarding the pitch, onset and duration of the music notes, without the need for external synchronization with a metronome or the maintenance of a fixed tempo.

O solfejo, em seu conceito mais amplo, é uma técnica usual no processo de ensinoaprendizagem musical, o qual envolve a realização vocal de melodias, considerando as alturas e as durações dos sons musicais registrados em partitura, devidamente associado à marcação de compassos por intermédio de gestos que definem a estrutura métrica e o andamento da peça musical. Este trabalho apresenta uma abordagem audiovisual para avaliação automática dessa prática de estudo pertinente à leitura e à estruturação musicais. O sistema proposto é dividido em três partes. A primeira efetua o reconhecimento visual dos gestos de marcação de compassos realizados pela mão, por intermédio de um classificador probabilístico. Um processo de alinhamento temporal garante o reconhecimento dos padrões de movimento mesmo em casos com variação de andamento, permitindo também a avaliação da precisão rítmica do aluno, quando comparado com um referencial metronômico. A segunda parte deste sistema obtém a transcrição melódica do canto a partir da análise do respectivo sinal de áudio. Os fragmentos melódicos detectados são agrupados e mapeados em relação às notas da partitura do exercício de solfejo, permitindo uma avaliação direta nota-a-nota da performance do canto. Por fim, a terceira parte do sistema proposto faz a integração entre o gesto de marcação de compassos e a transcrição melódica. Nesse caso, o gesto atua como um metrônomo, controlando o fluxo temporal. Assim, a avaliação nota-a-nota do solfejo pode ser empregada também em casos onde exista grande variação no andamento da peça. Tanto o processo de avaliação do gesto de marcação de compassos quanto a avaliação do canto são obtidos por intermédio de um classificador Bayesiano gerado a partir de avaliações reais, feitas por especialistas em música. Dessa forma, o sistema desenvolvido efetua o mapeamento advindo da opinião de especialistas humanos em um sistema de avaliação automática de solfejo executado por máquina, que é capaz de identificar as notas musicais cantadas pelo aluno em cada instante métrico determinado, devidamente conduzido pelo gesto, sem a necessidade de sincronização por um metrônomo ou manutenção de um andamento fixo. / Solfége is a general technique used in the music learning process, which involves the vocal performance of melodies, regarding the time and duration of musical sounds as specified in the music score, properly associated the meter-mimicking performed by the hand movement. This thesis presents an audiovisual approach for automatic assessment of this relevant musical study practice. The proposed system is divided into three parts. First, a probabilistic classifier recognizes the musical metric patterns drawn by the hand movement. A time alignment process assures the proper recognition of the movement patterns even in cases where there are changes in the musical tempo. Also, this process allows to estimate the accuracy of the rhythmic performance. In the second part of this work, audio analysis is applied to achieve the melodic transcription of the sung notes. The detected melodic fragments are then grouped and mapped into single notes, which are connected to their related notes on the music score of the solfège exercise. This mapping procedure allows the direct assessment (note by note) of the singing performance, even if there are slight discrepancies between the transcribed notes and the music score. Finally, the last part of the proposed system combines the gesture of meter-mimicking (video information) with the melodic transcription (audio information), where the hand movement works as a metronome, controlling the time flow (tempo) of the musical piece. Thus, the meter-mimicking is used to align the music score (ground truth) with the sung melody, allowing the assessment even in time dynamic scenarios. Both meter-mimicking and sung notes are evaluated by a set of Bayesian classifiers that were generated from real evaluations done by experts listeners. In this way, the developed model performs the mapping of the point of view from human experts into an automatic system which is able to make the solfege assessment, regarding the pitch, onset and duration of the music notes, without the need for external synchronization with a metronome or the maintenance of a fixed tempo.

The freshmen programming courses at the University of Economics in Prague offer a unique approach to learning the art of programming and software engineering. The introductory courses follow the Architecture First methodology that gives students the opportunity to learn programming from the top down, without being constrained by the specifics and syntax of any one programming language. It teaches the thought processes needed to build programs, allowing the student to absorb the big ideas of computer programming. The average number of freshmen at the faculty of Informatics and Statistics is around seven hundred students. The task of correcting programming assignment and preparing appropriate feedback would be a mammoth undertaking for teaching staff in most university settings worldwide that offer similar computing degrees. It is therefore quite often the case that the faculty provisions some sort of automated testing technology that can handle the volume and provide both the teaching staff and the students with the tools needed to manage the assignments. These automated tools or systems have been, and continue to be, the subject of many research topics across the world and continue to evolve as new technologies and teaching methods evolve. This study first introduces the theoretical background of automated assessment and grading tools and systems and then provides an analysis of the fields current state. Taking that as input to the next phase, the study uses that information to then design and implement a custom-built system that would enable the automated testing of the structure and other aspects of student assignments. The main goal for the resulting system is to provide an intuitive and convenient way of declaring what needs to be tested for a given assignment and then providing the mechanism to run those tests automatically. The resulting system, DynoGrader, dynamically validates student assignments at runtime using Java runtime annotation processing mechanisms and Java Reflection API.