To learn about computer operating systems, students at Virginia Tech implement a command-line shell in their Computer Systems course. Successfully implementing the shell requires a deep understanding of operating system abstractions and interactions. Students often struggle with the project because subtle errors can take hours to debug. In this work, we developed two hybrid domain-specific analysis tools to pinpoint the root causes of student errors: EshMD and ShellTrace. The EshMD tool models common errors in the shell and checks the student code against those models. To accomplish this, it monitors the specific calls the program is making and correlates those with expected changes in its environment. Students' errors are shown directly in the source code. The concept of EshMD can be applied to other programming projects by observing and modeling common bugs during implementation. The ShellTrace tool dynamically creates a specification from a reference solution based on how the reference solution makes use of operating system resources and then uses this specification to check that a student solution is functionally identical. The ShellTrace concept can be applied to other programs that exhibit similar resource dependencies. We deployed these tools in an undergraduate computer systems class and evaluated our tools based on the number of bugs detected and the students' perceptions of usefulness. We found that the tools detected a significant number of bugs and that the majority of students that made use of the tools found them valuable in debugging their submissions. / Master of Science / To learn about computer operating systems, students at Virginia Tech implement a command-line shell in their junior-level Computer Systems course. A command-line shell is a computer program that allows the user of a computer to run other programs by typing in the program names. The command-line shell will then run those programs on the user’s behalf. The command-line shell project requires students to understand and use the abstractions that the underlying operating system provides to a computer program. Students often struggle with the project because subtle errors in their implementation can take hours to address. In this work, we developed two analysis tools to pinpoint the root cause of student errors: EshMD and ShellTrace. The EshMD tool models common errors in the command-line shell and checks the student code against those models. Students’ errors are shown by directly pointing to the error locations in their code. The ShellTrace tool dynamically creates a project specification from a solution written by the course staff and checks that a student solution meets the specification. Generating from the staff solution allows checking the student solution without having to encode the specific project requirements into the tool itself. We deployed these tools in an undergraduate computer systems class and evaluated our tools based on the number of errors detected and the students’ perceptions of usefulness. We found that the tools detected a significant number of errors and that the majority of students that made use of the tools found them valuable in finding and fixing the errors in their submissions.

Measuring student learning is fundamental to any educational endeavor. A primary goal of many computer science education projects is to determine the extent to which a given instructional intervention has had an impact on student learning. However, the field of computing lacks valid and reliable assessment instruments for pedagogical or research purposes. Without such valid assessments, it is difficult to accurately measure student learning or establish a relationship between the instructional setting and learning outcomes. The goal of assessment research in computer science is to have valid ways of measuring student conceptions of fundamental topics, which will enable both research into how understanding of knowledge in the domain develops as well as enable curricular innovation and reform grounded in this knowledge. My dissertation work focused on three questions regarding assessment of introductory concepts in computer science. How can existing test development methods be applied and adapted to create a valid assessment instrument for CS1 conceptual knowledge? To what extent can pseudo-code be used as the mechanism for achieving programming language independence in an assessment instrument? And to what extent does the language independent instrument provide a valid measure of CS1 conceptual knowledge? I developed the Foundational CS1 (FCS1) Assessment instrument, the first assessment instrument for introductory computer science concepts that is applicable across a variety of current pedagogies and programming languages. I applied methods from educational and psychological test development, adapting them as necessary to fit the disciplinary context. I conducted think aloud interviews and a large scale empirical study to demonstrate that pseudo-code was an appropriate mechanism for achieving programming language independence. Student participants were able to read and reason in the pseudo-code syntax without difficulty and were able to transfer conceptual knowledge from their CS1 programming language to pseudo-code. Finally, I established the validity of the assessment using a multi-faceted argument, combining interview data, statistical analysis of results on the assessment, and exam scores. The contributions of this research are: (1) An example of how to bootstrap the process for developing the first assessment instrument for a disciplinary specific design-based field. (2) Identification that although it may not be possible to correlate scores between computer science exams created with different measurement goals, the validity claims of the individual assessments are not diminished. (3) A demonstration that novice computing students, at an appropriate level of development, can transfer their understanding of fundamental concepts to pseudo-code notation. (4) A valid assessment of introductory computing concepts for procedurally-based introductory computing courses taught in Java, Matlab, or Python at the university level.

As the demand for programming skills in today’s job market is rapidly increasing for disciplines outside of computing, CS courses have experienced spikes in enrollment for non-majors. Students in disciplines including art, design and biological sciences are now often required to take introductory CS courses. Previous research has shown the role of growth mindset, self-efficacy and relevance in student success within CS but such metrics are largely unknown for non-majors. In this thesis, we surveyed non-majors in CS1 at Cal Poly, San Luis Obispo during the early and late weeks of the quarter to gain insights on their growth mindset, their self-efficacy and the perceived relevance of the course to their lives. In our analysis, we discovered that non-majors’ levels of growth mindset and of self-efficacy decreased throughout the duration of CS1 with additional differences by gender. However, non-majors largely found that the material covered in CS1 was highly relevant to their academic and professional careers despite being challenged by it. These findings provide important insights into the experiences of non-majors learning to code and can help better serve a more diverse population of students.

The study of programming languages is a rich field within computer science, incorporating both the abstract theoretical portions of computer science and the platform specific details. Topics studied in programming languages, chiefly compilers or interpreters, are permanent fixtures in programming that students will interact with throughout their career. These systems are, however, considerably complicated, as they must cover a wide range of functionality in order to enable languages to be created and run. The process of educating students thus requires that the demanding workload of creating one of the systems be balanced against the time and resources present in a university classroom setting. Systems building upon these fundamental systems can become out of reach when the number of preceding concepts and thus classes are taken into account. Among these is the study of just-in-time (JIT) compilers, which marry the processes of interpreters and compilers for the purposes of a flexible and fast runtime. The purpose of this thesis is to present JITed, a framework within which JIT compilers can be developed with a time commitment and workload befitting of a classroom setting, specifically one as short as ten weeks. A JIT compiler requires the development of both an interpreter and a compiler. This poses a problem, as classes teaching compilers and interpreters typically feature the construction of one of those systems as their term project. This makes the construction of both within the same time span as is usually allotted for a single system infeasible. To remedy this, JITed features a prebuilt interpreter, that provides the runtime environment necessary for the compiler portion of a JIT compiler to be built. JITed includes an interface for students to provide both their own compiler and the functionality to determine which portions of code should be compiled. The framework allows for important concepts of both compilers in general and JIT compilers to be taught in a reasonable timeframe.