This thesis investigates methods of assessing students' mathematical ability by using the computer. It starts by reviewing the general types of assessment within mathematics educational software and then describes some different ways of presenting the assessment on the computer by the use of varying types of questions. In Chapter 2 there is a review of the literature and research conducted in the area of computer assessment of mathematics. In particular, the most prevalent dilemmas of computer aided learning and computer aided assessment are highlighted whilst looking forward at how the contents of further chapters in the thesis can help in addressing some of these difficulties. The following chapter gives an historical account of how the CALM(1) software has addressed some of the inherent difficulties of assessment and highlights the ways in which some of these hurdles have been overcome. The shortfalls of CALM are described and, where relevant, pointers to the parts of thesis which tackle these shortfalls are given. In particular, the work in Chapter 4 undertakes an improvement in the way simple mathematical expressions(2) can be handled as it shows how binary tree constructions can be utilised within an educational environment. Chapter 5 tests out two applications of the binary tree structures with the creation of a tool to aid student-computer communication of mathematics and by providing a method of comparing student-set questions against a true answer. The following chapter describes an educational experiment which set out to show how a computer can be used to assess students' mathematical ability during a formal university examination. It deals with very important educational issues which arise when performing such examinations and gives conclusions as to their educational validity. In particular, issues of student input, partial credit, objectivity, consistency, flexibility and efficiency are considered along with the impact that this research could have for future testing of mathematics. The final chapter describes how the thesis has been instrumental in further research and development within the field of computer assessment of mathematics. (1)CALM is the acronym for the Computer Aided Learning in Mathematics project at the Department of Mathematics, Heriot- Watt University. (2)in this thesis, the word expression is taken to be a mathematical entity which does not contain any comparison operators.

The objective of this thesis is to establish whether or not online, objective questions in elementary graph theory can be written in a way that exploits the medium of computer-aided assessment. This required the identification and resolution of question design and programming issues. The resulting questions were trialled to give an extensive set of answer files which were analysed to identify whether computer delivery affected the questions in any adverse ways and, if so, to identify practical ways round these issues. A library of questions spanning commonly-taught topics in elementary graph theory has been designed, programmed and added to the graph theory topic within an online assessment and learning tool used at Brunel University called Mathletics. Distracters coded into the questions are based on errors students are likely to make, partially evidenced by final examination scripts. Questions were provided to students in Discrete Mathematics modules with an extensive collection of results compiled for analysis. Questions designed for use in practice environments were trialled on students from 2007 – 2008 and then from 2008 to 2014 inclusive under separate testing conditions. Particular focus is made on the relationship of facility and discrimination between comparable questions during this period. Data is grouped between topic and also year group for the 2008 – 2014 tests, namely 2008 to 2011 and 2011 to 2014, so that it may then be determined what factors, if any, had an effect on the overall results for these questions. Based on the analyses performed, it may be concluded that although CAA questions provide students with a means for improving their learning in this field of mathematics, what makes a question more challenging is not solely based on the number of ways a student can work out his/her solution but also on several other factors that depend on the topic itself.

This thesis describes the development of Computer-Aided Assessment questions for elementary discrete and decision mathematics at the school/university interface, stressing the pedagogy behind the questions’ design and the development of methodology for assessing their efficacy in improving students’ engagement and perceptions, as well as on their exams results. The questions give instant and detailed feedback and hence are valuable as diagnostic, formative or summative tools. A total of 275 questions were designed and coded for five topics, numbers, sets, logic, linear programming and graph theory, commonly taught to students of mathematics, computer science, engineering and management. Pedagogy and programming problems with authoring questions were resolved and are discussed in specific topic contexts and beyond. The delivery of robust and valid objective questions, even within the constraints of CAA, is therefore feasible. Different question types and rich feedback comprising text, equations and diagrams that allow random parameters to produce millions of realisations at run time, can give CAA an important role in teaching mathematics at this level. Questionnaires identified that CAA was generally popular with students, with the vast majority seeing CAA not only as assessment but also as a learning resource. To test the impact of CAA on students’ learning, an analysis of the exam scripts quantified its effect on class means and standard deviations. This also identified common student errors, which fed into the question design and editing processes by providing evidence-based mal-rules. Four easily-identified indicators (correctly-written remainders, conversion of binary/octal/hexadecimal numbers, use of correct set notation {…} and consistent layout of truth tables) were examined in student exam scripts to find out if the CAA helps students to improve examination answers. The CAA answer files also provided the questions’ facilities and discriminations, potentially giving teachers specific information on which to base and develop their teaching and assessment strategies. We conclude that CAA is a successful tool for the formative/summative assessment of mathematics at this level and has a positive effect on students’ learning.

Diagrams are increasingly used in many design methods, and are being taught in a variety of contexts in higher education such as database conceptual design or software design in computer science. They are an important part of many assessments. Currently computer aided assessments are widely used for multiple choice questions. They lack the ability to assess a student's knowledge in a more comprehensive way, which is required for diagram-type student work. The aim of this research is to develop a semi-automatic assessment framework, which enables the use of computer to support the assessment process of diagrammatic solutions, with the focus of ensuring the consistency of grades and feedback on solutions. A novel trace model, that captures design traces of student solutions, was developed as a part of the framework and was used to provide the matching criteria for grouping the solutions. A new marking style, partial marking, was developed to mark these solution groups manually. The Case-Based Reasoning method is utilised in the framework to mark some of the groups automatically. A guideline for scenario writing was proposed to increase the efficiency of automatic marking. A prototype diagram editor, a marking tool and scenario writing environment were implemented for the proposed framework in order to demonstrate proof of concept. The results of experiments show that the framework is feasible to use in the formative assessment and it provides consistent marking and personalised feedback to the students. The framework also has the potential to significantly reduce the time and effort required by the examiner to mark student diagrams. Although the constructed framework was specifically used for the assessment of database diagrams, the framework is generic enough to be used for other types of graph-based diagram.

Why this thesis is needed. This thesis is motivated by the falling school results of Swedish 15-year-olds, a lack of IT tools in Swedish schools and a call for turning the theory on formative assessment into practice. Previous research that is used in the thesis. This thesis mainly builds on the research done by John Hattie which is presented in his book Visible Learning from 2009, Wiliam & Thompson's research on effective formative assessment from 2007, Pachler et al. research on formative e-assessment from 2010 and Panadero & Jonsson's research on scoring rubrics from 2013. Research question. What key aspects should be taken into consideration when designing and implementing an IT tool based on scoring rubrics which aims to support formative assessment according to the key strategies for effective formative assessment as identified by Wiliam & Thompson (2007)? Method. To answer this question I chose a qualitative approach of parallel paper prototyping where I iteratively exposed the design and my ideas during focus groups to teachers, a headmaster, employees at The Swedish National Agency for Education and high school students, followed by an evaluation of the key topics that surfaced during the focus groups. Results. The results consist of eight key aspects to take into consideration when designing and implementing such a tool. Discussion and future research. Finally I discuss the implications of my findings and present directions for future research which include the construction of the tool, further investigation of the key aspects identified in this study, identification of additional key aspects and more. / Varför denna avhandling behövs. Denna avhandling motiveras av de fallande skolresultaten hos svenska 15-åringar, en brist på IT-verktyg i svenska skolor och ett rop efter att vända forskning inom formativ bedömning till praktik. Tidigare forskning som används. Denna avhandling bygger i huvudsak på forskningen av John Hattie som presenteras i hans bok Visible Learning från 2009, William & Thompsons forskning på effektiv formativ bedömning från 2007, Pachler et al. forskning på formativ e-bedömning från 2010 och Panadero & Jonssons forskning på betygsmatriser från 2013. Forskningsfråga. Vilka nyckelaspekter ska tas i åtanke vid utveckling av ett IT-verktyg baserat på betygsmatriser som stödjer formativ bedömning enligt de fem nyckelstrategierna för effektiv formativ bedömning som formulerats av Wiliam & Thompson (2007)? Metod. För att besvara denna fråga valdes en kvalitativ metod där parallell pappersprototypning användes under ett antal iterationer av fokusgrupper under vilka designen och tidigare diskussionpunkter diskuterades med lärare, en rektor, anställda vid Skolverket samt gymnasiestudenter. Fokusgrupperna följdes upp med en utvärdering av de största diskussionpunkterna som dök upp. Resultat. Resultaten består av åtta nyckelaspekter att ha i åtanke vid utveckling av ett sådant verktyg. Diskussion och framtida forskning. Slutligen diskuteras implikationerna av resultaten och direktioner för framtida forskning framförs. Dessa direktioner inkluderar utveckling av verktyget, vidare utredning av de nyckelaspekter som hittats i denna avhandling, identifikation av ytterligare nyckelaspekter och mer.

The Swedish government has decided to add programming to the Swedish curriculum to strengthen students’ digital skills. The teachers who will teach programming do not always know programming themselves. Because of that, KTH Royal Institute of Technology researchers are planning to start an initiative of creating a Massive Open Online Course (MOOC) in fundamental programming in Swedish for those teachers. Interpreting error messages given by the compiler is one problem with learning programming. An aim of this study is to lower that threshold. The study seeks to identify common misconceptions about programming among novice programmers in order to design a static analyser that investigates code written by students and provides formative feedback to help students in their own learning process. The study combines the constructivist theory of learning with views about formative assessment in order to automate the process that is usually done by a teacher or teaching assistant when assessing code. A phenomenographic study is done in order to identify teachers’ perceptions of common misconceptions about programming by interviewing five active teachers at KTH. The results of that study are used to construct code examples that correspond to these misconceptions. Those results lead to the design and implementation of a software that detects these problems in code. That software is evaluated using a larger set of test data, consisting of 77 errors divided into five larger programs, inserted by independent individuals. From the initial study, five categories of misconceptions are given. Of the 77 errors, the majority are correctly positioned and almost all are given a good hint about the position. About a quarter of the errors are parse errors, which never reach the analysing part of the software that demands the program to be parsable. The study shows that we have succeeded both in designing and implementing a software that detects the identified misconceptions with good results. In the context of a MOOC, the software might require an extension with a more advanced parser and also dynamic analysis to be able to test the correctness of the students’ programs. The software is limited to handle the language Javascriptish, which is a subset of JavaScript. / För att stärka den digitala kompetensen bland svenska grundskole och gymnasieelever har programmering lagts in i läroplanen. De lärare som är tänkta att hålla i denna programmeringsundervisning har själva inte alltid tillräckliga kunskaper i programmering, varpå ett initiativ för att hålla en storskalig, öppen och internetbaserad kurs (MOOC) på svenska har startats av forskare på Kungliga Tekniska Högskolan (KTH). En av svårigheterna med att lära sig programmering är att förstå meddelandena som kompilatorn ger. Ett mål med denna studie är att sänka denna tröskel för studenten. Denna studie ämnar identifiera vanliga missuppfattningar om programmering hos nybörjare, för att designa en programvara som utför statisk analys av kod skriven av studenter och därefter ge återkoppling kring dessa missuppfattningar. Studien kombinerar den konstruktivistiska teorin om lärande med tankar om formativ bedömning för att automatisera den process som vanligtvis görs av en lärare eller lärarassistent vid bedömning av kod. En fenomenografisk studie görs för att identifiera lärares uppfattningar om vanliga missuppfattningar kring programmering genom att intervjua fem verksamma lärare på KTH. Resultatet från den studien används sedan för att konstruera kodexempel som belyser de identifierade missuppfattningarna. Därefter designas och implementeras en mjukvara som sedan evalueras genom att analysera totalt 77 fel, konstruerade av oberoende individer, uppdelade på fem större program. Den initiala studien resulterar i fem olika kategorier av missuppfattningar. Av de 77 evaluerade felen ger majoriteten en korrekt positionsangivelse och nästan alla ger en god indikation över var felet ligger. Omkring en fjärdedel av felen är parsningsfel, vilka aldrig når huvudmjukvaran som kräver att programmet är parsningsbart. Studien visar att vi lyckas designa och implementera en programvara som med goda resultat upptäcker vanliga missuppfattningar kring programmering hos nybörjare, baserat på det teoretiska ramverket. I kontexten av en kurs på internet kan programvaran behöva utvecklas med en mer avancerad syntaxanalys (eng. parser) samt lägga till dynamisk analys av program för att även kunna testa programmens korrekthet. Programvaran är begränsad till att analysera kod skriven i språket Javascriptish, vilken är en delmängd till JavaScript.