The resources required to have humans score extended written response items in English language learner (ELL) contexts has caused automated essay scoring (AES) to emerge as a desired alternative. However, these systems often rely heavily on indirect proxies of writing quality such as word, sentence, and essay lengths because of their strong correlation to scores (Vajjala, 2017). This has led to concern about the validity of the features used to establish the predictive accuracy of AES systems (Attali, 2007; Weigle, 2013). Reliance on construct-irrelevant features in ELL contexts also forfeits the opportunity to provide meaningful diagnostic feedback to test-takers or provide the second language acquisition (SLA) field with real insights (C.-F. E. Chen & Cheng, 2008). This thesis seeks to improve the validity and reliability of an AES system developed for ELL essays by employing a new set of features based on the acquisition order of English negation. Modest improvements were made to a baseline AES system's accuracy, showing the possibility and importance of engineering features relevant to the construct being assessed in ELL essays. In addition to these findings, a novel ordering of the sequence of English negation acquisition not previously described in SLA research emerged.

A predictive validity approach has been employed to find some implications to support evidences for Automated Essay Scoring (AES) systems. First, using R² values from multiple linear regression models, validity indices are compared first between multiple choice scores and essay scores across four AES systems. Secondly, R² values from models using only essay scores, the validity indices of four AES systems are hypothetically compared to see if how well AES systems could predict student outcome such as GPA. / text

Good writing skills are essential in the education system at all levels. However, the evaluation of essays is labor intensive and can entail a subjective bias. Automated Essay Scoring (AES) is a tool that may be able to save teacher time and provide more objective evaluations. There are several successful AES systems for essays in English that are used in large scale tests. Supervised machine learning algorithms are the core component in developing these systems. In this project four AES systems were developed and evaluated. The AES systems were based on standard supervised machine learning software, i.e., LDAC, SVM with RBF kernel, polynomial kernel and Extremely Randomized Trees. The training data consisted of 1500 high school essays that had been scored by the students' teachers and blind raters. To evaluate the AES systems, the agreement between blind raters' scores and AES scores was compared to agreement between blind raters' and teacher scores. On average, the agreement between blind raters and the AES systems was better than between blind raters and teachers. The AES based on LDAC software had the best agreement with a quadratic weighted kappa value of 0.475. In comparison, the teachers and blind raters had a value of 0.391. However the AES results do not meet the required minimum agreement of a quadratic weighted kappa of 0.7 as defined by the US based nonprofit organization Educational Testing Services. / Jag har utvecklat och utvärderat fyra system för automatisk betygsättning av uppsatser (AES). LDAC, SVM med RBF kernel, SVM med Polynomial kernel och "Extremely Randomized trees" som är standard klassificerarprogramvaror har använts som grunden för att bygga respektivt AES system.

Written skills are an essential evaluation criterion for a student’s creativity, knowledge, and intellect. Consequently, academic writing is a common part of university and college admissions applications, standardized tests, and classroom assessments. However, the task for teachers is quite daunting when it comes to essay scoring. Then Automated Essay Scoring may be a helpful tool in the decision-making by the teacher.  There have been many successful models with supervised or unsupervised machine learning algorithms in the eld of Automated Essay Scoring. This thesis work makes a comparative study among various neural network models with supervised machine learning algorithms and different linguistic feature combinations. It also proves that the same linguistic features are applicable to more than one language.  The models studied in this experiment include TextCNN, TextRNN_LSTM, Tex- tRNN_GRU, and TextRCNN trained with the essays from the Automated Student Assessment Prize (ASAP) from Kaggle competitions. Each essay is represented with linguistic features measuring linguistic complexity. Those features are divided into four groups: count-based, morphological, syntactic, and lexical features, and the four groups of features can form a total of 14 combinations.  The models are evaluated via three measurements: Accuracy, F1 score, and Quadratic Weighted Kappa. The experimental results show that models trained only with count-based features outperform the models trained using other feature combinations. In addition, TextRNN_LSTM performs best, with an accuracy of 54.79%, an F1 score of 0.55, and a Quadratic Weighted Kappa of 0.59, which beats the statistically-based baseline models.

Betygsättning upptar idag en stor del av lärares arbetstid och det finns en betydande inkonsekvens vid bedömning utförd av olika lärare. Denna studie ämnar undersöka vilken träffsäkerhet som en automtiserad bedömningsmodell kan uppnå. Tre maskininlärningsmodeller för klassifikation i form av Linear Discriminant Analysis, K-Nearest Neighbor och Random Forest tränas och testas med femfaldig korsvalidering på uppsatser från nationella prov i svenska. Klassificeringen baseras på språk och formrelaterade attribut inkluderande ord och teckenvisa längdmått, likhet med texter av olika formalitetsgrad och grammatikrelaterade mått. Detta utmynnar i ett maximalt quadratic weighted kappa-värde på 0,4829 och identisk överensstämmelse med expertgivna betyg i 57,53 % av fallen. Dessa resultat uppnåddes av en modell baserad på Linear Discriminant Analysis och uppvisar en högre korrelation med expertgivna betyg än en ordinarie lärare. Trots pågående digitalisering inom skolväsendet kvarstår ett antal hinder innan fullständigt maskininlärningsbaserad bedömning kan realiseras, såsom användarnas inställning till tekniken, etiska dilemman och teknikens svårigheter med förståelse av semantik. En delvis integrerad automatisk betygssättning har dock potential att identifiera uppsatser där behov av dubbelrättning föreligger, vilket kan öka överensstämmelsen vid storskaliga prov till en låg kostnad. / Today, a large amount of a teacher’s workload is comprised of essay scoring and there is a large variability between teachers’ gradings. This report aims to examine what accuracy can be acceived with an automated essay scoring system for Swedish. Three following machine learning models for classification are trained and tested with 5-fold cross-validation on essays from Swedish national tests: Linear Discriminant Analysis, K-Nearest Neighbour and Random Forest. Essays are classified based on 31 language structure related attributes such as token-based length measures, similarity to texts with different formal levels and use of grammar. The results show a maximal quadratic weighted kappa value of 0.4829 and a grading identical to expert’s assessment in 57.53% of all tests. These results were achieved by a model based on Linear Discriminant Analysis and showed higher inter-rater reliability with expert grading than a local teacher. Despite an ongoing digitilization within the Swedish educational system, there are a number of obstacles preventing a complete automization of essay scoring such as users’ attitude, ethical issues and the current techniques difficulties in understanding semantics. Nevertheless, a partial integration of automatic essay scoring has potential to effectively identify essays suitable for double grading which can increase the consistency of large-scale tests to a low cost.

Manually grading essays and providing comprehensive feedback pose significant challenges for writing instructors, requiring subjective assessments of various writing elements. Automated essay scoring (AES) systems have emerged as a potential solution, offering improved grading consistency and time efficiency, along with insightful analytics. However, the use of AES in English as a Second Language (ESL) remains rare. This dissertation aims to explore the implementation of AES in ESL education to enhance teaching and learning. The dissertation presents a study involving ESL teachers who learned to use a specific AES system called LightSide, a free and open text mining tool, to enhance writing instruction. The study involved observations, interviews, and a workshop where teachers learned to build their own AES using LightSide. The study aimed to address questions related to teacher interest in using AES, challenges faced by teachers, and the influence of the workshop on teachers' perceptions of AES. By exploring the use of AES in ESL education, this research provides valuable insights to inform the integration of technology and enhance the teaching and learning of writing skills for English language learners.

This thesis explored the integration of automated essay scoring (AES) software into the writing curriculum for high level ESOL students (levels 3, 4, and 5 on a 1-5 scale) at a high school in Miami, Fl. Issues for Haitian Creole speaking students were also explored. The Spanish and Haitian Creole speaking students were given the option to write notes, outlines, and planning sheets in their L1. After using AES in the middle of the writing process as a revision assistant tool, 24 students responded to a Likert Scale questionnaire. The students responded positively to the AES based on the results of the Likert scale questionnaire: 71% responded “agree” and “strongly agree” to the question “Other students would benefit from using writing software before handing in a final draft.” Also, the majority reported that they valued teacher feedback. None of the students chose to use their L1 to write notes/outlines.