In every learning or training environment, exercises are the basis for practical learning. Learners need to practice in order to acquire new abilities and perfect those gained previously. However, not every exercise is valid for every learner: learners require exercises that match their ability levels. Hence, difficulty of an exercise could be defined as the amount of effort that a learner requires to successfully complete the exercise (its learning cost). Too high difficulties tend to discourage learners and make them drop out, whereas too low difficulties are perceived as unchallenging, resulting in loss of interest. Correctly estimating difficulties is hard and error-prone problem that tends to be done manually using domain-expert knowledge. Underestimating or overestimating difficulty generates a problem for learners, increasing dropout rates in learning environments. This paper presents a novel approach to improve difficulty estimations by using Neuroevolution. The method is based on measuring the computational cost that Neuroevolution algorithms require to successfully complete a given exercise and establishing similarities with previously gathered information from learners. For specific experiments presented, a game called PLMan has been used. PLMan is a PacMan-like game in which users have to program the Artificial Intelligence of the main character using a Prolog knowledge base. Results show that there exists a correlation between students’ learning costs and those of Neuroevolution. This suggests that the approach is valid, and measured difficulty of Neuroevolution algorithms may be used as estimation for student's difficulty in the proposed environment.

Enrollment in full-time, virtual, K-12 schools is increasing while mathematics performance in these institutions is lacking compared to national averages. Scholarly literature lacks research studies using learning analytics to better predict student outcomes via student learning management system (LMS) interactions, specifically in the low performing area of middle school mathematics. The theoretical framework for this study was a combination of Hrastinski's theory of online learning as online participation and Moore's 3 types of interactions model of online student behavior. The purpose of this study was to address the current research gap in the full-time, K-12 eLearning field and determine whether 2 types of student LMS interactions could predict mathematics course performance. The research questions were developed to determine whether student clicks navigating course content page(s) or the number of times a student accessed resources predicted student performance in a full-time, virtual, mathematics course after student demographic variables were controlled for. This quantitative study used archived data from 238 seventh grade Math 7B students enrolled from January 8th-10th to May 22nd-25th in two Midwestern, virtual, K-12 schools. Hierarchical regressions were used to test the 2 research questions. Student clicks navigating the course content pages were found to predict student performance after the effects of student demographic covariates were controlled for. Similarly, the number of times a student accessed resources also predicted student performance. The findings from this study can be used to advise actionable changes in student support, build informative student activity dashboards, and predict student outcomes for a more insightful, data-driven, learning experience in the future.

Investigating howgroups communicate, build knowledge and expertise, reach consensus or collaboratively solve complex problems, became one of the main foci of contemporary research in learning and social sciences. Emerging models of communication and empowerment of networks as a form of social organization further reshaped practice and pedagogy of online education, bringing research on learning networks into the mainstream of educational and social science research. In such conditions, massive open online courses (MOOCs) emerged as one of the promising approaches to facilitating learning in networked settings and shifting education towards more open and lifelong learning. Nevertheless, this most recent educational turn highlights the importance of understanding social and technological (i.e., material) factors as mutually interdependent, challenging the existing forms of pedagogy and practice of assessment for learning in online environments. On the other hand, the main focus of the contemporary research on networked learning is primarily oriented towards retrospective analysis of learning networks and informing design of future tasks and recommendations for learning. Although providing invaluable insights for understanding learning in networked settings, the nature of commonly applied approaches does not necessarily allow for providing means for understanding learning as it unfolds. In that sense, learning analytics, as a multidisciplinary research field, presents a complementary research strand to the contemporary research on learning networks. Providing theory-driven and analytics-based methods that would allow for comprehensive assessment of complex learning skills, learning analytics positions itself either as the end point or a part of the pedagogy of learning in networked settings. The thesis contributes to the development of learning analytics-based research in studying learning networks that emerge fromthe context of learning with MOOCs. Being rooted in the well-established evidence-centered design assessment framework, the thesis develops a conceptual analytics-based model that provides means for understanding learning networks from both individual and network levels. The proposed model provides a theory-driven conceptualization of the main constructs, along with their mutual relationships, necessary for studying learning networks. Specifically, to provide comprehensive understanding of learning networks, it is necessary to account for structure of learner interactions, discourse generated in the learning process, and dynamics of structural and discourse properties. These three elements – structure, discourse, and dynamics – should be observed as mutually dependent, taking into account learners’ personal interests, motivation, behavior, and contextual factors that determine the environment in which a specific learning network develops. The thesis also offers an operationalization of the constructs identified in the model with the aim at providing learning analytics-methods for the implementation of assessment for learning. In so doing, I offered a redefinition of the existing educational framework that defines learner engagement in order to account for specific aspects of learning networks emerging from learning with MOOCs. Finally, throughout the empirical work presented in five peer-reviewed studies, the thesis provides an evaluation of the proposed model and introduces novel learning analytics methods that provide different perspectives for understanding learning networks. The empirical work also provides significant theoretical and methodological contributions for research and practice in the context of learning networks emerging from learning with MOOCs.

With the increasing pace of technological changes in the modern society, there has been a growing interest from educators, business leaders, and policymakers in teaching important higher-order skills which were identified as necessary for thriving in the present-day globalized economy. In this regard, one of the most widely discussed higher order skills is critical thinking, whose importance in shaping problem solving, decision making, and logical thinking has been recognized. Within the domain of distance and online education, the Community of Inquiry (CoI) model provides a pedagogical framework for understanding the critical dimensions of student learning and factors which impact the development of student critical thinking. The CoI model follows the social-constructivist perspective on learning in which learning is seen as happening in both individual minds of learners and through the discourse within the group of learners. Central to the CoI model is the construct of cognitive presence, which captures the student cognitive engagement and the development of critical thinking and deep thinking skills. However, the assessment of cognitive presence is challenging task, particularly given its latent nature and the inherent physical and time separation between students and instructors in distance education settings. One way to address this problem is to make use of the vast amounts of learning data being collected by learning systems. This thesis presents novel methods for understanding and assessing the levels of cognitive presence based on learning analytics techniques and the data collected by learning environments. We first outline a comprehensive model for cognitive presence assessment which builds on the well-established evidence-cantered design (ECD) assessment framework. The proposed assessment model provides a foundation of the thesis, showing how the developed analytical models and their components fit together and how they can be adjusted for new learning contexts. The thesis shows two distinct and complementary analytical methods for assessing students’ cognitive presence and its development. The first method is based on the automated classification of student discussion messages and captures learning as it is observed in the student dialogue. The second analytics method relies on the analysis of log data of students’ use of the learning platform and captures the individual dimension of the learning process. The developed analytics also extend current theoretical understanding of the cognitive presence construct through data-informed operationalization of cognitive presence with different quantitative measures extracted from the student use of online discussions. We also examine methodological challenges of assessing cognitive presence and other forms of cognitive engagement through the analysis of trace data. Finally, with the intent of enabling for the wider adoption of the CoI model for new online learning modalities, the last two chapters examine the use of developed analytics within the context of Massive Open Online Courses (MOOCs). Given the substantial differences between traditional online and MOOC contexts, we first evaluate the suitability of the CoI model for MOOC settings and then assess students’ cognitive presence using the data collected by the MOOC platform. We conclude the thesis with the discussion of practical application and impact of the present work and the directions for the future research.

Submitted by Automação e Estatística (sst@bczm.ufrn.br) on 2018-07-26T17:05:20Z No. of bitstreams: 1 IvanMaxFreireDeLacerda_TESE.pdf: 3236688 bytes, checksum: f8269c0c3bdbd5a0b84b45f9f4181c93 (MD5) / Approved for entry into archive by Arlan Eloi Leite Silva (eloihistoriador@yahoo.com.br) on 2018-07-26T19:49:52Z (GMT) No. of bitstreams: 1 IvanMaxFreireDeLacerda_TESE.pdf: 3236688 bytes, checksum: f8269c0c3bdbd5a0b84b45f9f4181c93 (MD5) / Made available in DSpace on 2018-07-26T19:49:52Z (GMT). No. of bitstreams: 1 IvanMaxFreireDeLacerda_TESE.pdf: 3236688 bytes, checksum: f8269c0c3bdbd5a0b84b45f9f4181c93 (MD5) Previous issue date: 2018-03-02 / O crescente aumento dos dados registrados em cursos ofertados na modalidade a distância proporciona a utilização de métodos computacionais adaptados a pesquisa e agrupamento de dados educacionais, visando a descoberta de comportamentos de aprendizado. Essa área de pesquisa possibilita o desenvolvimento de ferramentas automatizadas de acompanhamento, predição e intervenção visando o aprimoramento dos índices educacionais. Em virtude disso, este trabalho propõe uma metodologia para a análise de programas de ensino a distância com base na tecnologia Learning Analytics, utilizando os dados de acesso dos alunos ao Ambiente Virtual de Aprendizagem (AVA), identificando os padrões sequenciais de uso mais frequentes e classificando-os de acordo com as categorias de aprendizagem autorregulada. Para a mineração sequencial de dados sequenciais os algoritmos SPAM e VGEN foram aplicados aos bancos de dados de duas instituições educacionais. Além do desenvolvimento da metodologia, como resultado desse processamento, uma grande incidência de um comportamento não previsto pela teoria da aprendizagem autorregulada foi identificado, e para classifica-lo foi criado um padrão chamado baixa participação. / The increasing of the data registered in courses offered in the distance modality boost the use of computational methods adapted to the research and the grouping of educational data, aiming to discover learning behaviors patterns. This research area allows the development of automated monitoring, prediction and intervention tools aiming at improving the educational indexes. As a result, this work proposes a methodology for analyzing distance learning programs based on the Learning Analytics technology, using the students’ access data to the Learning Management System (LMS), identifying the most frequent sequential patterns of use and classifying them as according to the self-regulated learning categories. For a sequential mining of sequential data the SPAM and VGEN algorithms were applied to the databases of two educational institutions. In addition to the development of the methodology, as a result of processing, a high incidence of behavior not predicted in the self-regulated learning theory was identified, and to classify it was created a pattern called low participation.

Submitted by Lucienne Costa (lucienneferreira@ufcg.edu.br) on 2018-05-03T20:37:40Z No. of bitstreams: 1 DOUGLAS AFONSO TENÓRIO DE MENEZES – TESE (PPGCC) 2017.pdf: 30688716 bytes, checksum: aeb667914da2d303938d7e10953623eb (MD5) / Made available in DSpace on 2018-05-03T20:37:40Z (GMT). No. of bitstreams: 1 DOUGLAS AFONSO TENÓRIO DE MENEZES – TESE (PPGCC) 2017.pdf: 30688716 bytes, checksum: aeb667914da2d303938d7e10953623eb (MD5) Previous issue date: 2017-07 / Capes / O uso de ambientes virtuais de aprendizagem está cada vez mais frequente, e muitas vezes os dados que são gerados nestes ambientes não são explorados adequadamente, o que dificulta a geração de indicadores da qualidade dos programas de formação. A qua- lidade do aprendizado em um ambiente virtual de aprendizagem é determinada por uma organização adequada do material e das formas de ensino. Para tal, deverá ser levado em consideração, entre outros fatores, o histórico de sucessos e insucessos de realizações anteriores das disciplinas a serem ofertadas e do perfil específico das novas turmas de alu- nos. O tratamento adequado destes dados pode evidenciar indicativos importantes sobre o desempenho de uma turma, tais como o nível de comprometimento e a motivação dos alunos, fatores que podem influenciar diretamente no processo de aprendizagem. Estes dadospodemserutilizadosporespecialistasaoseremexibidoscomoumresumoemforma de visualizações gráficas adequadas, possibilitando uma rápida interpretação e percepção de indicativos importantes dos cursos e seus alunos. As visualizações auxiliam na com- preensão e análise dos dados gerados, ampliam a cognição e facilitam a compreensão das informações apresentadas. Esta tese apresenta uma proposta para visualização de dados educacionais, com o objetivo de analisar como a visualização de dados pode ajudar o professor a identificar e adequar um Design Instrucional problemático, por meio de dados históricos, de cursos já realizados, assim como durante a realização de uma nova edição do curso, auxiliando-o na melhoria do DI de um curso/disciplina. / The use of virtual learning environments is becoming more frequent, and often the data that are generated in these environments are not properly exploited, which makes it difficult to generate quality indicators of training programs. The quality of learning in these settings is determined by an appropriate organization of the material and forms of instruction. In order to do so, it should be taken into account, among other factors, the history of successes and failures of previous achievements of the disciplines to be offered and the specific profile of a new class of students. Adequate treatment of these data can show important indicators about a class’s performance, such as the level of commitment and motivation of the students, factors that can directly influence the learning process. These data can be used by specialists to be displayed as a summary in the form of ade- quate graphical visualizations, allowing a fast interpretation and perception of important indications of a course and its students. It helps to understand a set of data, facilitate the analysis of the generated data, to increase the cognition and the understanding of the presented information. This thesis presents a proposal for the visualization of educatio- nal data, in which the objective is to analyze how the visualization of data can help the teacher to identify and adapt a problematic Instructional Design, through historical data of courses already carried out, and during the realization of a new edition of the course, assisting the teacher in improving the Instructional Design of a course / discipline.

<p>CAL (Computer Assisted Learning) programs are widespread today in schools and families due to the effectiveness of CAL programs in improving students’ learning and task performance. The flourishing of CAL programs in education has brought large amounts of students’ learning data including log data, performance data, mouse movement data, eye movement data, video data, etc. These data can present students’ learning or problem-solving processes and reflect underlying cognitive processes. These data are valuable resources for educators to comprehend students’ learning and difficulties. However, few data analysis methods can analyze and present CAL data for educators quickly and clearly. Traditional video analysis methods can be time-consuming. Current visualization analysis methods are limited to simple charts or visualizations of a single data type. In this dissertation, I propose a visual learning analytic approach to analyze and present students' problem-solving data from CAL programs. More specifically, a visualization system was developed to present students’ problem-solving data, including eye movement, mouse movement, and performance data, to help educational researchers understand student problem-solving processes and identify students’ problem-solving strategies and difficulties. An evaluation experiment was conducted to compare the visualization system with traditional video analysis methods. Seven educational researchers were recruited to diagnose students’ problem-solving patterns, strategies, and difficulties using either the visualization system or video. The diagnosis task loads and evaluators’ diagnosis processes were measured and the evaluators were interviewed. The results showed that analyzing student problem-solving tasks using the proposed visualization method was significantly quicker than using the video method. In addition, diagnosis using the visualization system can achieve results at least as reliable as the video analysis method. Evaluators’ preferences between the two methods are summarized and illustrated in the dissertation. Finally, the implications of the visual analytic approach in education and data visualization areas are discussed. </p>

Learning Analytics är ett forskningsområde som innefattar insamling, mätning, analysering och rapportering av “big data” om studenter i deras lärmiljö. Syftet är att förstå och optimera studenters lärande, och deras studiemiljöer. Learning Analytics-tjänster kan bland annat hjälpa studenter att få en insikt i hur de bör studera för att vara tidseffektiva eller höja sina studieprestationer. Dessutom kan tjänsterna upptäcka och ge feedback till studenter som riskerar att misslyckas med sina kurser, samt skapa personliga visualiseringar för t.ex. tidsförbrukning per delmoment i en kurs, eller betygsfördelning. Denna studie använder sig av ett forskningsinstrument som kallas Student Expectations of Learning Analytics Questionnaire (SELAQ) och ämnar undersöka studenters attityd till 12 olika påståenden relaterade till Learning Analytics. Deltagarna i studien fick således svara på en enkät där de fick gradera hur mycket de instämde med de givna påståendena, under premissen att deras lärosäte hypotetiskt skulle börja implementera en Learning Analytics-tjänst. Resultaten från studien indikerar att SELAQ ger oss bra insikt i vilka förväntningar studenter på svenska lärosäten har på Learning Analytics. Resultaten visar bland annat också att studenterna har låga förväntningar kring de områden som rör feedbacken från Learning Analytics-tjänsten. Mer specifikt har de låg tillit till att undervisande personal kommer att leverera feedbacken på ett tillfredsställande vis till studenterna. Vidare visar resultatet att studenterna har högre förväntningar i frågor gällande inhämtning av samtycke och hantering av personlig data. / Learning Analytics is an area of research which includes collecting, measuring, analysing, and reporting “big data” about students and their learning environment. The purpose is to understand and optimise students’ learning and learning environments. Learning Analytics services can among other things help students gain insight into how they should study to be time-efficient or increase their study performances. Moreover, the services can detect and provide feedback to those students at risk of failing their courses, as well as create personalised visualizations about for example time consumption per parts of a course, or grade distribution. This study uses a research instrument called Student Expectations of Learning Analytics Questionnaire (SELAQ) and aims to examine students' attitudes to 12 different statements related to Learning Analytics. Thus, the participants in the study got to answer a survey where they had to rate how much they agreed with the given statements, based on the hypothetical premise that their university would start to implement a Learning Analytics service. The results from the study indicates that the SELAQ instrument gives us a good understanding about the expectations on Learning Analytics of students in Swedish higher education. The results also show, among other things, that the students have low expectations in areas related to the feedback from the Learning Analytics service. More specifically, they have low confidence that the teaching staff will deliver the feedback to the students in a satisfying way. Furthermore, the results show that the students have higher expectations in matters concerning the obtaining of consent and handling personal data.

Learning Analytics (LA) tools analyse data from Learning Management Systems (LMS) and are being increasingly adopted by Higher Education Institutions (HEI). Analysed LMS data can offer helpful insights into student academic performance and allow predictions based on various metrics to be made, it can also help provide personalised feedback and learning experiences/trajectories for students. Adoption of these systems, however, may be potentially hampered by factors such as lack of user trust, ethical issues regarding data collection, unintended negative consequences of its use and a lack of insight into data accuracy and statistical methods applied by the LA systems. To facilitate the adoption and implementation of LA and also to inform data collection policymaking at HEI it is important that values such as trust between the entity collecting data and the student whose data is being collected be maintained. Stakeholders’ views and values are important factors to be taken into consideration when developing these systems. The study was performed at two Swedish HEI, Stockholm University (SU) and Linnaeus University (Kalmar), and although both institutions utilise LMS and collect student data, at the time of writing, neither have defined strategies for using LA. The study consisted of an initial questionnaire to explore student awareness of data collection and usage and how comfortable students are with data collection and what values they consider important. This questionnaire was distributed to students attending courses at the aforementioned universities. A plugin developed for Moodle was then published to several summer courses at Linnaeus University (LNU) and subsequently followed up by a second questionnaire. The plugin presented students with an analytics dashboard where they could explore graphical representations of data being collected about them by the Moodle LMS. The second questionnaire attempted to gauge student perceptions after they reflected on the data collected by the HEI. This gives a more detailed understanding of the contexts in which data collection and usage are considered acceptable by students. It also offers insight into how students expect to be informed about data collection at their HEI. Further, it reveals how awareness seems to influence acceptance, values and identification of possible uses of LA together with associated ethical concerns. The results from the surveys indicate that awareness significantly impacts the value of privacy, in terms of who has access to and control of the data. Awareness also impacts how worried students are about data collection, an increased awareness of how the data can be used increases how worried students are concerning data collection and the use of this data. Furthermore, this awareness impacts students’ levels of engagement with institutions’ privacy policy documents. The results also reveal that acceptance of data collection and use is highly contextual and depends greatly on who the data is shared with and in what context.

Im Kontext der Hochschul- und Bildungsforschung wird Evaluation in ihrer Gesamtheit als Steuerungs- und Controlling-Instrument eingesetzt, um unter anderem Aussagen zur Qualität von Lehre, Forschung und Administration zu liefern. Auch wenn der Qualitätsbegriff an den Hochschulen bislang noch immer sehr unterschiedlich geführt wird, verfolgen die Beteiligten ein einheitliches Ziel – die Evaluation als zuverlässiges (internes) Präventions- und VorhersageInstrument in den Hochschulalltag zu integrieren. Dass dieses übergeordnete Ziel mit einigen Hürden verbunden ist, liegt auf der Hand und wird in der Literatur bereits vielfältig diskutiert (Benneworth & Zomer 2011; Kromrey 2001; Stockmann & Meyer 2014; Wittmann 2013). Die Evaluationsforschung bietet einen interdisziplinären Forschungszugang. Instrumente und Methoden aus unterschiedlichen (sozialwissenschaftlichen) Disziplinen, die sowohl qualitativer als auch quantitativer Natur sein können, kommen zum Einsatz. Mixed Method/Multi Data–Ansätze gelten dabei – trotz des unstreitbar höheren Erhebungs- und Verwertungsaufwandes – als besonders einschlägig in ihrer Aussagekraft (Döring 2016; Hewson 2007). Allerdings finden (Big) Data Analytics, Echtzeit- und Interaktionsanalysen nur sehr langsam einen Zugang zum nationalen Hochschul- und Bildungssystem. Der vorliegende Beitrag befasst sich mit der Bedeutung von (Big) Data Analytics in der Evaluation. Zum einen werden Herausforderungen und Potentiale aufgezeigt – zum anderen wird der Frage nachgegangen, wie es gelingen kann, (soziale) Daten (automatisiert) auf unterschiedlichen Aggregationsebenen zu erheben und auszuwerten. Es werden am Fallbeispiel der Evaluation von E-Learning in der Hochschullehre geeignete Erhebungsmethoden, Analyseinstrumente und Handlungsfelder vorgestellt. Die Fallstudie wird dabei in den Kontext der Computational Social Science (CSS) überführt, um einen Beitrag zur Entwicklung der Evaluationsforschung im Zeitalter von Big Data und sozialen Netzwerken zu leisten.