This work consists in proposing a methodology to improve automated evaluation of the ergonomic quality of web sites (with special focus on usability and accessibility) by static analysis of HTML code of their pages using the evaluation technique called guideline review. This methodology will be articulated around: - A framework that defines a systematic and consistent way for structuring guidelines in order to enable their automatic evaluation; - A guideline definition language (GDL) able to express guideline information in a sufficiently rich manner to enable an evaluation engine to perform automated evaluation of any GDL-compliant guideline; - Tools to support the proposed methodology. The tools are (1) a structuring tool (editor) to enable the specification and manipulation of guidelines structures (structured following the framework and expressed in a GDL-compliant form) and (2) an evaluation tool which uses the guidelines to conduct the evaluation of Web sites.

This thesis focuses on web site evaluation by using structural evaluation and scope of business based content comparison. Firstly, web site measurement techniques and evaluation methods are reviewed. Then a structural evaluation and content comparison method introduced. This thesis also includes a web based implementation of these methods for evaluating web sites which is partially automated for structural evaluation method.

The thesis investigates the automatic and semi-automatic evaluation of programming assignments in teaching of programming, exams and competitions. Manual evaluation of programming assignments is still dominant in programming teaching in Lithuania. In some contexts (i.e. programming competitions, basics of programming and algorithms course) there is widely applied automatic and semi-automatic evaluation of programming tasks based on black-box testing approach. However, automatic evaluation based on black-box testing is criticized for its inability to demonstrate the strengths and the weaknesses of the program being evaluated. Automatic evaluation is not capable of identifying the level of achievement in incomplete programs. The thesis addresses the problem of evaluation of practical programming assignments in Information Technology maturity exam (IT VBE). This problem involves requirements to evaluate reliably thousands of programming assignment solutions in a reasonable amount of time. The thesis presents analysis of systems for automatic evaluation of programming tasks and appropriate types of assignments. The analysis led to the improvement of the evaluation method. The most important result of the research is the proposed way to improve the semi-automatic evaluation and testing method by increasing its interactivity. A semi-automatic assessment system is expanded by adding the components responsible for the modification of submitted programs, monitoring changes in the... [to full text] / Disertacijoje nagrinėjamas automatinis ir pusiau automatinis programavimo užduočių vertinimas programavimo mokyme, egzaminuose ir varžybose. Šiuo metu informatikos mokyme Lietuvoje vis dar dominuoja rankinis programavimo užduočių vertinimas. Atskirose švietimo srityse (programavimo varžybos, algoritmavimo kursai) išplito automatinis ir pusiau automatinis programavimo užduočių vertinimas, paremtas juodosios dėžės testavimo principu. Tačiau automatinio vertinimo pagrindu gauti įverčiai kritikuojami kaip neatskleidžiantys visų pateiktos programos privalumų ir trūkumų. Ypatingai dideli vertinimo netikslumai gaunami vertinant nebaigtas programas. Disertacijoje sprendžiama Informacinių technologijų valstybinio brandos egzamino (IT VBE) praktinių programavimo užduočių vertinimo problema – siekiama gauti patikimus įvertinimus dideliam pateiktų programavimo užduočių sprendimų kiekiui per priimtiną laiką. Išanalizuotos automatinės programavimo užduočių vertinimo sistemos bei joms tinkami užduočių tipai atvedė prie idėjos tobulinti vertinimo metodą. Svarbiausias disertacijos rezultatas – pasiūlytas būdas tobulinti pusiau automatinį vertinimo ir testavimo metodą didinant jo interaktyvumą. Tam pusiau automatinė vertinimo sistema turi būti papildoma komponentais, leidžiančiais modifikuoti pateiktą programą, stebėti modifikuotos programos funkcionalumo pokyčius. Pasiūlytas patobulintas vertinimo metodas išbandytas praktiškai, sukurta IT VBE praktinių užduočių vertinimo sistema. Taikant šį... [toliau žr. visą tekstą]

Disertacijoje nagrinėjamas automatinis ir pusiau automatinis programavimo užduočių vertinimas programavimo mokyme, egzaminuose ir varžybose. Šiuo metu informatikos mokyme Lietuvoje vis dar dominuoja rankinis programavimo užduočių vertinimas. Atskirose švietimo srityse (programavimo varžybos, algoritmavimo kursai) išplito automatinis ir pusiau automatinis programavimo užduočių vertinimas, paremtas juodosios dėžės testavimo principu. Tačiau automatinio vertinimo pagrindu gauti įverčiai kritikuojami kaip neatskleidžiantys visų pateiktos programos privalumų ir trūkumų. Ypatingai dideli vertinimo netikslumai gaunami vertinant nebaigtas programas. Disertacijoje sprendžiama Informacinių technologijų valstybinio brandos egzamino (IT VBE) praktinių programavimo užduočių vertinimo problema – siekiama gauti patikimus įvertinimus dideliam pateiktų programavimo užduočių sprendimų kiekiui per priimtiną laiką. Išanalizuotos automatinės programavimo užduočių vertinimo sistemos bei joms tinkami užduočių tipai atvedė prie idėjos tobulinti vertinimo metodą. Svarbiausias disertacijos rezultatas – pasiūlytas būdas tobulinti pusiau automatinį vertinimo ir testavimo metodą didinant jo interaktyvumą. Tam pusiau automatinė vertinimo sistema turi būti papildoma komponentais, leidžiančiais modifikuoti pateiktą programą, stebėti modifikuotos programos funkcionalumo pokyčius. Pasiūlytas patobulintas vertinimo metodas išbandytas praktiškai, sukurta IT VBE praktinių užduočių vertinimo sistema. Taikant šį... [toliau žr. visą tekstą] / The thesis investigates the automatic and semi-automatic evaluation of programming assignments in teaching of programming, exams and competitions. Manual evaluation of programming assignments is still dominant in programming teaching in Lithuania. In some contexts (i.e. programming competitions, basics of programming and algorithms course) there is widely applied automatic and semi-automatic evaluation of programming tasks based on black-box testing approach. However, automatic evaluation based on black-box testing is criticized for its inability to demonstrate the strengths and the weaknesses of the program being evaluated. Automatic evaluation is not capable of identifying the level of achievement in incomplete programs. The thesis addresses the problem of evaluation of practical programming assignments in Information Technology maturity exam (IT VBE). This problem involves requirements to evaluate reliably thousands of programming assignment solutions in a reasonable amount of time. The thesis presents analysis of systems for automatic evaluation of programming tasks and appropriate types of assignments. The analysis led to the improvement of the evaluation method. The most important result of the research is the proposed way to improve the semi-automatic evaluation and testing method by increasing its interactivity. A semi-automatic assessment system is expanded by adding the components responsible for the modification of submitted programs, monitoring changes in the... [to full text]

京都大学 / 0048 / 新制・課程博士 / 博士(情報学) / 甲第17921号 / 情博第503号 / 新制||情||89(附属図書館) / 30741 / 京都大学大学院情報学研究科社会情報学専攻 / (主査)教授 石田 亨, 教授 田中 克己, 教授 黒橋 禎夫 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM

La enseñanza y evaluación automática a través de un sistema Computer Based Assessment (CBA) requiere de software especializado que se adapte a la tipología de actividades a tratar y evaluar. En esta tesis se ha desarrollado un entorno CBA que facilita el aprendizaje y evaluación de los principales temas de una asignatura de bases de datos. Para ello se han analizado las herramientas existentes en cada uno de estos temas (Diagramas Entidad/Relación, diagramas de clases, esquemas de bases de datos relacionales, normalización, consultas en álgebra relacional y lenguaje SQL) y para cada uno de ellos se ha analizado, diseñado e implementado un módulo de corrección y evaluación automática que aporta mejoras respecto a los existentes. Estos módulos se han integrado en un mismo entorno al que hemos llamado ACME-DB. / Teaching and automatic evaluation through Computer Based Assessment (CBA) requires of specialized software tools capable to correct different type of activities. In this work, we have analyzed the main topics of a database course (entity/ relationship diagrams, class diagrams, relational database schemas, normalization, relational algebra and SQL) and for each one of them we have reported the state of art on CBA tools. Afterwards, we have proposed a new CBA environment for teaching and learning database. This environment, denoted ACME-DB, is composed of different modules capable to automatically correct and evaluate exercises of main topics of a database course providing improvements over existing tools.

This study introduces an automated evaluation method for large language model (LLM) based chatbots in educational settings, utilizing LLM-as-a-Judge to assess their performance. Our results demonstrate the efficacy of this approach in evaluating the accuracy of three LLM-based chatbots (Llama 3 70B, ChatGPT 4, Gemini Advanced) across two subjects: history and biology. The analysis reveals promising performance across different subjects. On a scale from 1 to 5 describing the correctness of the judge itself, the LLM judge’s average scores for correctness when evaluating each chatbot on history related questions are 3.92 (Llama 3 70B), 4.20 (ChatGPT 4), 4.51 (Gemini Advanced); for biology related questions, the average scores are 4.04 (Llama 3 70B), 4.28 (ChatGPT 4), 4.09 (Gemini Advanced). This underscores the potential of leveraging the LLM-as-a-judge strategy to evaluate the correctness of responses from other LLMs.

O computador tem sido empregado na educação praticamente desde seu surgimento e a literatura tem apontado como vantagens de sua incorporação no processo de ensino-aprendizagem, seu potencial de promoção de interatividade e de resposta rápida (retroação). Mais recentemente, com a grande popularização da Web, o uso de Sistemas Gerenciadores de Cursos (SGC) passou a ser ferramenta necessária para personalizar o aprendizado do aluno e para sintetizar informações ao professor. Além disso os SGC viabilizaram o surgimento de uma grande quantidade de cursos na modalidade de Educação a Distância (EAD). Dentre os SGC um que merece destaque é o Moodle, em função de ser um sistema livre e por adotar uma arquitetura modular que permite a incorporação de novas ferramentas. Entretanto, nota-se atualmente uma carência no Moodle de ferramentas que proporcionem aprendizado interativo e que eventualmente permita retroação imediata. Deste modo, a proposta deste trabalho é o enriquecimento do Moodle, apresentando um novo pacote, o iTarefa, que possibilita o gerenciamento de atividades interativas. O iTarefa possibilita a incorporação de qualquer applet Java, desde que este esteja na forma de um Módulo de Aprendizagem Interativa (iMA). Neste trabalho serão apresentados, além do pacote iTarefa, os resultados de seu uso em disciplina de graduação e alguns minicursos para professores e alunos do ensino fundamental e médio. / The digital computer has been used in education almost since its beginning and the literature has pointed out several advantages of its use in the teaching-learning process, mainly related to its potential to promote interactivity and quick feedback. More recently, with the popularization of the Web, the use of Learning Management Systems (LMS) has become a fundamental tool to customize student learning and to synthesize information for the teacher. In addition, the LMS made possible the emergence of a large number of courses under the model of Distance Learning (or e-learning). Despite the several options of LMS, one that is worthwhile is the Moodle system. Two of its most valuable characteristics is that it is free software and its modular architecture allows the incorporation of new tools. However, there is currently a lack of tools in Moodle to promote interactive learning and eventually allow feedback. Thus, the purpose of this work is to enrich interactivity in Moodle presenting a new package, the iAssign, which enables the management of interactive activities. The iAssign allows the incorporation of any Java applet, since it is in a form of an Interactive Learning Module (iLM). This work, besides to present the iAssign package, it also presents the results of its use in undergraduate course and some short-courses for teachers and students of elementary and high school.

O computador tem sido empregado na educação praticamente desde seu surgimento e a literatura tem apontado como vantagens de sua incorporação no processo de ensino-aprendizagem, seu potencial de promoção de interatividade e de resposta rápida (retroação). Mais recentemente, com a grande popularização da Web, o uso de Sistemas Gerenciadores de Cursos (SGC) passou a ser ferramenta necessária para personalizar o aprendizado do aluno e para sintetizar informações ao professor. Além disso os SGC viabilizaram o surgimento de uma grande quantidade de cursos na modalidade de Educação a Distância (EAD). Dentre os SGC um que merece destaque é o Moodle, em função de ser um sistema livre e por adotar uma arquitetura modular que permite a incorporação de novas ferramentas. Entretanto, nota-se atualmente uma carência no Moodle de ferramentas que proporcionem aprendizado interativo e que eventualmente permita retroação imediata. Deste modo, a proposta deste trabalho é o enriquecimento do Moodle, apresentando um novo pacote, o iTarefa, que possibilita o gerenciamento de atividades interativas. O iTarefa possibilita a incorporação de qualquer applet Java, desde que este esteja na forma de um Módulo de Aprendizagem Interativa (iMA). Neste trabalho serão apresentados, além do pacote iTarefa, os resultados de seu uso em disciplina de graduação e alguns minicursos para professores e alunos do ensino fundamental e médio. / The digital computer has been used in education almost since its beginning and the literature has pointed out several advantages of its use in the teaching-learning process, mainly related to its potential to promote interactivity and quick feedback. More recently, with the popularization of the Web, the use of Learning Management Systems (LMS) has become a fundamental tool to customize student learning and to synthesize information for the teacher. In addition, the LMS made possible the emergence of a large number of courses under the model of Distance Learning (or e-learning). Despite the several options of LMS, one that is worthwhile is the Moodle system. Two of its most valuable characteristics is that it is free software and its modular architecture allows the incorporation of new tools. However, there is currently a lack of tools in Moodle to promote interactive learning and eventually allow feedback. Thus, the purpose of this work is to enrich interactivity in Moodle presenting a new package, the iAssign, which enables the management of interactive activities. The iAssign allows the incorporation of any Java applet, since it is in a form of an Interactive Learning Module (iLM). This work, besides to present the iAssign package, it also presents the results of its use in undergraduate course and some short-courses for teachers and students of elementary and high school.

The aim of this thesis is to design an automated scanning system for components for the purpose of inspecting their dimensions and tolerances. The theoretical introduction provides the reader with basic information on the topic of 3D scanning. The work also includes the design of own scanning system. The greatest attention is paid to the design of a program using the PCL library. The aim of the program is automatic processing of data from a 3D scanner and evaluation of the required dimensions of the scanned component. The final part of the work is devoted to testing the proposed solution.