Made available in DSpace on 2016-04-29T14:23:36Z (GMT). No. of bitstreams: 1 Antonio Carlos Bento.pdf: 19317474 bytes, checksum: 3345ae7a48f0769e446ab9888361725d (MD5) Previous issue date: 2015-06-18 / The research focuses on the performance evaluation methodology within the models of multiple-choice questions from methodeutics criteria of Fuzzy Logic proposed by Lotfi Zadeh associated with statistical reasoning presented by Santos, Pigari and Tolentino present in the theories of uncertainty discussed by Russell and Norvig and item response theory presented by Fontanive. The study was based on an exploratory and experimental research used questionnaires that were submitted to academic groups. The collected data were compared with a bibliography on the field evaluation, education and logic, resulting in the organization of a methodology that enabled the construction of a prototype computer system to validate the realized and expected outcomes studies. Following the classification s heuristics Clancey it concludes by identifying a set of rules that have been identified as strong enough to guide work and software modeling, aimed at building models of multiple-choice questions based Fuzzy Logic rules. Finally, the results were gathered and presented by means of a system prototype able to characterize the main established rules, as well as studies, being able to be fully utilized in the implementation of performance assessments that may present evidence with expected results, supporting identifying random responses during evaluations / A pesquisa enfoca a metodologia da avaliação de desempenho, dentro dos modelos de questões do tipo múltipla escolha, a partir de critérios metodêuticos da Lógica Fuzzy (Lógica Difusa ou nebulosa) proposta por Zadeh Lotfi associados ao raciocínio estatístico apresentado por Santos, Pigari e Tolentino presente nas teorias da incerteza discutido por Russell e Norvig e teoria da resposta ao item apresentado por Fontanive. O estudo partiu de uma pesquisa exploratória e experimental que utilizou questionários que foram submetidos a grupos de acadêmicos. Os dados colhidos foram cotejados com uma bibliografia sobre o campo de avaliação-educação e lógica, resultando na organização de uma metodologia que possibilitou a construção de um sistema protótipo computacional para a validação dos estudos realizados e resultados esperados. Seguindo a Classificação Heurística de Clancey a pesquisa conclui pela identificação de um conjunto de regras que foram identificadas como suficientemente fortes para nortear trabalhos e de modelagem de softwares, visando a construção de modelos de questões do tipo múltipla escolha baseado em regras de Lógica Fuzzy. Finalmente, os resultados foram reunidos e apresentados por meio de um sistema protótipo capaz de caracterizar as principais regras estabelecidas, bem como os estudos realizados, sendo capaz de serem utilizados plenamente na aplicação de avaliações de desempenho que possam apresentar evidências com resultados esperados, oferecendo suporte a identificação de respostas ao acaso durante avaliações

Nursing programs have traditionally used teacher-developed multiple-choice (MCQ) examinations to prepare students for licensure. Researchers have determined that poorly constructed MCQ tests used as formative and summative evaluations may penalize nursing students and impact progression and retention in nursing programs. The purpose of this exploratory case study was to examine issues related to the use of teacher-developed MCQ examinations as the only method of student assessment in the theory component of nursing courses. The National League for Nursing Core Competencies for Nurse Educators and the revised Bloom's Taxonomy were used as the conceptual frameworks for this study. The Director of the Nursing Program and 9 faculty members participated. Data were collected from a review of documents, 2 focus groups, faculty-maintained diaries, and an interview. During data analysis, categories were identified and themes emerged, revealing the key findings. Using a single method alone to assess student learning limited the opportunity for formative assessment, the ability to assess higher order thinking, and the development of metacognition on the part of students. To assist faculty in creating assessments of student learning that would address these themes, a 3-day faculty professional development project followed by 4 monthly lunch and learn sessions was designed. Providing additional faculty development in assessment methods may promote positive social change as it may ultimately increase the retention of qualified students to meet the demand for registered nurses within the community.

The essay analyses the discursive construction of national identity in the present naturalisation test of the Federal Republic of Germany. The essay includes an overview of immigration to Germany, and a survey of political measures to improve the integration of immigrants. The language and structure of representative multiple-choice questions (and answers) of the naturalisation test are analysed by using the method of critical discourse analysis (CDA). The theoretical background of this study is grounded in cultural studies. The methodological framework consists of a combination of critical discourse analysis and textual analysis. Criteria of the linguistic analysis are: the situational context of the text, thematic roles, deixis, lexical repetitions, modality, coherence (including implicit meanings and presuppositions), intertextuality and interdiscursivity, competence and performance. The analysis shows that national identity is conceptualized by the multiple-choice questions of the naturalisation test. National identity is above all constructed by the German language. One of the qualifications which the examinee has to fulfil is competence in German on the level B1 (of the Common European Framework of Reference for Languages). Linguistic competence is necessary in order to answer the questions. National identity is linguistically created by using alterity. Binary oppositions are constructed by stipulations and presumptions about migrants living in Germany. These oppositions are created by giving three alternative answers, which represent prejudices about foreigners; we is represented by an idealized construction of Germans, and the other is represented by stereotypical assumptions about foreigners. National identity is created by the content of the questions, too. Many questions deal with German laws and standards, which implies that being a German means to be law-abiding. The present study shows that German identity is constructed by language and the construction of alterity.

In this study, I investigate how successful provided response questions, such as multiple choice questions, are as an assessment format compared to the conventional constructed response questions. Based on the literature on mathematics assessment, I firstly identify an assessment taxonomy, consisting of seven mathematics assessment components, ordered by cognitive levels of difficulty and cognitive skills. I then develop a theoretical framework, for determining the quality of a question, with respect to three measuring criteria: discrimination index, confidence index and expert opinion. The theoretical framework forms the foundation against which I construct the Quality Index (QI) model for measuring how good a mathematics question is. The QI model gives a quantitative value to the quality of a question. I also give a visual representation of the quality of a question in terms of a radar plot. I illustrate the use of the QI model for quantifying the quality of mathematics questions in a particular undergraduate mathematics course, in both of the two assessment formats – provided response questions (PRQs) and constructed response questions (CRQs). I then determine which of the seven assessment components can best be assessed in the PRQ format and which can best be assessed in the CRQ format. In addition I also investigate student preferences between the two assessment formats. / Thesis (PhD)--University of Pretoria, 2009. / Mathematics and Applied Mathematics / unrestricted

Academics encounter problems with the selection, evaluation, testing and implementation of e-assessment software tools. The researcher experienced these problems while adopting e-assessment at the university where she is employed. Hence she undertook this study, which is situated in schools and departments in Computing-related disciplines, namely Computer Science, Information Systems and Information Technology at South African Higher Education Institutions. The literature suggests that further research is required in this domain. Furthermore, preliminary empirical studies indicated similar disabling factors at other South African tertiary institutions, which were barriers to long-term implementation of e-assessment. Despite this, academics who are adopters of e-assessment indicate satisfaction, particularly when conducting assessments with large classes. Questions of the multiple choice genre can be assessed automatically, leading to increased productivity and more frequent assessments. The purpose of this research is to develop an evaluation framework to assist academics in determining which e-assessment tool to adopt, enabling them to make more informed decisions. Such a framework would also support evaluation of existing e-assessment systems. The underlying research design is action research, which supported an iterative series of studies for developing, evaluating, applying, refining, and validating the SEAT (Selecting and Evaluating an e-Assessment Tool) Evaluation Framework and subsequently an interactive electronic version, e-SEAT. Phase 1 of the action research comprised Studies 1 to 3, which established the nature, context and extent of adoption of e-assessment. This set the foundation for development of SEAT in Phase 2. During Studies 4 to 6 in Phase 2, a rigorous sequence of evaluation and application facilitated the transition from the manual SEAT Framework to the electronic evaluation instrument, e-SEAT, and its further evolution. This research resulted in both a theoretical contribution (SEAT) and a practical contribution (e-SEAT). The findings of the action research contributed, along with the literature, to the categories and criteria in the framework, which in turn, contributed to the bodies of knowledge on MCQs and e-assessment. The final e-SEAT version, the ultimate product of this action research, is presented in Appendix J1. For easier reference, the Appendices are included on a CD, attached to the back cover of this Thesis.. / Computing / PhD. (Information Systems)

A common evaluation method for students in the context of reading comprehension is the use of Multiple Choice Questions. A student must read a text and a question, and then choose the correct answer from a set of options, one of which one is the correct answer, and the other options are wrong. The wrong options are called distractors. Creating Multiple Choice Question exams is time-consuming, and a task that is open for automation. Distractor Generation is the task of generating wrong, but plausible options for Multiple Choice Questions. It is a task that can be addressed with Machine Learning and Large Language Models. As this task has been addressed in languages such as English, and Swedish, this work addresses the task for the Spanish language. This work achieves 3 objectives. The first one is the creation of a Multiple Choice Question dataset in Spanish with distractors, by manually tagging distractors from the dataset SQuAD-es. The newly created dataset with distractors is called SQuAD-es-dist. The second one is automatically generating distractors with machine learning methods. A BERT model is fine-tuned to generate distractors, and a GPT model is used through zero-shot learning to generate distractors. The third one is a human study on the generated distractors to evaluate the plausibility and usability of the distractors. Although both methods show to be effective, yet not perfect, at generating distractors, the GPT model shows better applicability and a higher capacity to confuse students in the task of Distractor Generation. / En vanlig utvärderingsmetod för studenter i samband med läsförståelse är användningen av flervalsfrågor. En elev måste läsa en text och en fråga, och sedan välja rätt svar från en uppsättning alternativ, varav ett är rätt svar och de andra alternativen är fel. De felaktiga alternativen kallas distraktorer. Att skapa prov med flervalsfrågor är tidskrävande och en uppgift som är öppen för automatisering. Distraktorgenerering är uppgiften att generera felaktiga, men rimliga alternativ för flervalsfrågor. Det är en uppgift som kan lösas med maskininlärning och stora språkmodeller. Eftersom denna uppgift har behandlats på språk som engelska och svenska, behandlar detta arbete uppgiften för det spanska språket. Detta arbete uppnår 3 mål. Den första är skapandet av ett dataset med flervalsfrågor på spanska med distraktorer, genom manuell taggning av distraktorer från datasetet SQuAD-es. Det nyskapade datasetet med distraktorer kallas SQuAD-es-dist. Den andra metoden är att automatiskt generera distraktorer med maskininlärningsmetoder. En BERT-modell finjusteras för att generera distraktorer, och en GPT-modell används genom zero-shot-inlärning för att generera distraktorer. Den tredje metoden är en mänsklig studie av de genererade distraktorerna för att utvärdera hur rimliga och användbara distraktorerna är. Även om båda metoderna visar sig vara effektiva, men inte perfekta, för att generera distraktorer, visar GPT-modellen bättre tillämpbarhet och en högre kapacitet att förvirra studenter i uppgiften att generera distraktorer / Para evaluar a alumnos en el contexto de comprensión de lectura se usan las preguntas de opción múltiple. El alumno debe leer un texto y una pregunta y, a continuación, elegir la respuesta correcta entre un conjunto de opciones, una de las cuales es la respuesta correcta y las demás opciones son incorrectas. Las opciones incorrectas se denominan distractores. La creación de exámenes con preguntas de opción múltiple requiere mucho tiempo, y es una tarea susceptible a la automatización. La Generación de Distractores es la tarea de generar opciones erróneas, pero plausibles, para Preguntas de Elección Múltiple. Es una tarea que puede abordarse con Aprendizaje Automático y Grandes Modelos de Lenguaje. Dado que esta tarea ha sido explorada en idiomas como el inglés, y el sueco, este trabajo aplica la tarea para el idioma español. Este trabajo alcanza 3 objetivos. El primero es la creación de un conjunto de datos de preguntas de respuesta múltiple en español con distractores, etiquetando manualmente los distractores del conjunto de datos SQuAD-es. El nuevo conjunto de datos con distractores se denomina SQuAD-es-dist. La segunda consiste en generar distractores automáticamente con métodos de aprendizaje automático. Se entrena y ajusta un modelo BERT para generar distractores y se utiliza un modelo GPT mediante ”zeroshot learning” para generar distractores. El tercero es un estudio humano sobre los distractores generados para evaluar la aplicabilidad y usabilidad de los distractores. Aunque ambos métodos muestran ser eficaces, pero no perfectos, en la generación de distractores, el modelo GPT muestra una mejor aplicabilidad y una mayor capacidad para confundir a los estudiantes en la tarea de Generación de Distractores.

Artificial Intelligence (AI) is a useful tool in education, with the potential to enhance teaching through the use of digital learning environments (DLEs) such as intelligent tutoring systems (ITSs). This thesis investigates the development and evaluation of an AI-assisted web-based platform for pure question-based learning (pQBL) in upper secondary school mathematics. The study aims to assess the prototype's effects compared to traditional classroom instruction and the quality of AI-generated multiple-choice questions (MCQs) using OpenAI’s GPT-4. To do this, the following research questions were formulated: (1) How can an AI-assisted platform for pQBL be designed and implemented? (2) To what extent do GPT-4 generated MCQs meet quality criteria? (3) How do learning outcomes compare between students using AI-assisted pQBL materials and those receiving traditional instruction? The study utilized principles of Question Construction and Prompt Engineering, ensuring AI-generated MCQs adhered to predefined quality standards. Human evaluation by a Teacher Quality Control panel refined and validated the generated MCQs. The effectiveness of the developed prototype was tested using a pretest/posttest design where score changes were analyzed using paired-samples t-tests. The findings indicate that generating high-quality MCQs remains a challenge. Human evaluation is essential to ensure educational efficacy, as only a fraction of the AI-generated MCQs met quality standards. User testing revealed comparable learning outcomes between the pQBL and control groups, with statistically significant results, though limitations in the pretest/posttest design may have influenced these findings. This research highlights the potential and limitations of AI in education, suggesting the need for further studies on AI-assisted tools, including exploring other language models and conducting longitudinal assessments to optimize learning outcomes. / AI utgör en viktig resurs inom utbildning och har potential att förbättra undervisningen genom digitala lärmiljöer (DLE), såsom intelligenta handledningssystem (ITS). Detta examensarbete undersöker utvecklingen och utvärderingen av en AI-stödd webbaserad plattform för rent frågebaserat lärande (pQBL) i matematik på gymnasiet. Studien syftar till att bedöma plattformens effektivitet jämfört med traditionell klassrumsundervisning och att utvärdera kvaliteten på flervalsfrågor som genereras av OpenAI:s GPT-4. För att uppnå detta formulerades följande forskningsfrågor: (1) Hur kan en AI-stödd plattform för pQBL designas och implementeras? (2) I vilken utsträckning uppfyller GPT-4- genererade flervalsfrågor fördefinierade kvalitetskriterier? (3) Hur skiljer sig inlärningsresultaten mellan elever som använder AI-stött pQBL-material och de som får traditionell klassrumsundervisning? Studien tillämpade principer för fråge- och promptkonstruktion för att säkerställa att de AI-genererade flervalsfrågorna följde fördefinierade kvalitetsstandarder. En panel av lärare genomförde mänsklig utvärdering för att förbättra och validera de genererade flervalsfrågorna. Effekten av den utvecklade prototypen testades genom en förtest/eftertest-design där poängförändringen analyserades genom parvisa t-tester. Resultaten indikerar att det finns utmaningar i att generera högkvalitativa flervalsfrågor. Mänsklig utvärdering är nödvändig för att säkerställa kvaliteten hos de AI-genererade flervalsfrågornas eftersom endast en mindre del av dem uppfyllde kvalitetskraven. Användartester visade jämförbara inlärningsresultat mellan pQBL-gruppen och kontrollgruppen med statistiskt signifikanta resultat även om begränsningar i förtest/eftertest-designen kan ha påverkat dessa resultat. Denna forskning belyser AI:s potential och begränsningar inom utbildning och föreslår behovet av ytterligare studier kring AI-drivna verktyg, inklusive undersökning av andra språkmodeller och genomförande av longitudinella studier för att optimera inlärningsresultaten.

This study investigated the effective use of online Multiple-Choice Questions (MCQs) with immediate formative feedback, and the granting of partial credit for correct second or third chance answers when assessing and assisting students’ conceptual learning at higher cognitive levels. The research sample comprised first year engineering science students at the Tshwane University of Technology (TUT), Pretoria campus. The differences between using online MCQ-assessment for problem-solving calculations and using constructed written questions (CRQs)1 in the assessment of problem-solving calculations were explored. Furthermore, the differences between the assessment of problem-solving calculations using online MCQs without immediate formative feedback, and with immediate formative feedback and the granting of partial credit were analysed. The findings revealed that students’ marks were lower when answering problem-solving calculations using online MCQs without immediate formative feedback than when answering the same questions using CRQs. This clearly indicates that using online MCQs without immediate formative feedback is not effective in assessing scientific problem-solving calculations. Alternatively, online MCQs proved effective in assessing problem-solving calculations when immediate formative feedback and partial credit were employed. The statistical analysis showed that students performed significantly better when immediate formative feedback was given and partial credit was granted for correct second or third attempts. This was due to online MCQs utilising immediate formative feedback, which made it possible to grant partial credit when students chose the correct answers after feedback. This showed that online MCQs with immediate formative feedback and partial credit being granted can be an effective assessment tool for scientific problem-solving calculations. It increases performance and supports learning from assessment. Students can thus correct their calculations whilst in the process of doing them. / Science and Technology Education / M. Ed. (Science Education)