University of Crete (UOC)
Κατασκευή μηχανογραφημένης προσαρμοστικής εξέτασης μέσω των στατιστικών υποδειγμάτων της θεωρίας απόκρισης ερωτήματος (IRT)
Abstract
dc:descriptionIntroduction and Purpose: Traditional assessment methods (Paper and Pencil, P&P) in Higher Education present significant limitations regarding accuracy, efficiency, and psychological impact, particularly in subjects like Statistics. The purpose of this doctoral dissertation was the holistic investigation of Computerized Adaptive Testing (CAT) as an alternative, examining psychometric, psychological, and metacognitive dimensions. Methodology: The research was conducted in two main phases. In the first phase (Calibration), an item bank was developed through the psychometric analysis of data from a large initial sample (N=680), estimating the Item Response Theory (IRT) parameters (difficulty, discrimination) of the items based on the 2-Parameter Logistic (2PL) model. In the second phase (Main Study), 213 undergraduate students from a Department of Education participated in a quasi-experimental, within-subjects, counterbalanced design. Participants were assessed using both a traditional (P&P) test and the calibrated CAT system. Data collected included performance, time, anxiety, and perceptions. Phase 2 analyses included comparative tests, Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA), as well as Logistic and Linear Regression. Results: The findings demonstrated the clear superiority of CAT. Psychometrically, it was significantly more efficient (83% time reduction), achieving accuracy with ~10 items. Students achieved significantly higher performance in CAT (M=5.40) compared to P&P (M=1.92), as the algorithm overcame the "floor effects" of the P&P test. Psychologically, CAT induced significantly less anxiety, increased self-confidence, and was preferred by the majority (55.4%). Psychometric analysis (EFA and CFA) revealed and validated a three-factor model of the CAT experience, distinguishing the dimensions of "Practicality" (F1), "Trust" (F2), and "Affect" (F3), with excellent model fit to the data (e.g., CFI=.976, SRMR=.036). Logistic Regression (Push-Pull model, AUC=0.968) showed that preference for CAT is predicted by low P&P performance and high P&P anxiety (Push factors), as well as by high Trust (F2) in the CAT method and more time spent in CAT (Pull factors). Linear Regression (Adjusted R²=0.588) indicated that performance in CAT is strongly predicted by knowledge (as measured by P&P performance), test time in the CAT, number of items (negatively), and the student's perceived performance in the CAT. The low predictive power of subjective experience alone supports the objectivity of CAT measurement, as performance was determined almost exclusively by objective factors (knowledge, behavior) rather than by students' affective perceptions. Finally, the research highlighted the trade-off between psychometric optimization and the psychological experience of the examinees, documenting "Metacognitive Pessimism" (systematic underestimation of performance in CAT) and the "Paradox of the High-Ability Student" (lower preference despite high performance). Conclusions and Contribution: This dissertation substantiates that CAT is a superior assessment method, capable of enhancing accuracy and fairness while reducing anxiety and increasing self-confidence of the examinees. The original contribution lies in the development and initial psychometric validation (via EFA and CFA) of the CAT experience model, the empirical demonstration of the psychological and metacognitive consequences of algorithmic design, and the quantification of factors (Push-Pull) determining acceptance and success in adaptive environments.
Degree
thesis:*- Grantor dc:publisher
- University of Crete (UOC)
- Year dc:date
- 2025
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Eleftherakis, Georgios
- Ελευθεράκης, Γεώργιος
Subjects
dc:subject × 42- Μηχανογραφημένη Προσαρμοστική Αξιολόγηση
- Θεωρία Απόκρισης στο Στοιχείο
- Αξιολόγηση στη Στατιστική
- Μεταγνώση
- Μαθηματικό άγχος
- Εκπαιδευτική αξιολόγηση
- Μεταγνωσιακή Απαισιοδοξία
- Εκπαιδευτική τεχνολογία
- Μοντέλο Ώθησης-Έλξης
- Ψυχομετρική Ανάλυση
- Τριτοβάθμια εκπαίδευση
- Επιβεβαιωτική παραγοντική ανάλυση
- Λογιστική παλινδρόμηση
- Γραμμική παλινδρόμηση
- Εμπειρία εξέτασης φοιτητών
- Computerized Adaptive Testing - CAT
- Item Response Theory (IRT)
- Assessment in Statistics
- Metacognition
- Math anxiety
- educational assessment
- Metacognitive Pessimism
- Educational technology
- Push-Pull Model
- Psychometric Analysis
- Higher education
- Confirmatory factor analysis (CFA)
- Logistic regression analysis
- Linear regression
- Student examination experience
- Μαθηματικά
- Φυσικές Επιστήμες
- Στατιστική και Πιθανότητες
- Εκπαίδευση
- Κοινωνικές Επιστήμες
- Επιστημονικοί τομείς της εκπαίδευσης
- Mathematics
- Natural Sciences
- Statistics and Probability
- Education
- Social Sciences
- Education, Scientific Disciplines
Rights
- Language dc:language
- gre
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/61652