This dissertation involves a holistic and interconnected examination of the validity, reliability, and fairness of the Online Research and Comprehension Assessment (ORCA). The ORCA is a large-scale assessment of New Literacies that challenges Grade 7 students to solve research problems (e.g., Does playing video games harm your eyes?) by locating, critically evaluating, and synthesizing online information in order to communicate their results in online genres such as email and wiki. My goal in this study was to understand how ORCA test score data should be used and interpreted, in what context, and for what purposes. A secondary goal was to examine the cognitive and metacognitive processes required to support research writing in online contexts.
The study involved three interconnected phases. The first was a systematic, mixed methods literature review of 101 peer-reviewed texts from the last 50 years in order to articulate the construct underlying the ORCA. Finding no construct in the literature that considered the important ways in which the Internet has changed the construct of writing, I opted to conceptualize one of my own. This construct also serves as the theoretical framework for the rest of the dissertation.
In the next phase of the study, I again explored the construct underlying the ORCA, but this time via a mixed methods investigation of the response processes—both cognitive and metacognitive—elicited by the ORCA. By observing both expert and novice participants’ response processes, I analyzed the extent to which the tasks and types of responses elicited by the ORCA fit the intended construct. Further, by observing response processes, I was also able to analyze construct underrepresentation and construct-irrelevant variance, which are fundamental to the ORCA’s appraisal. The results suggested that there are complex and sophisticated cognitive and metacognitive processes underlying the ORCA and online research writing more generally, many of which are unique to online contexts. Further, both quantitative and qualitative results suggest significant differences between novice and expert groups.
The third phase of this research concludes with an integrated consideration of the ORCA’s validity, reliability, and fairness. Here, I analyzed data collected from the previous two phases; previous validation work done on the ORCA by my colleagues; and new forms of validation evidence collected for this study. I did so in order to build a comprehensive validity argument to demonstrate the ways in which ORCA test scores should be used and interpreted, and the consequences which follow. I used cued retrospective reporting, semi-structured interviews, Venn diagrams, surveys, and writing artefacts to investigate the response processes elicited by the ORCA and to compare and contrast those to the writing practices that participants used in their school, work, and/or personal lives. I also completed an extensive analysis of the sample of observations permitted by the ORCA juxtaposing those with the target domain. Results of this study indicate that the ORCA provides an important form of assessment data regarding 21st century literacies previously neglected on traditional assessments. Limitations of the ORCA such as construct-irrelevant variance and construct underrepresentation are also explored. The results of the study suggest how the ORCA could be re-designed to improve the validity of inferences made.
| Identifer | oai:union.ndltd.org:uottawa.ca/oai:ruor.uottawa.ca:10393/38696 |
| Date | 14 January 2019 |
| Creators | Corrigan, Julie A. |
| Contributors | Slomp, David, Lévesque, Stéphane G. |
| Publisher | Université d'Ottawa / University of Ottawa |
| Source Sets | Université d’Ottawa |
| Language | English |
| Detected Language | English |
| Type | Thesis |
| Format | application/pdf |
Page generated in 0.003 seconds