This qualitative multi-method study investigates potential construct-related outcomes of using a high-stakes standardized literacy test that is based on a limited construct. This study presents comprehensive construct and outcomes analyses of the Ontario Secondary School Literacy Test (OSSLT), as one example of a high-stakes literacy testing program, focusing on critical literacies in particular. Critical literacies are shown to be underrepresented in the test’s construct sample despite being valued in the relevant educational domain being measured. The two central research questions shaping this study are as follows: (1) What are students learning about critical literacies in an educational context that includes the OSSLT?; (2) How do students perceive the OSSLT to be contributing to, and/or hindering, their development of critical literacy skills? This study is structured as an arts-informed multiple case study. Participants included ten Grade 11 and 12 students who contributed to the study through multiple research activities, including interviews and group conversations, questionnaires, activity handouts, as well as journaling, photography, and graffiti walls activities. The findings of the outcomes analysis (1) identify limitations with regards to students’ understandings of literacy, including being unfamiliar with a concept of critical literacies; (2) demonstrate that students align their understandings of literacies with the reading and writing skills that are valued on the high-stakes literacy test while also wondering what else might count as literacy; and (3) suggest that the OSSLT is perceived by students to be influencing their literate identities, their relationships with literacies, and their learning trajectories. As an original contribution to knowledge, this study demonstrates how an in-depth analysis of a test’s literacy construct can be performed, and this study presents a qualitative multi-method methodology for conducting research into the outcomes of literacy education in an educational context that includes a high-stakes literacy test.