Policy guidelines of the Education Quality and Accountability Office (EQAO) state that accommodations and participation decisions on the Ontario Secondary School Literacy Test (OSSLT) do not threaten validity. However, these issues are contentious in American large-scale testing. New approaches integrate test access, administration, accommodation and participation within a unified fairness and validity construct. The current study, based on demographic and outcome data for the entire population of OSSLT-eligible students from 2006 to 2009, demonstrates changing patterns in accommodations and participation decisions across schools and years. In particular, English language learners are found to be considerably underrepresented among students receiving special needs accommodations. This has implications for the valid interpretation and fair use of test scores. Recommendations are proposed for improving fairness, consistency, and validity in administering accommodations and participation.

Policy guidelines of the Education Quality and Accountability Office (EQAO) state that accommodations and participation decisions on the Ontario Secondary School Literacy Test (OSSLT) do not threaten validity. However, these issues are contentious in American large-scale testing. New approaches integrate test access, administration, accommodation and participation within a unified fairness and validity construct. The current study, based on demographic and outcome data for the entire population of OSSLT-eligible students from 2006 to 2009, demonstrates changing patterns in accommodations and participation decisions across schools and years. In particular, English language learners are found to be considerably underrepresented among students receiving special needs accommodations. This has implications for the valid interpretation and fair use of test scores. Recommendations are proposed for improving fairness, consistency, and validity in administering accommodations and participation.

This study built and supported arguments for the use of diagnostic assessment in English as a second language (ESL) academic writing. In the two-phase study, a new diagnostic assessment scheme, called the Empirically-derived Descriptor-based Diagnostic (EDD) checklist, was developed and validated for use in small-scale classroom assessment. The checklist assesses ESL academic writing ability using empirically-derived evaluation criteria and estimates skill parameters in a way that overcomes the problems associated with the number of items in diagnostic models. Interpretations of and uses for the EDD checklist were validated using five assumptions: (a) that the empirically-derived diagnostic descriptors that make up the EDD checklist are relevant to the construct of ESL academic writing; (b) that the scores derived from the EDD checklist are generalizable across different teachers and essay prompts; (c) that performance on the EDD checklist is related to performance on other measures of ESL academic writing; (d) that the EDD checklist provides a useful diagnostic skill profile for ESL academic writing; and (e) that the EDD checklist helps teachers make appropriate diagnostic decisions and has the potential to positively impact teaching and learning ESL academic writing. Using a mixed-methods research design, four ESL writing experts created the EDD checklist from 35 descriptors of ESL academic writing. These descriptors had been elicited from nine ESL teachers’ think-aloud verbal protocols, in which they provided diagnostic feedback on ESL essays. Ten ESL teachers utilized the checklist to assess 480 ESL essays and were interviewed about its usefulness. Content reviews from ESL writing experts and statistical dimensionality analyses determined that the underlying structure of the EDD checklist consists of five distinct writing skills: content fulfillment, organizational effectiveness, grammatical knowledge, vocabulary use, and mechanics. The Reduced Reparameterized Unified Model (Hartz, Roussos, & Stout, 2002) then demonstrated the diagnostic quality of the checklist and produced fine-grained writing skill profiles for individual students. Overall teacher evaluation further justified the validity claims for the use of the checklist. The pedagogical implications of the use of diagnostic assessment in ESL academic writing were discussed, as were the contributions that it would make to the theory and practice of second language writing instruction and assessment.

This thesis considers how statistics about adult literacy have produced a new transnational norm of what it means to “be literate” and asks what has been produced by demarcating a calculable threshold of capable literate conduct. Analyzing literacy as a form of conduct enables investigation of the political dimensions of governmental interest in literate conduct and consideration of what subjects, relationships and forms of power are produced by various problematizations. Genealogical analysis of the currently dominant governing rationality, what is termed the psychometrological regime, revealed that Level Three of the International Adult Literacy Survey (IALS) has been constructed as a threshold between people who can act as autonomous, entrepreneurial subjects and those who cannot. In the case of Ontario, this threshold becomes an indicator of “employability” and produces a singular and problematic population who are subjected to coercive educational interventions. Tactics and techniques in the province’s Literacy and Basic Skills (LBS) policy construct literacy programs as sites responsible for transforming subjects below the threshold into human capital assets; this represents a significant departure from the original mission of community-based agencies. Data from interviews with educators in these programs indicate that adult literacy workers occupy an uneasy position between the demands of policy, their pastoral relationships with learners, and the complex realities faced by adults who struggle with print. While these educators may choose to disobey some policy imperatives they nonetheless act, at times unwittingly, as agents of governance. By highlighting the impossibilities produced by the neoliberal problematization of literacy, and the negotiations that literacy workers perform in the face of such dilemmas, this research contributes to thinking through how to transform coercive and authoritarian tendencies currently governing literate conduct.

This thesis considers how statistics about adult literacy have produced a new transnational norm of what it means to “be literate” and asks what has been produced by demarcating a calculable threshold of capable literate conduct. Analyzing literacy as a form of conduct enables investigation of the political dimensions of governmental interest in literate conduct and consideration of what subjects, relationships and forms of power are produced by various problematizations. Genealogical analysis of the currently dominant governing rationality, what is termed the psychometrological regime, revealed that Level Three of the International Adult Literacy Survey (IALS) has been constructed as a threshold between people who can act as autonomous, entrepreneurial subjects and those who cannot. In the case of Ontario, this threshold becomes an indicator of “employability” and produces a singular and problematic population who are subjected to coercive educational interventions. Tactics and techniques in the province’s Literacy and Basic Skills (LBS) policy construct literacy programs as sites responsible for transforming subjects below the threshold into human capital assets; this represents a significant departure from the original mission of community-based agencies. Data from interviews with educators in these programs indicate that adult literacy workers occupy an uneasy position between the demands of policy, their pastoral relationships with learners, and the complex realities faced by adults who struggle with print. While these educators may choose to disobey some policy imperatives they nonetheless act, at times unwittingly, as agents of governance. By highlighting the impossibilities produced by the neoliberal problematization of literacy, and the negotiations that literacy workers perform in the face of such dilemmas, this research contributes to thinking through how to transform coercive and authoritarian tendencies currently governing literate conduct.

Le modèle de Rasch est utilisé de nos jours dans de nombreuses applications en sciences sociales et en médecine. Parmi les applications de ce modèle, on trouve l’étude de la qualité psychométrique des items d’un test, le calibrage des items pour les tests adaptatifs, la production des mesures d’habileté en sciences de l’éducation. Il est particulièrement mis à profit dans des enquêtes internationales à grande échelle comme l’enquête PISA (Programme for International Student Assessment). L’une des hypothèses que doivent vérifier l’ensemble des items du test devant mesurer un trait donné est celle de l’unidimensionnalité, c’est-à dire que tous mis ensemble ne doivent mesurer que le trait en étude, et la réponse que donne un individu à chacun de ces items n’est fonction que du niveau de ce trait chez cet individu. Il se pose donc l’épineuse question de la détermination de la dimensionnalité de l’outil de mesure, car l’objectif étant de ne conserver ensemble que des items concourant à mesurer un seul et même trait. En pratique, les tests auxquels sont soumis les individus ne sont pas strictement unidimensionnels car nos réponses sont aussi conditionnées par nos habitudes et notre milieu. Le plus important selon Stout (1987) est d’avoir un test ayant une dimension dominante, car sinon on devra utiliser des modèles multidimensionnels qui s’avèrent souvent complexes et difficiles à interpréter pour un preneur de décision non expert en mesure. Notre travail a consisté à explorer un ensemble de conditions dans lesquelles le modèle de Rasch unidimensionnel peut produire des mesures acceptables malgré la présence de plusieurs traits déterminants dans les données. Nous avons travaillé avec des données bidimensionnelles simulées, et avons mis à profit le modèle linéaire multiple et les statistiques d’ajustement infit t du modèle de Rasch unidimensionnel. / Today, the Rasch model is most used in many applications of the social sciences and in medicine. Among the applications of this model, one can cite the study of the psychometric qualities of test items, items calibration in adaptive testing and the production of skill measures in education science. It is particularly used in international large-scale surveys such as PISA (Programme for International Student Assessment) survey. One of the assumptions test items selected to measure a given trait must satisfied is the unidimensionality assumption, that is all items put together should measure the trait under study, and the response given by an individual to each of these items is a function only of the level of the trait that the individual possesses. This raises the issue of determining the dimensionality of a measurement tool, because the goal is to keep only items that contribute to measure the single trait. In practice, not all test instruments developed to collect data from individuals are strictly unidimensional because our responses are also influenced by our habits and our environment. According to Stout (1987) the most important thing is to have a test with a dominant dimension, otherwise we will use multivariate models that are often complex and difficult to interpret for a decision maker who is not an expert in measurement theory. Our work has been to explore a set of conditions under which the Rasch model can produce acceptable measures despite the presence of several dimensions in the data. We worked with two-dimensional simulated data and have used the multiple linear regression model and infit statistics t produced by the unidimensional Rasch model. / Le logiciel de simulation des données et d'analyse est Conquest V.3

This study investigated secondary school students’ errors and misconceptions in algebra with a view to expose the nature and origin of those errors and to make suggestions for classroom teaching. The study used a mixed method research design. An algebra test which was pilot-tested for its validity and reliability was given to a sample of grade 11 students in an urban secondary school in Ontario. The test contained questions from four main areas of algebra: variables, algebraic expressions, equations, and word problems. A rubric containing the observed errors was prepared for each conceptual area. Two weeks after the test, six students were interviewed to identify their misconceptions and their reasoning. In the interview process, students were asked to explain their thinking while they were doing the same problems again. Some prompting questions were asked to facilitate this process and to clarify more about students’ claims. The results indicated a number of error categories under each area. Some errors emanated from misconceptions. Under variables, the main reason for misconceptions was the lack of understanding of the basic concept of the variable in different contexts. The abstract structure of algebraic expressions posed many problems to students such as understanding or manipulating them according to accepted rules, procedures, or algorithms. Inadequate understanding of the uses of the equal sign and its properties when it is used in an equation was a major problem that hindered solving equations correctly. The main difficulty in word problems was translating them from natural language to algebraic language. Students used guessing or trial and error methods extensively in solving word problems. Some other difficulties for students which are non-algebraic in nature were also found in this study. Some of these features were: unstable conceptual models, haphazard reasoning, lack of arithmetic skills, lack or non-use of metacognitive skills, and test anxiety. Having the correct conceptual (why), procedural (how), declarative (what), and conditional knowledge (when) based on the stage of the problem solving process will allow students to avoid many errors and misconceptions. Conducting individual interviews in classroom situations is important not only to identify errors and misconceptions but also to recognize individual differences.

This study investigated secondary school students’ errors and misconceptions in algebra with a view to expose the nature and origin of those errors and to make suggestions for classroom teaching. The study used a mixed method research design. An algebra test which was pilot-tested for its validity and reliability was given to a sample of grade 11 students in an urban secondary school in Ontario. The test contained questions from four main areas of algebra: variables, algebraic expressions, equations, and word problems. A rubric containing the observed errors was prepared for each conceptual area. Two weeks after the test, six students were interviewed to identify their misconceptions and their reasoning. In the interview process, students were asked to explain their thinking while they were doing the same problems again. Some prompting questions were asked to facilitate this process and to clarify more about students’ claims. The results indicated a number of error categories under each area. Some errors emanated from misconceptions. Under variables, the main reason for misconceptions was the lack of understanding of the basic concept of the variable in different contexts. The abstract structure of algebraic expressions posed many problems to students such as understanding or manipulating them according to accepted rules, procedures, or algorithms. Inadequate understanding of the uses of the equal sign and its properties when it is used in an equation was a major problem that hindered solving equations correctly. The main difficulty in word problems was translating them from natural language to algebraic language. Students used guessing or trial and error methods extensively in solving word problems. Some other difficulties for students which are non-algebraic in nature were also found in this study. Some of these features were: unstable conceptual models, haphazard reasoning, lack of arithmetic skills, lack or non-use of metacognitive skills, and test anxiety. Having the correct conceptual (why), procedural (how), declarative (what), and conditional knowledge (when) based on the stage of the problem solving process will allow students to avoid many errors and misconceptions. Conducting individual interviews in classroom situations is important not only to identify errors and misconceptions but also to recognize individual differences.

Educators are increasingly being asked to interact with data to facilitate students’ learning in the classroom. However, as an educational measurement community, we have little understanding of the factors and/or contexts that facilitate educators’ successful use of data. Educators’ use of score reports and the relationship to the intended use is integral to the concept of validity. A conceptual model, “Data Habit of Mind,” is proposed to study educators’ understanding, interpretation and potential applications of results from large-scale assessments. The metaphor, “Habit of Mind,” was originally coined by Robert Sternberg and Dan Keating, and has been applied in the education sector to describe educators’ habits of inquiry when interacting with assessments. Based on an extensive review of the literature, Data Habit of Mind is defined as a combination of statistical literacy and score report interpretation. Statistical literacy is the extent to which an individual is able to describe, organize and reduce, represent, and analyze and interpret data. Score report interpretation is the extent to which an individual is able to describe, summarize, question, and propose an application for a given set of elements on a score report. The combination of these two makes up an individual’s Data Habit of Mind. Twenty educators were interviewed to assess their level of statistical literacy and their score report interpretation skills. A cognitive interview approach was used to capture the educators’ cognitive processes as they solved performance-based tasks, and protocol analysis procedures were used to encode the responses into the conceptual model. Descriptions of educators’ Data Habit of Mind were then generated through qualitative matrix analysis. Four groups of educators were identified based on the patterns of relationship between their statistical literacy and score report interpretation scores. Demographic factors, including teaching experience, gender and educational background were not meaningful predictors of educators’ Data Habit of Mind. These results add to our understanding of how educators interpret and use test results and have implications for test validation processes.

The phrase "opportunities to learn" (OTL) is most commonly interpreted in institutional, or inter-individual, terms but it can also be viewed as a cognitive, or intra-individual, phenomenon. How student learning heterogeneity (LH) - learning differences manifested when children's understanding is later assessed - is understood varies by OTL interpretation. In this study, I argue that the cognitive underpinning of learning disability, learning difficulty, typical achievement, and gifted achievement in mathematics is not well understood in part because of the ambiguity of LH assumptions in previous studies. Data from 104,315 Ontario students who had responded to provincially-mandated mathematics tests in grades 3, 6, and 9 dataset were analyzed using latent trait analysis (LTM) and latent class analysis (LCA). The tests were constructed to distinguish four achievement levels per grade and, either five curriculum strands (grades 3 and 6), three strands (grade 9 applied) or four strands (grade 9 academic). Best-fitting LTM models reflected 3- or 4-factors (grade 9 applied and grades 3, 6, 9 academic, respectively). Best-fitting LCA solutions reflected 4- or 5-classes (grade 3, 6 and grade 9 applied, academic, respectively). There were differences in relative proportions of students who were distributed across levels and classes. Moreover, grade 9 models were more complex than the reported four achievement levels. To explore intrinsic modeled results further, latent factors were plotted against latent classes. Implications of institutional versus cognitive interpretations are discussed.