Thesis advisor: Ina V.S. Mullis / As an extension of the effort devoted to updating the questionnaires for TIMSS and PIRLS 2011, this dissertation explored a new reporting strategy for contextual questionnaire data. The study investigated the feasibility of constructing "global indicators" from a large number of diverse background variables, which could provide policy makers and practitioners with meaningful information on effective learning environments. Four broad constructs of effective learning environments were derived from the TIMSS and PIRLS Contextual Frameworks for 2011. These were: 1) effective school environments for learning to read, 2) effective home environments for learning to read, 3) effective classroom environments for learning mathematics, and 4) students' motivation to learn mathematics. Using the TIMSS and PIRLS 2011 Frameworks, the conceptual definitions of the constructs were formulated as constructs maps. Next, relevant questionnaire items were identified that addressed each aspect of the construct maps, capitalizing on the full range of background information in the TIMSS 2007 and PIRLS 2006 International Databases. The questionnaire items were used to create sets of variables for scaling, and subsequent to principal component analysis to confirm scale unidimensionality, the variables were combined into 1-Parameter IRT (Rasch) scales. The idea of conveying the meaning of the broad contextual scales through item mapping was explored, as well as reporting country-by-country results on the global scales. The scaling was successful and it was concluded that contextual information could be reported more globally in future cycles of TIMSS and PIRLS. However, the study also demonstrated that it is extremely complicated to choose background constructs at the right level of aggregation for both analysis and reporting. It is difficult to develop scales that summarize data for educational policy makers without loss of vital information. / Thesis (PhD) — Boston College, 2011. / Submitted to: Boston College. Lynch School of Education. / Discipline: Educational Research, Measurement, and Evaluation.

Guidelines to screen and select common items for vertical scaling have been adopted from equating. Differences between vertical scaling and equating suggest that these guidelines may not apply to vertical scaling in the same way that they apply to equating. For example, in equating the examinee groups are assumed to be randomly equivalent, but in vertical scaling the examinee groups are assumed to possess different levels of proficiency. Equating studies that examined the characteristics of the common-item set stress the importance of careful item selection, particularly when groups differ in ability level. Since in vertical scaling cross-level ability differences are expected, the common items' psychometric characteristics become even more important in order to obtain a correct interpretation of students' academic growth. This dissertation applied two screening criteria and two selection approaches to investigate how changes in the composition of the linking sets impacted the nature of students' growth when creating vertical scales for two elementary mathematics tests. The purpose was to observe how well these equating guidelines were applied in the context of vertical scaling. Two separate datasets were analyzed to observe the impact of manipulating the common items' content area and targeted curricular grade level. The same Rasch scaling method was applied for all variations of the linking set. Both the robust z procedure and a variant of the 0.3-logit difference procedure were used to screen unstable common items from the linking sets. (In vertical scaling, a directional item-difficulty difference must be computed for the 0.3-logit difference procedure.) Different combinations of stable common items were selected to make up the linking sets. The mean/mean method was used to compute the equating constant and linearly transform the students' test scores onto the base scale. A total of 36 vertical scales were created. The results indicated that, although the robust z procedure was a more conservative approach to flagging unstable items, the robust z and the 0.3-logit difference procedure produced similar interpretations of students' growth. The results also suggested that the choice of grade-level-targeted common items affected the estimates of students' grade-to-grade growth, whereas the results regarding the choice of content-area-specific common items were inconsistent. The findings from the Geometry and Measurement dataset indicated that the choice of content-area-specific common items had an impact on the interpretation of students' growth, while the findings from the Algebra and Data Analysis/Probability dataset indicated that the choice of content-area-specific common items did not appear to significantly affect students' growth. A discussion of the limitations of the study and possible future research is presented.