This simulation study was designed to assess the impact of three ad hoc procedures for handling missing level two (here, school) identifiers in multilevel modeling. A multiple membership data structure was generated and both conventional hierarchical linear modeling (HLM) and multiple membership random effects modeling (MMREM) were employed. HLM models purely hierarchical data structures while MMREM appropriately models multiple membership data structures. Two of the ad hoc procedures investigated involved removing different subsamples of students from the analysis (HLM-Delete and MMREM-Delete) while the other procedure retained all subjects and involved creating a pseudo-identifier for the missing level two identifier (MMREM-Unique). Relative parameter and standard error (SE) bias were calculated for each parameter estimated to assess parameter recovery. Across the conditions and parameters investigated, each procedure had some level of substantial bias. MMREM-Unique and MMREM-Delete resulted in the least amount of relative parameter bias while HLM-Delete resulted in the least amount of relative SE bias. Results and implications for applied researchers are discussed. / text

This report focuses on introducing two statistical models for dealing with data involving complex social structures. Appropriate handling of data structures is a concern in the context of educational settings. From base single-level data to complex hierarchical with cross-classifications and multiple-memberships, we explain and demonstrate their distinction and establish appropriate regression models. Real data from the National Center for Education Statistics (NECS) is used to demonstrate different way of handling a cross-classified data structure as well as appropriate models. Results will be presented and compared to examine the practical operation for each model. / text

The purpose of the simulation was to assess the impact of weight pattern assignment when using the multiple membership random effects model (MMREM). In contrast with most previous methodological research using the MMREM, mobility was not randomly assigned; rather the likelihood of student mobility was generated as a function of the student predictor. Two true weights patterns were used to generate the data (random equal and random unequal). For each set of generated data, the true correct weights and two incorrect fixed weight patterns (fixed equal and fixed unequal) that are similar to those used in practice by applied researchers were used to estimate the model. Several design factors were manipulated including the percent mobility, the ICC, and the true generating values of the level one and level two mobility predictors. To assess parameter recovery, relative parameter bias was calculated for the fixed effects and random effects variance components. Standard error (SE) bias was also calculated for the standard errors estimated for each fixed effect. Substantial relative parameter bias differences between weight patterns used were observed for the level two school mobility predictor across conditions as well as the level two random effects variance component, in some conditions. Substantial SE bias differences between weight patterns used were also found for the school mobility predictor in some conditions. Substantial SE and parameter bias was found for some parameters for which it was not anticipated. The results, discussion, future directions for research, and implications for applied researchers are discussed.

This study was designed to investigate the impact of multiple-membership data structures in multilevel modeling. Multiple-membership arises when lower level units (e.g., students) are nested within more than one higher level unit (e.g., schools). In this case, more than one school will contribute to students' academic achievement and progress. In reality, it is inappropriate to assume a pure nesting of a student within a single school. While use of HLM requires either deletion of the cases involving multiple-membership or exclusion of prior schools attended, MMREM includes students who attend multiple schools and controls for the effect of all schools on student outcomes. The simulation study found level two variability underestimation and corresponding level one variability overestimation when multiple membership data structures were ignored. The study also revealed that when HLM failed to include multiple membership data structures, it underestimated school level predictor. With an increased numbers of mobile students under the No Child Left Behind (NCLB) Act, researchers need to understand MMREM and correctly apply it to multiple membership data structures. This MMREM approach will help improve the generalizability of findings and will improve the validity of the statistical results. / text