Because both validity and reliability indices are a function of the scores on a given administration of a measure, their values can often vary across samples. It is a common mistake to say that a test is reliable when in fact it is not the test that is reliable but the scores on the test that are reliable. In 1998, vacha-haase proposed a fixed-effects meta-analytic method for evaluating reliability that is similar to validity generalization studies called reliability generalization (rg). This study was conducted to evaluate alternative analysis strategies for the meta-analysis method of reliability generalization when the reliability estimates are not statistically independent. Five approaches for handling the violation of independence were implemented: ignoring the violation and treating each observation as independent, calculating one mean or median from each study, randomly selecting only one observation per study, or using a mixed effects model. This Monte Carlo study included five factors in the method. These factors were (a) the coefficient alpha, (b) sample size in the primary studies, (c) number of primary studies in the rg study, (d) number of reliability estimates from each, and (e) the degree of violation of independence where the strength of the dependence is related to the number of reliability indices (i.e. coefficient alpha) derived from a simulated set of examines and the magnitude of the correlation between the journal studies (with intra-class correlation icc = 0, .0l , .30, and .90). These factors were used to simulate samples under known and controlled population conditions. In general, the results suggested that the type of treatment does not have a noticeable impact on the accuracy of the reliability results but that researchers should be cautious when the intra-class correlation is relatively large. In addition, the simulations in this study resulted in very poor confidence band coverage. This research suggested that RG meta-analysis methods are appropriate for describing the overall average reliability of a measure or construct but the RG researcher should be careful in regards to the construction of confidence intervals.

This research aims to identify and evaluate the systemic factors which may be related to decrease in Grade 6 learner’s achievement in Mozambique between 2000 and 2007, looking for possible changes in Educational Effectiveness over that period. SACMEQ III learner results from Grade 6 Reading and Mathematics showed an overall mean decrease from 2000 to 2007. The main research question addressed in this study is: What are the systemic contextual factors associated with decrease in achievement in Reading and Mathematics between 2000 and 2007 in Mozambique? The conceptual framework underpinning this research presents the education system in terms of inputs, processes and outputs (Howie, 2002). Hierarchical Linear Models, based on trend design approach (Nilsen, & Gustafsson, 2014) was used to assess the variation in learner achievement decrease associated with changes in schools inputs and processes. The findings suggest that school-level factors linked to inputs and antecedents have a strong effect on the decrease in learner’s achievement, compared to the processes and practices. Moreover, learners’ background factors, specifically parent’s education and use of language of instruction at home, seem to be the crucial factors associated with learner achievement decrease. When school level variables related to parent’s education, use of language of instruction at home, are included in the model, the amount of variation accounted for, showed an increase (from 23.5 % to 37.7 %,). One can argue that the variation accounted for variables such as parents’ education, whilst use of language of instruction could be indicative of changes in learner’s intake composition between 2000 and 2007. Implications of these findings on the assumptions for large scale assessment studies in developing countries, such as Mozambique, are key issues. For instance, a question could be raised about the “trend” assumption of large scale assessment: To what extent can the trend level of achievement be measured where the learner’s intake composition is changing over the time? In the SACMEQ studies a stronger longitudinal design is needed to investigate how both school and intake factors influence achievement. / Thesis (PhD)--University of Pretoria, 2017. / Science, Mathematics and Technology Education / PhD / Unrestricted