Mode of access: Internet via World Wide Web. Available at http://hdl.handle.net/1947/9667. / "September 2008" Title from PDF cover (viewed on 25 September, 2009) Includes bibliographical references.

No description available.

No description available.

This thesis explores the transformations of evidence in health research synthesis studies – studies that bring together evidence from a number of research reports on the same/ similar topic. It argues that health research synthesis is a broad and intriguing field in a state of pre-formation, in spite of the fact that it may appear well established if equated with its exemplar method – the systematic review inclusive of meta-analysis. Transformations of evidence are processes by which pieces of evidence are modified from what they are in the primary study report into what is needed in the synthesis study while, supposedly, having their integrity fully preserved. Such processes have received no focused attention in the literature. Yet they are key to the validity and reliability of synthesis studies. This work begins to describe them and explore their frequency, scope and drivers. A ‘meta-scientific’ perspective is taken, where ‘meta-scientific’ is understood to include primarily ideas from the philosophy of science and methodological texts in health research, and, to a lesser extent, social studies of science and psychology of science thinking. A range of meta-scientific ideas on evidence and factors that shape it guide the analysis of processes of “data extraction” and “coding” during which much evidence is transformed. The core of the analysis involves the application of an extensive Analysis Framework to 17 highly heterogeneous research papers on cancer. Five non-standard ‘injunctions’ complement the Analysis Framework – for comprehensiveness, extensive multiple coding, extreme transparency, combination of critical appraisal and critique, and for first coding as close as possible to the original and then extending towards larger transformations. Findings suggest even lower credibility of the current overall model of health research synthesis than initially expected. Implications are discussed and a radical vision for the future proposed.

Reliability generalization (RG) is a method for meta-analysis of reliability coefficients to estimate average score reliability across studies, determine variation in reliability, and identify study-level moderator variables influencing score reliability. A total of 107 peer-reviewed RG studies published from 1998 to 2013 were systematically reviewed to characterize the meta-analytic methods employed and to evaluate quality of reporting practice against standards for transparency in meta-analysis reporting. Most commonly, RG studies meta-analyzed alpha coefficients, which were synthesized using an unweighted, fixed-effects model applied to untransformed coefficients. Moderator analyses most frequently included multiple regression and bivariate correlations employing a fixed-effects model on untransformed, unweighted coefficients. Based on a unit-weighted scoring system, mean reporting quality for RG studies was statistically less than that for a comparison study of 198 meta-analyses in the organizational sciences across 42 indicators; however, means were not statistically significantly different between the two studies when evaluating reporting quality on 18 indicators deemed essential to ethical reporting practice in meta-analyses. Since its inception a wide variety of statistical methods have been applied to RG, and meta-analysis of reliability coefficients has extended to fields outside of psychological measurement, such as medicine and business. A set of guidelines for conducting and reporting RG studies is provided.

The psychological impacts of infertility have been well documented in the literature, providing evidence to support that at least some women who confront infertility are at risk for heightened distress and depressive symptoms. In response to this accumulated evidence, it has been argued that psychoeducational interventions may provide an important component to the treatment of infertility. While several theoretical models postulate the effects of stress on infertility treatment outcomes, results of these investigations have led to conflicting conclusions. However, a synthesis of the accumulated data examining the effects of stress on ART treatment outcomes was nonexistent until the conduct of this study. Therefore, the purpose of this study was to investigate the impact of stress on ART treatment outcomes and to determine whether psychoeducational interventions mitigate the impact of stress experienced by women during an ART treatment program. Two hypotheses were tested: 1. Increased levels of stress will be associated with a lower likelihood of Assisted Reproductive Technology (ART) treatment success, and 2. Psychoeducational interventions will mitigate the effects of stress experienced during Assisted Reproductive Technology (ART) treatment. A meta-analysis analyzing the results for each hypothesis was tested through a hierarchical linear regression model. A total of 13 studies, representing 43 effect sizes, were included in the analysis investigating the relationship between stress and ART treatment outcomes. Results of the HLM regression model suggest that stress has a small negative association with ART treatment outcomes (d=0.2012, p< .05). The analysis investigating the relationship between psychoeducational interventions and stress included a total of 4 studies, representing 12 effect sizes. Empirical evidence gathered through this analysis revealed that the effect of psychoeducational interventions on the stress experienced by women participating in an ART treatment program were not statistically significant (d=0.3071, p>.05). However, because this analysis was based on such a small sample of studies, generalizations regarding the efficacy of psychoeducational interventions cannot be made. Therefore, research aimed at investigating the impacts of a variety of programs should continue in an effort to provide more conclusive information.

Because both validity and reliability indices are a function of the scores on a given administration of a measure, their values can often vary across samples. It is a common mistake to say that a test is reliable when in fact it is not the test that is reliable but the scores on the test that are reliable. In 1998, vacha-haase proposed a fixed-effects meta-analytic method for evaluating reliability that is similar to validity generalization studies called reliability generalization (rg). This study was conducted to evaluate alternative analysis strategies for the meta-analysis method of reliability generalization when the reliability estimates are not statistically independent. Five approaches for handling the violation of independence were implemented: ignoring the violation and treating each observation as independent, calculating one mean or median from each study, randomly selecting only one observation per study, or using a mixed effects model. This Monte Carlo study included five factors in the method. These factors were (a) the coefficient alpha, (b) sample size in the primary studies, (c) number of primary studies in the rg study, (d) number of reliability estimates from each, and (e) the degree of violation of independence where the strength of the dependence is related to the number of reliability indices (i.e. coefficient alpha) derived from a simulated set of examines and the magnitude of the correlation between the journal studies (with intra-class correlation icc = 0, .0l , .30, and .90). These factors were used to simulate samples under known and controlled population conditions. In general, the results suggested that the type of treatment does not have a noticeable impact on the accuracy of the reliability results but that researchers should be cautious when the intra-class correlation is relatively large. In addition, the simulations in this study resulted in very poor confidence band coverage. This research suggested that RG meta-analysis methods are appropriate for describing the overall average reliability of a measure or construct but the RG researcher should be careful in regards to the construction of confidence intervals.

Publication bias is one threat to validity that researchers conducting meta-analysis studies confront. Two primary goals of this research were to examine the degree to which publication bias impacts the results of a random effects meta-analysis and to investigate the performance of five statistical methods for detecting publication bias in random effects meta-analysis. Specifically, the difference between the population effect size and the estimated meta-analysis effect size, as well as the difference between the population effect size variance and the meta-analysis effect size variance, provided an indication of the impact of publication bias. In addition, the performance of five statistical methods for detecting publication bias (Begg Rank Correlation with sample size, Begg Rank Correlation with variance, Egger Regression, Funnel Plot Regression, and Trim and Fill) were estimated with Type I error rates and statistical power. The overall findings indicate that publication bias notably impacts the meta-analysis effect size and variance estimates. Poor FTSe I error control was exhibited in many conditions by most of the statistical methods. Even when Type I error rates were adequate the power was small, even with larger samples and greater numbers of studies in the meta-analysis.

The focus of this study was to conduct a meta-analytic investigation to combine the results obtained in independent studies aimed at determining the effectiveness of using digital games, as opposed to traditional methods, as a strategy for improving students' performance in high school mathematics. The major question of this study is: "Does the research on the use of games in high school mathematics support the use of games as a teaching strategy for improving student achievement?" To answer this question, meta-analysis was employed. Meta-analysis synthesizes and analyzes the quantitative data collected in independent and multiple empirical studies carried out on similar topics, situations, and hypotheses in order to reach a general judgment regarding the results of these studies. To determine which studies to use, specific criteria including articles published in refereed journals, thesis, and dissertation studies with experimental and control groups, research with effect size, sample size, standard deviation, and means. Based on these criteria, it was decided to include six experimental studies in the meta-analysis. The result showed that there was no significant differences between the use of digital games and traditional methods to teach mathematics in high school. The weighting factor of the two variables, standard deviation and number of participants, may account for the lack of support for gaming over traditional method of instruction.

Methodological studies of health research are undertaken to investigate the practice of research. They have been instrumental in inciting developments in the design, conduct, analysis, and reporting of health research. Due in part to the field’s diversity, these studies can be difficult to identify in databases. As these studies have not been comprehensively examined to date, the overarching goal of this thesis was to characterize methodological studies and to investigate how they have been labelled and reported in the literature. First, we demonstrate how methodological studies are conducted to provide guidance to end-users—in this case physiatrists and rehabilitation researchers—in a methods guidance paper on pilot and feasibility studies (PAFS), a type of health research design. Second, we performed a pilot study testing the feasibility of searching for and identifying methodological studies in literature databases. Third, based on the pilot study findings and previous research, we outline a protocol for the development of a reporting guideline for methodological studies of health research. Lastly, as part of the first phase of the reporting guideline development process, we performed a review of methodological studies focusing on those that specifically investigated PAFS. In a case study of rehabilitation research, a third of studies labelled as PAFS did not outline any feasibility outcomes, and few provided progression plans to definitive studies. Guidance was focused on providing recommendations and resources for assessing feasibility to help reduce the prevalence of small studies disguised as PAFS, which wastes research resources. In the pilot of methodological studies, preliminary findings on nomenclature and reporting reinforced the notion that there are many names used to describe studies with similar intentions. It was also determined feasible to build a search strategy to identify methodological studies in literature databases. Subsequent findings from the review of methodological studies illustrated that reporting practices are the most common aspect of research investigated. Study design names such as ‘methodological review’, ‘systematic review’, and ‘systematic survey’ were often used to describe studies with similar motives, i.e., to synthesize data from previously published research, whether the synthesis approach was quantitative or qualitative. Existing reporting checklists were rarely used, and when used not appended, possibly due to irrelevance of fields oriented to studies with persons. This work demonstrates the necessity and importance of consensus on reporting and nomenclature for making methodological studies more accessible to the health research community. / Dissertation / Doctor of Philosophy (PhD)