Objectives: The aim of this dissertation was to examine bias in randomized controlled trials (RCTs) and non-randomized studies (NRS) in surgery using the literature evaluating laparoscopy and conventional (i.e. open) surgery for the treatment of colon cancer as a case study. The objectives were 1) to develop a conceptual framework for bias in comparative NRS; 2) to compare effect estimates from NRS with those from RCTs at low risk of bias; 3) to explore the impact of NRS-design attributes on estimates of treatment effect. Methods: The methods included a modified framework synthesis, systematic review of the literature, random-effects meta-analyses, and frequentist and Bayesian meta-regression. The Cochrane Risk of Bias Tool was used to classify trials as Strong RCTs (i.e. low risk of bias) or Typical RCTs (i.e. unclear or high risk of bias). Results: A conceptual framework for bias in comparative NRS was developed and it contains 37 individual sources of bias or “items”. These items were organized within 6 overarching “domains”: selection bias, information bias, performance bias, detection bias, attrition bias, and selective reporting bias. Our analyses revealed that NRS were associated with more extreme estimates of benefit for laparoscopy than Strong RCTs when examining subjective outcomes. The odds ratios from NRS were 36% smaller (i.e. demonstrating more benefit for laparoscopy) than those from Strong RCTs for the outcome post-operative complications (Ratio of Odds Ratios, ROR 0.64, [0.42, 0.97], p=0.04). Similar exaggerated benefit was seen among NRS when assessing length of stay, (Difference in Mean Differences, -2.15 days, [-4.08, -0.21], p=0.03). This pattern was not observed with the objective outcomes peri-operative mortality and number of lymph nodes harvested. Analyses adjusted for period effects and between-study case-mix yielded similar findings. Finally, effect estimates in NRS did not consistently vary according to the presence or absence of nine design characteristics identified from the conceptual framework. Conclusions: We have demonstrated that the results of surgical NRS can be significantly biased as compared with those of low risk of bias RCTs when evaluating subjective outcomes. However, none of the nine NRS-design characteristics examined was consistently associated with biased effect estimates.

Objectives: The aim of this dissertation was to examine bias in randomized controlled trials (RCTs) and non-randomized studies (NRS) in surgery using the literature evaluating laparoscopy and conventional (i.e. open) surgery for the treatment of colon cancer as a case study. The objectives were 1) to develop a conceptual framework for bias in comparative NRS; 2) to compare effect estimates from NRS with those from RCTs at low risk of bias; 3) to explore the impact of NRS-design attributes on estimates of treatment effect. Methods: The methods included a modified framework synthesis, systematic review of the literature, random-effects meta-analyses, and frequentist and Bayesian meta-regression. The Cochrane Risk of Bias Tool was used to classify trials as Strong RCTs (i.e. low risk of bias) or Typical RCTs (i.e. unclear or high risk of bias). Results: A conceptual framework for bias in comparative NRS was developed and it contains 37 individual sources of bias or “items”. These items were organized within 6 overarching “domains”: selection bias, information bias, performance bias, detection bias, attrition bias, and selective reporting bias. Our analyses revealed that NRS were associated with more extreme estimates of benefit for laparoscopy than Strong RCTs when examining subjective outcomes. The odds ratios from NRS were 36% smaller (i.e. demonstrating more benefit for laparoscopy) than those from Strong RCTs for the outcome post-operative complications (Ratio of Odds Ratios, ROR 0.64, [0.42, 0.97], p=0.04). Similar exaggerated benefit was seen among NRS when assessing length of stay, (Difference in Mean Differences, -2.15 days, [-4.08, -0.21], p=0.03). This pattern was not observed with the objective outcomes peri-operative mortality and number of lymph nodes harvested. Analyses adjusted for period effects and between-study case-mix yielded similar findings. Finally, effect estimates in NRS did not consistently vary according to the presence or absence of nine design characteristics identified from the conceptual framework. Conclusions: We have demonstrated that the results of surgical NRS can be significantly biased as compared with those of low risk of bias RCTs when evaluating subjective outcomes. However, none of the nine NRS-design characteristics examined was consistently associated with biased effect estimates.

When using evidence from non-randomized studies (NRS) to answer questions about the effects of environmental exposures on health, it is important to assess risk of bias (RoB) of individual studies as part of determining the certainty in the body of evidence. The recently released RoB in Non-randomized Studies of Interventions (ROBINS-I) instrument has undergone careful development and piloting on NRS of health interventions. A key feature of ROBINS-I is evaluating the RoB of studies against an ideal target trial, therefore establishing a structured comparison of RoB against a reference standard. While several instruments exist to evaluate the RoB of NRS of exposure, none of them use such a structured comparison of RoB. Using the fundamental design of ROBINS-I, we explored development of a version of the instrument to evaluate RoB in studies of environmental exposure. We identified important modifications necessitating a distinct instrument: The RoB instrument for NRS of exposures. This work highlights the importance of standardized methods for environmental health decision making, proposes a modified instrument to evaluate the RoB of NRS of exposures, provides guidance for the implementation of the instrument and integration into structured evidence-synthesis frameworks (such as GRADE [Grading of Recommendations Assessment, Development and Evaluation]), and presents evidence on the reliability and validity of the instrument. The RoB instrument for NRS of exposures delivers a standardized instrument that systematic review authors and guideline developers can use to evaluate RoB in NRS of exposures. The nature of these methodological changes allow better integration of RoB assessment in the environmental health field with GRADE. / Thesis / Doctor of Philosophy (PhD) / When making a decision about interventions to reduce or remove an environmental exposure, evidence is needed to weigh the desirable and undesirable consequences of the decision. No research study is perfect. Most of the studies documenting environmental exposures cannot control for the fact that people who might be highly exposed may have different characteristics compared to those who have low levels of exposure other than just the exposure itself. For example, people exposed to more environmental air pollution living in inner cities may also be more likely to smoke or have occupational exposures that could predispose them to lung cancer than those exposed to lower levels of air pollution. Understanding limitations in studies that address those questions informs our certainty that the data represents the truth. The greater the confidence we have in the data, the more likely we are to be certain that removing or reducing exposure will lead to a desirable outcome. A tool can be used to walk people through the evaluation of limitations within each study. However, it is important that the tool evaluates the correct limitations within the study. It is also important that people using the tool can apply it reliably. Without a reliable or valid tool to evaluate the limitations of the studies, it can be difficult to inform decisions on whether or not to implement specific policies. In our study, we tested the ability of a new and well-developed tool (ROBINS for interventions) to identify the limitations in studies linking environmental exposures to health outcomes. Based on the findings from our evaluation, we modified our protocol to see if we could improve our ability to evaluate these studies of environmental exposures. We asked people with an understanding of scientific methods to independently evaluate 35 studies with our modified tool (ROBINS for exposures). We compared those responses to see whether all the reviewers came up with similar decisions and if their decision was similar or different than the conclusion they made using more commonly used tools. Based on our results, we determined that our modified tool does provide a consistent evaluation of study limitations and accurately measures the limitations present in studies of exposure. This tool can be used to inform decisions about removing or reducing one’s exposure to environmental hazards.