The aim is to determine the effect of reference standard related bias on estimates of plain radiograph reading performance using studies conducted in clinical practice. Data were extracted on study eligibility, clinical and reference standard characteristics and reading performance. The choice of reference standards and the prevalence of bias are presented descriptively. Associations between bias and reading performance are estimated using a regression model that produces relative diagnostic odds ratios (RDOR) with 95% confidence intervals (CIs). Three of the 20 eligible studies addressed all five reference standard related biases; 15 studies addressed three or more. When the reference standard report is influenced by knowledge of an observer's opinion this is associated with a significant overestimation in reading performance (RDOR, 3.7; 95% CI, 1.6 to 8.3; p¿=¿0.01). There is limited evidence that reading performance is inflated when the observer is aware of the reference standard report before commenting on the radiograph (RDOR, 1.7; 95% CI, 0.6 to 5.1) and deflated when a less valid reference standard is used (RDOR, 0.5; 95% CI, 0.1 to 2.5). There is no evidence that reading performance is affected by application of the reference standard depending on an observer's opinion and using different reference standards in the same study. In conclusion we found variation in the choice and application of reference standards in studies of plain radiograph reading performance, but only when reference standards report in the knowledge of an observer's opinion does this contribute to a significant overestimation in reading performance.