Research-based decision making has been advanced as a way for professionals to make a determination about the effectiveness of a potential treatment. However, informed consumers of research need to be able to determine what constitutes evidence-based practices and what criteria can be used to determine if evidence-based practices have been met. This study was a synthesis of research that involved a critical review of the empirical research studies reported in Volume 47 of the Journal of Speech, Language, and Hearing Research (JSLHR) published in 2004. This methodological research synthesis evaluated (a) the research designs used in the JSLHR studies, (b) information and rationale used to inform population validity assessment decisions, and (c) the extent to which the sampling designs, population validity rating, data analysis procedures, and the specification of generalizations and conclusions provide sufficient evidence to determine an overall rating of conclusion validity. Results indicated that less than one-fifth of the 105 research synthesis population of studies used experimental research designs. Additionally, the vast majority of the research synthesis population of studies (83.8%) were observational research designs. Only five studies out of the research synthesis population of studies (4.8%) were determined to have high population validity. In contrast, 84.8 percent of the research synthesis population of studies were found to have low population validity. That is, the studies did not contain adequate information or description of the essential sampling concerns. The vast majority or 75.3 percent of the research synthesis population of studies were rated as having low conclusion validity. Approximately one-fifth of the 105 research synthesis study population (22 studies or 20.9%) were found to have moderate conclusion validity while less than five percent of the total studies (4 of 105 studies or 3.8%) were found to have high conclusion validity. A meaningful relationship between population validity ratings and conclusion validity ratings was established. Since 81 of 105 studies have identical ratings for both population and conclusion validity, the accuracy of the prediction model developed for this study is 77.1 percent.