{"id":{"repo_id":"utahstate","oai_identifier":"oai:digitalcommons.usu.edu:etd-2030"},"canonical_url":"https://search.ndltd.org/etd/utahstate/oai:digitalcommons.usu.edu:etd-2030","repository":{"repo_id":"utahstate","name":"Utah State University","base_url":"https://digitalcommons.usu.edu/do/oai/"},"display":{"title":"Evaluating Reliability and Validity Evidence for Merrill's 2007 5 Star Instrument","abstract":"Merrill’s 2007 5 Star instrument, hereafter called the M-5 Star, is based on Merrill’s well recognized First Principles of Instruction. However, the instrument has not been tested for reliability and validity. In a pilot study, Cropper’s version of his instrument (C-5 Star) showed some reliability and validity evidence, but the M-5 Star needed similar evidence. To address this gap in the literature, the purpose of this study is to assess the reliability and validity evidence for M-5 Star. Raters were drawn from a graduate course in online course evaluation and asked to rate a sample (N = 6) of exclusively online university classes using M-5 Star and three comparison instruments. The comparison instruments also purport to examine course quality but lack the emphasis on instructional strategies in M-5 Star. Interrater reliability evidence for the M-5 Star and the comparison instruments was moderate to substantial (M5-Star ICC = .56, p = .001); Texas IQ ICC = .43, p = .001; WebCT ICC = .75, p = .001: SREB ICC = .53, p = .001). However, interrater reliability was tentative because rater pair scores were averaged, biasing the scores toward agreement and inflating ICC. Low correlations between M-5 Star (the criterion) and the comparison measures indicate divergent validity support that M-5 Star is measuring a different core concept of quality in online classes. M-5 Star correlation with WebCT was r = .39, p = .44 (r2 = .15), with WebCT was r = .44, p = .38 (r2 = .20), and with SREB was r = .43, p = .39 (r2 = .19). In addition to divergent validity analysis, a content validity index (CVI) analysis was undertaken using experts in the area of First Principles. According to First Principles experts, other than a few items on the rubric, the vast majority of the M-5 Star CVI results support close alignment with Merrill’s First Principles of Instruction. Of the 63 M-5 Star individual items, 56 (89%) of them received high scores on Aiken’s CVI that were significant at the .10 level. Study limitations are discussed at length, alongside calls for future research and practical and scholarly significance for the research.","abstract_html":"Merrill’s 2007 5 Star instrument, hereafter called the M-5 Star, is based on Merrill’s well recognized First Principles of Instruction. However, the instrument has not been tested for reliability and validity. In a pilot study, Cropper’s version of his instrument (C-5 Star) showed some reliability and validity evidence, but the M-5 Star needed similar evidence. To address this gap in the literature, the purpose of this study is to assess the reliability and validity evidence for M-5 Star. Raters were drawn from a graduate course in online course evaluation and asked to rate a sample (N = 6) of exclusively online university classes using M-5 Star and three comparison instruments. The comparison instruments also purport to examine course quality but lack the emphasis on instructional strategies in M-5 Star. Interrater reliability evidence for the M-5 Star and the comparison instruments was moderate to substantial (M5-Star ICC = .56, p = .001); Texas IQ ICC = .43, p = .001; WebCT ICC = .75, p = .001: SREB ICC = .53, p = .001). However, interrater reliability was tentative because rater pair scores were averaged, biasing the scores toward agreement and inflating ICC. Low correlations between M-5 Star (the criterion) and the comparison measures indicate divergent validity support that M-5 Star is measuring a different core concept of quality in online classes. M-5 Star correlation with WebCT was r = .39, p = .44 (r2 = .15), with WebCT was r = .44, p = .38 (r2 = .20), and with SREB was r = .43, p = .39 (r2 = .19). In addition to divergent validity analysis, a content validity index (CVI) analysis was undertaken using experts in the area of First Principles. According to First Principles experts, other than a few items on the rubric, the vast majority of the M-5 Star CVI results support close alignment with Merrill’s First Principles of Instruction. Of the 63 M-5 Star individual items, 56 (89%) of them received high scores on Aiken’s CVI that were significant at the .10 level. Study limitations are discussed at length, alongside calls for future research and practical and scholarly significance for the research.","abstract_has_math":false,"creators":["Cropper, Max Hale"],"institution":"Hosted by Utah State University Libraries","degree_name":"Doctor of Philosophy (PhD)","degree_level":"Dissertation","degree_discipline":"Instructional Technology and Learning Sciences","degree_department":null,"school":null,"contributors":["Andrew Walker","M. David Merrill","Byron R. Burnham"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":null,"date_issued":"","date_published":null,"updated_at":"2026-11-19T08:46:51Z","subjects":["Merrill's 2007 5 Star Instrument","Reliablity","Validity","Comparison","Education"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[{"key":"dc:identifier.doi","label":"DOI","values":["https://doi.org/10.26076/3d24-57c5"],"render_values":[{"text":"https://doi.org/10.26076/3d24-57c5","href":"https://doi.org/10.26076/3d24-57c5","code":true}]}],"person_identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.usu.edu/etd/1039","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://digitalcommons.usu.edu/etd/1039","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://digitalcommons.usu.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adigitalcommons.usu.edu%3Aetd-2030","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Andrew Walker","M. David Merrill","Byron R. Burnham"]},{"key":"dc:creator","label":"Author","values":["Cropper, Max Hale"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2001-09-29T07:00:00Z"]},{"key":"dc:date.created","label":"Dc Date Created","values":["2011-05-01T07:00:00Z"]},{"key":"dc:publisher","label":"Institution","values":["Hosted by Utah State University Libraries"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Instructional Technology and Learning Sciences"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Doctor of Philosophy (PhD)"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Merrill's 2007 5 Star Instrument","Reliablity","Validity","Comparison","Education"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.usu.edu/etd/1039"]},{"key":"dc:identifier.doi","label":"DOI","values":["https://doi.org/10.26076/3d24-57c5"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Merrill’s 2007 5 Star instrument, hereafter called the M-5 Star, is based on Merrill’s well recognized First Principles of Instruction. However, the instrument has not been tested for reliability and validity. In a pilot study, Cropper’s version of his instrument (C-5 Star) showed some reliability and validity evidence, but the M-5 Star needed similar evidence. To address this gap in the literature, the purpose of this study is to assess the reliability and validity evidence for M-5 Star. Raters were drawn from a graduate course in online course evaluation and asked to rate a sample (N = 6) of exclusively online university classes using M-5 Star and three comparison instruments. The comparison instruments also purport to examine course quality but lack the emphasis on instructional strategies in M-5 Star. Interrater reliability evidence for the M-5 Star and the comparison instruments was moderate to substantial (M5-Star ICC = .56, p = .001); Texas IQ ICC = .43, p = .001; WebCT ICC = .75, p = .001: SREB ICC = .53, p = .001). However, interrater reliability was tentative because rater pair scores were averaged, biasing the scores toward agreement and inflating ICC. Low correlations between M-5 Star (the criterion) and the comparison measures indicate divergent validity support that M-5 Star is measuring a different core concept of quality in online classes. M-5 Star correlation with WebCT was r = .39, p = .44 (r2 = .15), with WebCT was r = .44, p = .38 (r2 = .20), and with SREB was r = .43, p = .39 (r2 = .19). In addition to divergent validity analysis, a content validity index (CVI) analysis was undertaken using experts in the area of First Principles. According to First Principles experts, other than a few items on the rubric, the vast majority of the M-5 Star CVI results support close alignment with Merrill’s First Principles of Instruction. Of the 63 M-5 Star individual items, 56 (89%) of them received high scores on Aiken’s CVI that were significant at the .10 level. Study limitations are discussed at length, alongside calls for future research and practical and scholarly significance for the research."]},{"key":"dc:description.note","label":"Dc Description Note","values":["<p>This work made publicly available electronically on September 29, 2011.</p>"]},{"key":"dc:title","label":"Title","values":["Evaluating Reliability and Validity Evidence for Merrill's 2007 5 Star Instrument"]}]}],"canonical_facts":{"dc:contributor":["Andrew Walker","M. David Merrill","Byron R. Burnham"],"dc:creator":["Cropper, Max Hale"],"dc:date.available":["2001-09-29T07:00:00Z"],"dc:date.created":["2011-05-01T07:00:00Z"],"dc:description.abstract":["Merrill’s 2007 5 Star instrument, hereafter called the M-5 Star, is based on Merrill’s well recognized First Principles of Instruction. However, the instrument has not been tested for reliability and validity. In a pilot study, Cropper’s version of his instrument (C-5 Star) showed some reliability and validity evidence, but the M-5 Star needed similar evidence. To address this gap in the literature, the purpose of this study is to assess the reliability and validity evidence for M-5 Star. Raters were drawn from a graduate course in online course evaluation and asked to rate a sample (N = 6) of exclusively online university classes using M-5 Star and three comparison instruments. The comparison instruments also purport to examine course quality but lack the emphasis on instructional strategies in M-5 Star. Interrater reliability evidence for the M-5 Star and the comparison instruments was moderate to substantial (M5-Star ICC = .56, p = .001); Texas IQ ICC = .43, p = .001; WebCT ICC = .75, p = .001: SREB ICC = .53, p = .001). However, interrater reliability was tentative because rater pair scores were averaged, biasing the scores toward agreement and inflating ICC. Low correlations between M-5 Star (the criterion) and the comparison measures indicate divergent validity support that M-5 Star is measuring a different core concept of quality in online classes. M-5 Star correlation with WebCT was r = .39, p = .44 (r2 = .15), with WebCT was r = .44, p = .38 (r2 = .20), and with SREB was r = .43, p = .39 (r2 = .19). In addition to divergent validity analysis, a content validity index (CVI) analysis was undertaken using experts in the area of First Principles. According to First Principles experts, other than a few items on the rubric, the vast majority of the M-5 Star CVI results support close alignment with Merrill’s First Principles of Instruction. Of the 63 M-5 Star individual items, 56 (89%) of them received high scores on Aiken’s CVI that were significant at the .10 level. Study limitations are discussed at length, alongside calls for future research and practical and scholarly significance for the research."],"dc:description.note":["<p>This work made publicly available electronically on September 29, 2011.</p>"],"dc:identifier":["https://digitalcommons.usu.edu/etd/1039"],"dc:identifier.doi":["https://doi.org/10.26076/3d24-57c5"],"dc:publisher":["Hosted by Utah State University Libraries"],"dc:subject":["Merrill's 2007 5 Star Instrument","Reliablity","Validity","Comparison","Education"],"dc:title":["Evaluating Reliability and Validity Evidence for Merrill's 2007 5 Star Instrument"],"thesis:degree_discipline":["Instructional Technology and Learning Sciences"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Doctor of Philosophy (PhD)"]},"updated_at":"2026-11-19T08:46:51Z"}