This study compared the effectiveness, cost and efficiency of four formative evaluation conditions: (a) revision based on learner data (RLD), (b) revision based on expert data (RED), (c) revision based on both learner and expert data (RBD), and (d) revision without data (RND). Two more conditions were present in the study: materials in draft (MID) and no treatment (NT). The NT condition consisted of students who were tested without exposure to the instructional materials. The instruction that was formatively evaluated was a six-page article describing the relationship between diet and cancer. The article was written by chemistry professors for an "undergraduate chemistry course for non-science students". Undergraduates (n = 187) provided the effectiveness data. They randomly received one of the four formatively evaluated versions of the article, read the article, answered questions on an objective test, and indicated their confidence with respect to their responses. Professional revisors (n = 8) provided cost data. Each revisor provided cost estimates for all formative evaluation conditions. Efficiency was provided by combining effectiveness with cost data. Effectiveness differences were found between the MID and RLD, and the MID and RBD. The mean test scores, as well as the mean confidence-weighed test scores, of both RLD and RBD were significantly higher than those of the MID. Cost differences indicated three levels of cost. RND was the least costly formative evaluation condition. RLD and RED were equivalent in cost and more costly than RND. RBD was the most costly formative evaluation condition. With respect to efficiency, RLD was recommended. RLD was the least costly condition that was significantly more effective than MID. (Abstract shortened by UMI.)

Despite the emphasis that is placed on collecting learner data in formative evaluation, there has been little research to support its use in the revision of instructional materials. In addition, although the literature suggests that different revisers making changes to the same instructional materials will yield different revisions, little research has been conducted to confirm this. The present study addressed these two issues by having four instructional designers, acting as revisers, make revisions to a draft version of an instructional presentation while thinking aloud. A coding scheme was applied to the think-aloud protocols. Revisions were sorted using a product attribute categorization system consisting of three main categories central to evaluating instructional materials. The results suggest that revisers do give a considerable amount of attention to learner data when revising materials. In addition, there was a great deal of uniformity among the revisions that were generated by the different revisers.

The literature on formative evaluation advocates the use of both subject matter and learner experts to review instructional materials, yet there has been little research to support this recommendation. The present study investigates the distinguishing characteristics of these two expert reviewers, in particular the type and amount of oral feedback they produced and the cognitive processes they engaged in. The think-aloud method was employed to obtain feedback about a six page instructional unit from eight experts; four subject matter and four learner experts. Comments from the experts were transcribed, segmented and coded according to three coding systems. Results indicated that the two groups produced similar data. In addition, the findings showed that both groups referred to similar domains of knowledge, evoked or constructed similar plans and identified their task as detecting problems. These findings contradict the use of both types of experts during formative evaluation. Some practical recommendations for practitioners are offered.

No description available.

No description available.

Note:

No description available.

M.Ed. (Education) / Please refer to full text to view abstract

Typescript (photocopy). / Digitized by Kansas Correctional Industries

No description available.