One approach to bridging the gap between cognitively principled assessment, instruction, and learning is to provide the score user with meaningful details about the examinee's test performance. Several researchers have demonstrated the utility of modeling item characteristics, such as difficulty, in light of item features and the cognitive skills required to solve the item, as a way to link assessment and instructional feedback. The next generation of the Test of English as a Foreign Language (TOEFL) will be launched in 2005, with new task types that integrate listening, reading, writing and speaking—the four modalities of language. Evidence centered design (ECD) principles are being used to develop tasks for the new TOEFL assessment. ECD provides a framework within which to design tasks, to link information gathered from those tasks back to the target of inference through the statistical model, and to evaluate each facet of the assessment program in terms of its connection to the test purpose. One of the primary goals of the new exam is to provide users with a score report that describes the English language proficiencies of the examinee. The purpose of this study was to develop an item difficulty model as the first step in generating descriptive score reports for the new TOEFL assessment. Task model variables resulting from the ECD process were used as the independent variables, and item difficulty estimates were used as the dependent variable in the item difficulty model. Tree-based regression was used to estimate the nonlinear relationships among the item and stimulus features and item difficulty. The proposed descriptive score reports capitalized on the item features that accounted for the most variance in item difficulty. The validity of the resulting proficiency statements were theoretically supported by the links among the task model variables and student model variables evidenced in the ECD task design shells, and empirically supported by the item difficulty model. Directions for future research should focus on improving the predictors in the item difficulty model, determining the most appropriate proficiency estimate categories, and comparing item difficulty models across major native language groups.

This dissertation was motivated by an observed incongruity between the methodology employed in current second language teaching—the communicative approach—and the pedagogical orientation found in the majority of Spanish placement tests—the form-focused methods that characterize traditional language teaching. Two main aspects were identified as particularly needing attention: the general absence of a listening comprehension measure and the persisting presence of features from pre-communicative days. The first goal of this study was to determine whether adding a listening comprehension measure to a test of vocabulary, grammar, and reading comprehension has an effect on its accuracy to assess Spanish proficiency. The second goal was to determine whether there are any significant differences in the performance of a subset of items written in accordance to communicative guidelines (experimental items) and another subset of items that follow a more form-based approach (control items), taken from existing exams. A review of the literature on the communicative approach to language teaching and testing from the 1970s until present provided the necessary guidelines for the writing of the experimental items and the selection of the control items. An analysis of the most salient features found in Spanish placement exams, as well as a review of the latest research on computer-based language testing are also included. The research instrument was a computer-based, multiple-choice, self-scoring test, specifically designed for this study. The test was administered during the 2001 Spring semester to a total of 233 students enrolled in elementary and intermediate Spanish language classes at two universities. The analyses of the data confirmed that there is a significant effect associated with the methodology reflected in the items on a test. The listening section was found to accurately discriminate among levels of skill. The study also demonstrated that it is possible to develop a test that: (1) has the advantages of delivery by computer, (2) has features that are in accordance with the communicative approach employed in second language instruction, and (3) is a valid measure of Spanish ability. More importantly, the test demonstrated the feasibility of including a component for the assessment of listening skills.