No description available.

Research has shown that the increased use of computer-based testing has brought about new challenges. With the ease of online test administration, a large number of items are necessary to maintain the item bank and minimise the exposure rate. However, the traditional item development process is time-consuming and costly. Thus, alternative ways of creating items are necessary to improve the item development process. Automatic Item Generation (AIG) is an effective method in generating items rapidly and efficiently. AIG uses algorithms to create questions for testing purposes. However, many of these generators are in the closed form, available only to the selected few. There is a lack of open source, publicly available generators that researchers can utilise to study AIG in greater depth and to generate items for their research. Furthermore, research has indicated that AIG is far from being understood, and more research into its methodology and the psychometric properties of the items created by the generators are needed for it to be used effectively. The studies conducted in this thesis have achieved the following: 1) Five open source item generators were created, and the items were evaluated and validated. 2) Empirical evidence showed that using a weak theory approach to develop item generators was just as credible as using a strong theory approach, even though they are theoretically distinct. 3) The psychometric properties of the generated items were estimated using various IRT models to assess the impact of the template features used to create the items. 4) Joint responses and response time modelling was employed to provide new insights into cognitive processes that go beyond those obtained by typical IRT models. This thesis suggests that AIG provides a tangible solution for improving the item development process for content generation and reducing the procedural cost of generating a large number of items, with the possibility of a unified approach towards test administration (i.e. adaptive item generation). Nonetheless, this thesis focused on rule-based algorithms. The application of other forms of item generation methods and the potential for measuring the intelligence of artificial general intelligence (AGI) is discussed in the final chapter, proposing that the use of AIG techniques create new opportunities as well as challenges for researchers that will redefine the assessment of intelligence.

abstract: The current study employs item difficulty modeling procedures to evaluate the feasibility of potential generative item features for nonword repetition. Specifically, the extent to which the manipulated item features affect the theoretical mechanisms that underlie nonword repetition accuracy was estimated. Generative item features were based on the phonological loop component of Baddelely's model of working memory which addresses phonological short-term memory (Baddeley, 2000, 2003; Baddeley & Hitch, 1974). Using researcher developed software, nonwords were generated to adhere to the phonological constraints of Spanish. Thirty-six nonwords were chosen based on the set item features identified by the proposed cognitive processing model. Using a planned missing data design, two-hundred fifteen Spanish-English bilingual children were administered 24 of the 36 generated nonwords. Multiple regression and explanatory item response modeling techniques (e.g., linear logistic test model, LLTM; Fischer, 1973) were used to estimate the impact of item features on item difficulty. The final LLTM included three item radicals and two item incidentals. Results indicated that the LLTM predicted item difficulties were highly correlated with the Rasch item difficulties (r = .89) and accounted for a substantial amount of the variance in item difficulty (R2 = .79). The findings are discussed in terms of validity evidence in support of using the phonological loop component of Baddeley's model (2000) as a cognitive processing model for nonword repetition items and the feasibility of using the proposed radical structure as an item blueprint for the future generation of nonword repetition items. / Dissertation/Thesis / M.A. Educational Psychology 2011

多選題測驗試題已證明能有效地評估學生的學習成效，然而，以人為方式建立題庫是一件耗時費力的工作。藉由電腦高速運算的能力，電腦輔助產生試題系統能有效率地建置大規模的題庫，同時減少人為的干預而得以保持試題的隱密性。受惠於網路上充裕的文字資源，本研究發展一套克漏詞試題出題系統，利用既有的語料自動產生涵蓋各種不同主題的克漏詞試題。藉由分析歷屆大學入學考試的資料，系統可產生類似難度的模擬試題，並且得到出題人員在遴選測驗標的方面的規律性。在產生試題的過程中導入詞義辨析的演算法，利用詞典與selectional preference模型的輔助，分析句子中特定詞彙的語義，以擷取包含測驗編撰者所要測驗的詞義的句子，並以collocation為基礎的方法篩選誘答選項。實驗結果顯示系統可在每產生1.6道試題中，得到1道可用的試題。我們嘗試產生不同類型的試題，並將這套系統融入網路線上英文測驗的環境中，依學生的作答情形分析試題的鑑別度。 / Multiple-choice tests have proved to be an efficient tool for measuring students’ achievement. Manually constructing tests items, however, is a time- consuming and labor-intensive task. Harnessing the computing power of computers, computer-assisted item generation offers the possibility of creating large amount of items, thereby alleviating the problem of keeping the items secure. With the abundant text resource on the Web, this study develops a system capable of generating cloze items that cover a wide range of topics based on existing corpra. By analyzing training data from the College Entrance Examinations in Taiwan, we identify special regularities of the test items, and our system can generate items of similar style based on results of the analysis. We propose a word sense disambiguation-based method for locating sentences in which designated words carry specific senses, and apply collocation-based methods for selecting distractors. Experimental results indicate that our system was able to produce a usable item for every 1.6 items it returned. We try to create different types of items and integrate the reported item generator in a Web-based system for learning English. The outcome of on-line examinations is analyzed in order to estimate the item discrimination of the test items generated by our system.