Differential item functioning (DIF) can occur across age, gender, ethnic, and/or linguistic groups of examinee populations. Therefore, whenever there is more than one group of examinees involved in a test, a possibility of DIF exists. It is important to detect items with DIF with accurate and powerful statistical methods. While finding a proper DIP method is essential, until now most of the available methods have been dominated by applications to large scale testing contexts. Since the early 1990s, Ramsay has developed a nonparametric item response methodology and computer software, TestGraf (Ramsay, 2000). The nonparametric item response theory (IRT) method requires fewer examinees and items than other item response theory methods and was also designed to detect DIF. However, nonparametric IRT's Type I error rate for DIF detection had not been investigated. The present study investigated the Type I error rate of the nonparametric IRT DIF detection method, when applied to moderate-to-small-scale testing context wherein there were 500 or fewer examinees in a group. In addition, the Mantel-Haenszel (MH) DIF detection method was included. A three-parameter logistic item response model was used to generate data for the two population groups. Each population corresponded to a test of 40 items. Item statistics for the first 34 non-DIF items were randomly chosen from the mathematics test of the 1999 TEVISS (Third International Mathematics and Science Study) for grade eight, whereas item statistics for the last six studied items were adopted from the DIF items used in the study of Muniz, Hambleton, and Xing (2001). These six items were the focus of this study.

Differential item functioning (DIF) can occur across age, gender, ethnic, and/or linguistic groups of examinee populations. Therefore, whenever there is more than one group of examinees involved in a test, a possibility of DIF exists. It is important to detect items with DIF with accurate and powerful statistical methods. While finding a proper DIP method is essential, until now most of the available methods have been dominated by applications to large scale testing contexts. Since the early 1990s, Ramsay has developed a nonparametric item response methodology and computer software, TestGraf (Ramsay, 2000). The nonparametric item response theory (IRT) method requires fewer examinees and items than other item response theory methods and was also designed to detect DIF. However, nonparametric IRT's Type I error rate for DIF detection had not been investigated. The present study investigated the Type I error rate of the nonparametric IRT DIF detection method, when applied to moderate-to-small-scale testing context wherein there were 500 or fewer examinees in a group. In addition, the Mantel-Haenszel (MH) DIF detection method was included. A three-parameter logistic item response model was used to generate data for the two population groups. Each population corresponded to a test of 40 items. Item statistics for the first 34 non-DIF items were randomly chosen from the mathematics test of the 1999 TEVISS (Third International Mathematics and Science Study) for grade eight, whereas item statistics for the last six studied items were adopted from the DIF items used in the study of Muniz, Hambleton, and Xing (2001). These six items were the focus of this study. / Education, Faculty of / Educational and Counselling Psychology, and Special Education (ECPS), Department of / Graduate

Two procedures for detection of differential item functioning (DIF) for polytomous items were generalized to detection of differential testlet functioning (DTLF). The methods compared were the logistic discriminant function analysis procedure for uniform and non-uniform DTLF (LDFA-U and LDFA-N), and the Mantel score test procedure. Further analysis included comparison of results of DTLF analysis using the Mantel procedure with DIF analysis of individual testlet items using the Mantel-Haenszel (MH) procedure. Over 600 chi-squares were analyzed and compared for rejection of null hypotheses. Samples of 500, 1,000, and 2,000 were drawn by gender subgroups from the NELS:88 data set, which contains demographic and test data from over 25,000 eighth graders. Three types of testlets (totalling 29) from the NELS:88 test were analyzed for DTLF. The first type, the common passage testlet, followed the conventional testlet definition: items grouped together by a common reading passage, figure, or graph. The other two types were based upon common content and common process. as outlined in the NELS test specification.

Objective. The Cochrane risk of bias tool (CRoB) is one of the most widely used tools for assessing the risk of bias of clinical trials. However, it was criticized for its poor inter-rater reliability, lack of clear and detailed guidelines for its application, and no clear distinguishing between reporting quality from real quality in implementation. This study aims to develop a framework (or improved CRoB, iCRoB) so as to improve the inter-rater reliability of the CRoB in its first 4 domains: sequence generation, allocation concealment, blinding of participants and personnel, and blinding of outcome assessment, through providing: i) a structured pathway for assessing risk of bias assessment; and ii) a comprehensive dictionary of scenarios for each domain. / Methods. The study is consisted of 4 steps: / i) Step 1: Develop a step-by-step structured pathway for assessing the risk of bias. / ii) Step 2: Identify and summarize possible scenarios that are used in literature to describe a domain in clinical trials by using a qualitative content analysis approach. A random sample of 100 Cochrane systematic reviews (SRs) was taken from the Cochrane Database of Systematic Reviews. Each review was carefully scrutinized for this purpose. / iii) Step 3: Merge the scenarios identified from the sample with those already provided in the CRoB. The combined list of scenarios extends the current coverage of the CRoB and forms a more comprehensive dictionary of scenarios for use in the future. The bias assessment pathway and the new dictionary of scenarios in combination are the new components added or contribution to the CRoB to form the iCRoB. / v) Step 4: Conduct a randomized controlled study that allocated at random 8 raters equally into either using the CRoB or our new iCRoB. 150 clinical trials were randomly selected from the fore-mentioned 100 SRs for the inter-rater reliability comparison. Both inter-rater reliability among individual raters (measured with Fleiss’ κ) and that across rater pairs (measured with weighted Cohen’s κ) were computed. Data analyses were conducted by using STATA version 13.0. / Results. A structured pathway for systematically assessing bias was designed, which helps classify a study into one of 5 categories for each risk of bias domain based on the information provided in the report of a trial: Category A: a trial reports in details how a bias reduction method was conducted and it is also deemed by the assessor to be conducted adequately; Category B: a trial reports in details how a bias reduction method was conducted but it is deemed by the assessor to be conducted inadequately; Category C: a trial reports that a bias reduction method was conducted but no detailed description was given which can be used to judge whether it was done adequately; Category D: a trial reports that a bias reduction method was not conducted; Category E: a trial does not mention at all whether or not a bias reduction method was conducted. / A total of 34, 36, 26 and 20 scenarios were generated for sequence generation, allocation concealment, blinding of participants and personnel, and blinding of outcome assessment, respectively. We extended the current CRoB list of scenarios by a number of 20, 23, 26 and 20 respectively for the 4 bias reduction domains. / Our trial results showed that the iCRoB had a higher inter-rater reliability across rater pairs than the original CRoB for every bias reduction domain. The weighted κ was 0.71 and 0.81 for sequence generation respectively for CRoB and iCRoB; 0.53 and 0.61 for allocation concealment respectively for CRoB and iCRoB; 0.56 for blinding of participants and personnel in CRoB, 0.68 for blinding of participants and 0.70 for blinding of personnel ini CRoB; and 0.19 and 0.43 for blinding of outcome assessment respectively for CRoB and iCRoB. / Conclusion. We developed the iCRoB including a standard pathway and extended substantively the dictionary of scenarios for making the judgement on risk of bias in the reports of clinical trials. Our iCRoB showed a higher reliability than the current CRoB in all the domains examined. The iCRoB can be recommended for improving the assessment of bias in clinical trials. / 目的：考柯藍偏倚風險評估工具(CRoB)是最廣泛應用的用於評價臨床試驗偏倚風險的工具之一。然而，CRoB 有以下三個缺陷：評價者間信度低，缺乏明確和詳細的應用說明和沒有明確區分報告質量和方法學質量。本研究擬制定一個新的工具iCRoB 用以提高CRoB 前4 項指標的評價者間信度。這4項指標分別為隨機序列生成，分配隱藏，對研究對象和研究者實施盲法，和對結局評估者實施盲法。本研究通過以下2 點實現這一目的：i) 提供一個結構化路徑用以評估偏倚風險；ii) 為每個研究指標提供一個廣泛包含偏倚風險評估相關描述場景的字典。 / 方法：本研究包含以下4 個步驟： / 第1 步：制定一個用以評估偏倚風險的結構化路徑。 / 第2 步：從考柯藍系統綜述數據庫中隨機抽取100 篇系統綜述，應用定性內容分析法從中確定並總結出臨床試驗中與偏倚風險相關的可能的描述場景。 / 第3 步：將從100 個樣本中總結的描述場景與CRoB 中已有的場景合併，從而擴大CRoB 的描述場景的覆蓋範圍，得到一個更廣泛包含偏倚風險評估相關描述場景字典。偏倚風險評估的結構化路徑和包含場景描述的字典共同形成了本研究中新制定的iCRoB，用以評估臨床試驗的偏倚風險。 / 第4 步：在一個隨機對照研究中，8 名評價者被隨機平均分配至CRoB 組或者iCRoB 組。在上述100個系統綜述所納入的臨床試驗中隨機抽取150 個臨床試驗用以比較CRoB 和iCRoB 的評價者間信度。評價者間信度的比較包括個體評價者間信度（用Fleiss’κ 測量）和配對評價者間信度(用加權Cohen’s κ 測量)的比較。數據採用Stata 13.0 進行統計分析。 / 結果：本研究成功的制定了一個用於系統評價偏倚風險的結構化路徑，在該結構化路徑中，每個偏倚風險相關的指標在一個臨床研究中將分為以下5 類： / A 類：臨床試驗詳細描述了預防偏倚的措施的實施，根據描述可以判定該措施的實施能預防偏倚的產生; B 類：臨床試驗詳細描述了預防偏倚的措施的實施，根據描述可以判定該措施的實施不能預防偏倚的產生; C 類：臨床試驗報告採取了預防偏倚的措施，但未描述這一過程如何實施，從而無法判斷其實施是否正確; D 類：臨床試驗報告沒有採取任何預防偏倚的措施; E 類：臨床試驗沒有報告是否採取了預防偏倚的措施。 / 本研究分別為隨機序列生成，分配隱藏，對研究對象和研究者實施盲法，和對結局評估者實施盲法收集了34，36，26 和20 個描述場景。與CRoB 提供的描述場景比較，iCRoB 分別為隨機序列生成，分配隱藏，對研究對象和研究者實施盲法，和對結局評估者實施盲法增加了20，23，26 和20 個新的描述場景。 / 隨機對照試驗結果顯示，iCRoB 中每個研究指標的配對評價者間信度均高於CRoB，其中，隨機序列生成加權κ 為0.71（CRoB）和0.81（iCRoB），分配隱藏加權κ 為0.53（CRoB）和0.61（iCRoB），對研究對象和研究者實施盲法加權κ 為0.56（CRoB），對研究對象實施盲法加權κ 為0.68（iCRoB），對研究者實施盲法加權κ 為0.70（iCRoB），對結局評估者實施盲法加權κ 為0.19（CRoB）和0.43（iCRoB）。 / 結論：本研究通過制定一個由偏倚風險評估的結構化路徑和包含場景描述的字典組成的iCRoB，用以改善CRoB 中對臨床試驗中隨機序列生成，分配隱藏，對研究對象和研究者實施盲法，和對結局評估者實施盲法偏倚風險評估過程。相比於CRoB，iCRoB 在每個研究指標中均顯示出更好的配對評價者間信度。這些結果證明評價者間信度可以通過提供結構化偏倚風險評估路徑和更全面的描述場景字典而提高。 / Wu, Xinyin. / Thesis Ph.D. Chinese University of Hong Kong 2014. / Includes bibliographical references (leaves 93-105). / Abstracts also in Chinese. / Title from PDF title page (viewed on 09, September, 2016). / Wu, Xinyin. / Detailed summary in vernacular field only. / Detailed summary in vernacular field only. / Detailed summary in vernacular field only. / Detailed summary in vernacular field only. / Detailed summary in vernacular field only. / Detailed summary in vernacular field only. / Detailed summary in vernacular field only. / Detailed summary in vernacular field only.y066 / Detailed summary in vernacular field only. / Detailed summary in vernacular field only. / Detailed summary in vernacular field only.