This study was conducted to investigate the reliability of the Phonics Diagnostic Inventory (PDI), a curriculum-based, specific skill mastery measurement tool for diagnosing and informing the treatment of decoding weaknesses. First, a modified one-parameter item response theory model was employed to identify the properties of potential items for inclusion in each subtest to then inform the construction of subtests using the most reliable items. Second, the properties of each subtest were estimated and examined. The test information and test characteristic curves (TCC) for the newly developed forms are reported. Finally, the accuracy and sensitivity of PDI cut scores for each subtest were examined. Specifically, based upon established cut scores, the accuracy with which students would be identified as in need of support and those who are not in need of support were investigated. The PDI generated from this research was found to more reliably diagnose specific decoding deficits in mid-year second grade students than initially constructed forms. Research also indicates further examination of cut scores is warranted to maximize decision consistency. Implications for future studies are also discussed.