We evaluate the long-term test–retest reliability and procedural validity of phobia diagnoses in the UM-CIDI, the version of the Composite International Diagnostic Interview, used in the US National Co-morbidity Survey (NCS) and a number of other ongoing large-scale epidemiological surveys. Test–retest reliabilities of lifetime diagnoses of simple phobia, social phobia, and agoraphobia over a period between 16 and 34 months were K = 0·46, 0·47, and 0·63, respectively. Concordances with the Structured Clinical Interview for DSM-III-R (SCID) were K = 0·45, 0·62, and 0·63, respectively. Diagnostic discrepancies with the SCID were due to the UM-CIDI under-diagnosing. Post hoc analysis demonstrated that modification of UM-CIDI coding rules could dramatically improve cross-sectional procedural validity for both simple phobia (K = 0·57) and social phobia (K = 0·95). Based on these results, it seems likely that future modification of CIDI questions and coding rules could lead to substantial improvements in diagnostic validity.