Monitoring the accuracy of memory is an automatic but essential process of memory encoding and retrieval. Retrospective memory confidence judgments are making effective and efficient decisions based on one's memories. The neural processes involved in retrospective confidence ratings were investigated with EEG and fMRI using a recognition memory task designed such that participants also rated their confidence in their memory response. Correct trials (hits and correct rejections) were examined for differences related to the participants' level of confidence in their response. There were significant differences in electrophysiological activity (in the FN400 and the late parietal component) associated with confidence rating, with mean deflection increasing as confidence decreased. fMRI analysis revealed activity that appeared to be specific to the process of confidence rating. Activity was found to increase in the medial frontal, lateral frontal, and lateral parietal cortices as confidence decreases, but only for hits. In the lateral frontal, lateral parietal, and medial parietal cortices, activity decreased as confidence increased. These data indicate that there are neural mechanisms specifically related to making retrospective memory confidence judgments.