The perirhinal cortex (PRC) is a medial temporal lobe (MTL) structure known to be involved in assessing whether an object is familiar (i.e., meaningful) or novel. Recent evidence shows that the PRC is sensitive to the familiarity of both whole object configurations and their parts, and suggests the PRC may modulate part familiarity responses in V2. Here, using functional magnetic resonance imaging (fMRI), we investigated age-related decline in the PRC's sensitivity to part/configuration familiarity and assessed its functional connectivity to visual cortex in young and older adults. Participants categorized peripherally presented silhouettes as familiar ("real-world") or novel. Part/configuration familiarity was manipulated via three silhouette configurations: Familiar (parts/configurations familiar), Control Novel (parts/configurations novel), and Part-Rearranged Novel (parts familiar, configurations novel). "Real-world" judgments were less accurate than "novel" judgments, although accuracy did not differ between age groups. The fMRI data revealed differential neural activity, however: In young adults, a linear pattern of activation was observed in left hemisphere (LH) PRC, with Familiar > Control Novel > Part-Rearranged Novel. Older adults did not show this pattern, indicating age-related decline in the PRC's sensitivity to part/configuration familiarity. A functional connectivity analysis revealed a significant coupling between the PRC and V2 in the LH in young adults only. Older adults showed a linear pattern of activation in the temporopolar cortex (TPC), but no evidence of TPC-V2 connectivity. This is the first study to demonstrate age-related decline in the PRC's representations of part/configuration familiarity and its covariance with visual cortex.