Brain aging is a heterogeneous phenomenon, and this thesis illustrates how the course of aging can vary within individuals over time and between individuals as a function of age, sex, and genetic variability. We used two contrasts from magnetic resonance imaging (MRI), namely spin-lattice T1-weighted imaging, and quantitative susceptibility mapping (QSM) from gradient-echo images, to picture the aging brain, by means of morphometric measures and brain-iron concentrations. Within each study, the same rigorous imaging acquisitioning protocols were used over large samples sizes of 167-183 individuals, which contribute to the uniqueness of the studies. Most of the current knowledge about the aging brain rests on the foundation of cross-sectional age-related differences, and studies I and III contribute to current knowledge with longitudinal designs to investigate individual rates of change. The importance of genetic variation in relation to regional brain changes was addressed with a specific emphasis on functional polymorphisms involved in pro-inflammatory responses. These studies further shed light on the importance of bi-directional relations between structural integrity and maintained cognitive abilities over time. Study II is the largest study to date to have quantitative susceptibility estimates examined in healthy adults, and the first in-vivo report to show a lowering in overall subcortical brain iron estimates in women from midlife to old age. Studies I and III are unique by examining longitudinal differences in anatomical brain regions using high resolution images from a 4 Tesla scanner. Peripheral vascular risk factors were not strong determinants of either brain- or cognitive changes in the studied samples. The results are discussed in the context of cognitive reserve, the brain maintenance hypothesis, and potential influences of hormones, inflammation and oxidative stress.