We perceive the world through sensory experience. Sensory information is registered and processed by our brain in a modality specific fashion. Interestingly, studies have shown that the visual cortex of early but not late blind subjects is able to respond to touch or sound (Sadato et al., 1996; Buchel et al., 1998; Weeks et al., 2000; Gougoux et al., 2009). Here, we investigated whether sensory parcellation in adult cortex is innate or is acquired during early postnatal life in an experience-dependent manner. Furthermore, we studied the anatomical substrates and molecular pathways possibly involved in cross-modal activation and its plasticity. First, mice were reared from birth in total darkness until adulthood (DR) to replicate the human blind condition. Cross-modal activity and the underlying circuitry were analyzed. We found that DR visual cortex was strongly activated by sound stimulation using functional imaging, single-unit recording, and c-Fos immunohistochemistry. Functional analysis was followed by anatomical tracing studies, which showed ectopic projections from the auditory thalamus and auditory cortex into the secondary visual area in DR animals. The second half of our study looked at how visual experience affects cross-modal plasticity. We found that cross-modal activity and ectopic connectivity is present in normally reared young mice (25 postnatal days: P25). Normal sensory experience through the first two months of postnatal life was sufficient to decrease the number of ectopic inputs. Interestingly, exposing DR mice to visual experience as adults established transient functional sensory specificity in the visual cortex without eliminating the ectopic anatomical inputs. Lastly, we tested several molecular pathways that can potentially regulate cross-modal plasticity. We found that myelin signaling and cholinergic modulation controls the duration of cross-modal plasticity and consolidates sensory modularization. Overall, our work proposes a model of how cross-modal inputs into early sensory areas are pruned or retained depending on early life experience. This study provides insight into how the cortex develops functional specificity, and help approach disorders that exhibit abnormal sensory integration and disrupted neuronal connectivity such as Autism Spectrum Disorder.