Rutting is one of the major distresses in asphalt pavements and it increases road roughness and traps water, which leads to wet-weather accidents due to the loss of tire-pavement friction and hydroplaning. The fundamental mechanisms of rutting have not been well addressed because of the complexity of asphalt mixtures. A comprehensive characterization of the asphalt mixtures in compression was accomplished by mechanistically modeling the inherent anisotropy, viscoelasticity, viscoplasticity and viscofracture of the material. The inherent anisotropy due to preferentially oriented aggregates was characterized by a microstructural parameter (i.e., modified vector magnitudes) which could be rapidly and accurately measured by lateral surface scanning tests and physically related to anisotropic modulus ratio. The anisotropic viscoelasticity was represented by complex moduli and Poisson's ratios in separate orthogonal directions that were determined by an efficient testing protocol. Master curve models were proposed for the magnitude and phase angle of these complex variables. The viscoplasticity were intensively modeled by an anisotropic viscoplastic model which incorporated 1) modified effective stresses to account for the inherent and stress-induced anisotropy; 2) a new model to provide a smooth and convex yield surface and address the material cohesion and internal friction; 3) a non-associated flow rule to consider the volumetric dilation; and 4) a temperature and strain rate dependent strain hardening function. The viscofracture resulting from the crack growth in compression led to the stress-induced anisotropy and was characterized by anisotropic damage densities, the evolution of which was modeled by the anisotropic pseudo J-integral Paris' laws. Results indicated that the undamaged asphalt mixtures were inherently anisotropic and had vertical to horizontal modulus ratios from 1.2 to 2.0 corresponding to the modified vector magnitudes from 0.2 and 0.5. The rutting would be underestimated without including the inherent anisotropy in the constitutive modeling. Viscoelastic and viscoplastic deformation developed simultaneously while the viscofracture deformation occurred only during the tertiary flow, which was signaled by the increase of phase angle. Axial and radial strain decomposition methods were proposed to efficiently separate the viscoplasticity and viscofracture from the viscoelasticity. Rutting was accelerated by the occurrence of cracks in tertiary flow. The asphalt mixture had a brittle (splitting cracks) or ductile (diagonal cracks) fracture when the air void content was 4% and 7%, respecitvely. The testing protocol that produced the material properties is efficient and can be completed in one day with simple and affordable testing equipment. The developed constitutive models can be effectively implemented for the prediction of the rutting in asphalt pavements under varieties of traffic, structural, and environmental conditions.