Spectral interferometry, also known as spectral-domain white light or low coherence interferometry, has seen numerous applications in sensing and metrology of physical parameters. It can provide phase or optical path information of interest in single shot measurements with exquisite sensitivity and large dynamic range. As fast spectrometer became more available in 21st century, spectral interferometric techniques start to dominate over time-domain interferometry, thanks to its speed and sensitivity advantage. In this work, a dual-modality phase/birefringence imaging system is proposed to offer a quantitative approach to characterize phase, polarization and spectroscopy properties on a variety of samples. An interferometric spectral multiplexing method is firstly introduced by generating polarization mixing with specially aligned polarizer and birefringence crystal. The retardation and orientation of sample birefringence can then be measured simultaneously from a single interference spectrum. Furthermore, with the addition of a Nomarski prism, the same setup can be used for quantitative differential interference contrast (DIC) imaging. The highly integrated system demonstrates its capability for noninvasive, label-free, highly sensitive birefringence, DIC and phase imaging on anisotropic materials and biological specimens, where multiple intrinsic contrasts are desired. Besides using different intrinsic contrast regime to quantitatively measure different biological samples, spectral multiplexing interferometry technique also finds an exquisite match in imaging single anisotropic nanoparticles, even its size is well below diffraction limit. Quantitative birefringence spectroscopy measurement over gold nanorod particles on glass substrate demonstrates that the proposed system can simultaneously determine the polarizability-induced birefringence orientation, as well as the scattering intensity and the phase differences between major/minor axes of single nanoparticles. With the anisotropic nanoparticles' spectroscopic polarizability defined prior to the measurement with calculation or simulation, the system can be further used to reveal size, aspect ratio and orientation information of the detected anisotropic nanoparticle. Alongside developing optical anisotropy imaging systems, the other part of this research describes our effort of investigating the sensitivity limit for general spectral interferometry based systems. A complete, realistic multi-parameter interference model is thus proposed, while corrupted by a combination of shot noise, dark noise and readout noise. With these multiple noise sources in the detected spectrum following different statistical behaviors, Cramer-Rao Bounds is derived for multiple unknown parameters, including optical pathlength, system-specific initial phase, spectrum intensity as well as fringe visibility. The significance of the work is to establish criteria to evaluate whether an interferometry-based optical measurement system has been optimized to its hardware best potential. An algorithm based on maximum likelihood estimation is also developed to achieve absolute optical pathlength demodulation with high sensitivity. In particular, it achieves Cramer-Rao bound and offers noise resistance that can potentially suppress the demodulation jump occurrence. By simulations and experimental validations, the proposed algorithm demonstrates its capability of achieving the Cramer-Rao bound over a large dynamic range of optical pathlengths, initial phases and signal-to-noise ratios. / PHD / Optical imaging is unique for its ability to use light to provide both structural and functional information from microscopic to macroscopic scales. As for microscopy, how to create contrast for better visualization of detected objects is one of the most important topic. In this work, we are aiming at developing a noninvasive, label-free and quantitative imaging technique based on multiple intrinsic contrast regimes, such as intensity, phase and birefringence. Spectral multiplexing interferometry method is firstly introduced by generating spectral interference with polarization mixing. Multiple parameters can thus be demodulated from single-shot interference spectrum. With Jones Matrix analysis, the retardation and orientation of sample birefringence can be measured simultaneously. A dual-modality phase/birefringence imaging system is proposed to offer a quantitative approach to characterize phase, polarization and spectroscopy properties on a variety of samples. The high integrated system can not only deliver label-free, highly sensitive birefringence, DIC and phase imaging of anisotropic materials and biological specimens, but also reveal size, aspect ratio and orientation information of anisotropic nanoparticles of which the size is well below diffraction limit. Alongside developing optical imaging systems based on spectral interferometry, the other part of this research describes our effort of investigating the sensitivity limit for general spectral interferometry based systems. The significance of the work is using Cramer-Rao Bounds to establish criteria to evaluate whether an optical measurement system has been optimized to its hardware best potential. An algorithm based on maximum likelihood estimation is also developed to achieve absolute optical pathlength demodulation with high sensitivity. In particular, it achieves Cramer-Rao bound and offers noise resistance that can potentially suppress the demodulation jump occurrence.