Cette thèse présente l'étude de l'interaction de lumière cohérente avec une couche sub-longueur d'onde de vapeur alcaline atomique confinée en nano-cellule et applications pour la formation de résonances optiques étroites.Nous développons un modèle théorique décrivant l'interaction résonante de lumière laser avec la couche mince de vapeur alcaline en présence d'un champ magnétique. Nous montrons qu'en raison d'un régime transitoire d'interaction, seuls les atomes lents contribuent au signal et leur spectre de transmission est essentiellement sans effet Doppler. La nature des spectres obtenus fait de la spectroscopie de transmission, en nano-cellule, une technique pratique pour l'étude de transitions très rapprochées et l'évolution de leur comportement dans un champ magnétique. Des expériences réalisées pour des champs magnétiques jusqu'à 7000 G montrent un excellent accord entre théorie et expérience.Nous explorons aussi la rotation Faraday du plan de polarisation de la lumière lors de sa propagation dans la couche mince atomique. Bien que l'angle de rotation soit très faible, nous observons que les résonances des spectres de rotation Faraday sont plus étroites que celles de transmission. Enfin, nous étudions de nouvelles possibilités pour former des résonances optiques étroites et montrons qu'un traitement par deuxième dérivée des spectres de transmission donne le meilleur rétrécissement de raies parmi toutes les méthodes étudiées dans cette thèse. / This thesis presents the study of coherent light interaction with a sub-wavelength atomic alkali vapor layer confined in a nano-cell and applications for the formation of narrow optical resonances.We develop a theoretical model describing the resonant interaction of the laser light with the thin alkali vapor layer in the presence of an external static magnetic field. We show that due to a transient regime of interaction, only slow atoms contribute to the signal and their transmission spectrum is essentially Doppler-free. The nature of the obtained spectra makes the transmission spectroscopy from a nano-cell a convenient technique to perform studies of closely-spaced atomic transitions and investigate their behavior in magnetic fields. Experimental realizations for magnetic field up to 7000~G show an excellent agreement between theory and experiment.We also explore the Faraday rotation of the plane polarization of light with the propagation through the thin atomic slab. We see that despite a small angle of rotation, Faraday rotation spectra exhibit resonances narrower than that for transmission. At last, we investigate new possibilities to form narrow optical resonances in nano-cells and show that second derivation processing of transmission spectra yields the strongest line narrowing among all methods studied in this thesis.

Noise-immune cavity-enhanced optical heterodyne molecular spectro-metry (NICE-OHMS) is one of the most sensitive laser-based absorption techniques. The high sensitivity of NICE-OHMS is obtained by a unique combination of cavity enhancement (for increased interaction length with a sample) with frequency modulation spectrometry (for reduction of noise). Moreover, sub-Doppler detection is possible due to the presence of high intensity counter-propagating waves inside an external resonator, which provides an excellent spectral selectivity. The high sensitivity and selectivity make NICE-OHMS particularly suitable for trace gas detection. Despite this, the technique has so far not been often used for practical applications due to its technical complexity, originating primarily from the requirement of an active stabilization of the laser frequency to a cavity mode. The main aim of the work presented in this thesis has been to develop a simpler and more robust NICE-OHMS instrumentation without compro-mising the high sensitivity and selectivity of the technique. A compact NICE-OHMS setup based on a fiber laser and a fiber-coupled electro-optic modulator has been constructed. The main advantage of the fiber laser is its narrow free-running linewidth, which significantly simplifies the frequency stabilization procedure. It has been demonstrated, using acetylene and carbon dioxide as pilot species, that the system is capable of detecting relative absorption down to 3 × 10-9 on a Doppler-broadened transition, and sub-Doppler optical phase shift down to 1.6 × 10-10, the latter corresponding to a detection limit of 1 × 10-12 atm of C2H2. Moreover, the potential of dual frequency modulation dispersion spectrometry (DFM-DS), an integral part of NICE-OHMS, for concentration measurements has been assessed. This thesis contributes also to the theoretical description of Doppler-broadened and sub-Doppler NICE-OHMS signals, as well as DFM-DS signals. It has been shown that the concentration of an analyte can be deduced from a Doppler-broadened NICE-OHMS signal detected at an arbitrary and unknown detection phase, provided that a fit of the theoretical lineshape to the experimental data is performed. The influence of optical saturation on Doppler-broadened NICE-OHMS signals has been described theoretically and demonstrated experimentally. In particular, it has been shown that the Doppler-broadened dispersion signal is unaffected by optical saturation in the Doppler limit. An expression for the sub-Doppler optical phase shift, valid for high degrees of saturation, has been derived and verified experimentally up to degrees of saturation of 100.