The primary study of this thesis is the experimental realization of the non-equilibrium dynamics of a quantum inverted pendulum as examined in the collective spin dynamics of a spin-1 Bose-Einstein condensate. In order to compare experimental results with the simulation past the low depletion limit, current simulation techniques needed to be extended to model atomic loss. These extensions show that traditional measurements of the system evolution (e.g. measuring the mean and standard deviation of the evolving quantity) were insufficient in capturing the quantum nature of the evolution. It became necessary to look at higher order moments and cumulants of the distributions in order to capture the quantum fluctuations. Extending the implications of the loss model further, it is possible that the system evolves in a way previously unpredicted. Spin-mixing from a hyperbolic fixed point in the phase space and low noise atom counting form the core of the experiment to measure the evolution of the distributions of the spin populations. The evolution of the system is also compared to its classical analogue, the momentum-shortened inverted pendulum. The other experimental study in this thesis is mapping the mean-field phase space. The mean-field phase space consists of different energy contours that are divided into both phase-winding trajectories and closed orbits. These two regions are divided by a separatrix whose orbit has infinite period. Coherent states can be created fairly accurately within the phase space and allowed to evolve freely. The nature of their subsequent evolution provides the shape of the phase space orbit at that initial condition. From this analysis a prediction of the nature of the entire phase space is possible.

The primary study of this thesis is spin-nematic squeezing in a spin-1 condensate. The measurement of spin-nematic squeezing builds on the success of previous experiments of spin-mixing together with advances in low noise atom counting. The major contributions of this thesis are linking theoretical models to experimental results and the development of the intuition and tools to address the squeezed subspaces. Understanding how spin-nematic squeezing is generated and how to measure it has required a review of several theoretical models of spin-mixing as well as extending these existing models. This extension reveals that the squeezing is between quadratures of a spin moment and a nematic (quadrapole) moment in abstract subspaces of the SU(3) symmetry group of the spin-1 system. The identification of the subspaces within the SU(3) symmetry allowed the development of techniques using RF and microwave oscillating magnetic fields to manipulate the phase space in order to measure the spin-nematic squeezing. Spin-mixing from a classically meta-stable state, the phase space manipulation, and low noise atom counting form the core of the experiment to measure spin-nematic squeezing. Spin-nematic squeezing is also compared to its quantum optics analogue, two-mode squeezing generated by four-wave mixing. The other experimental study in this thesis is performing spin-dependent photo-association spectroscopy. Spin-mixing is known to depend on the difference of the strengths of the scattering channels of the atoms. Optical Feshbach resonances have been shown to be able to alter these scattering lengths but with prohibitive losses of atoms near the resonance. The possibility of using multiple nearby resonances from different scattering channels has been proposed to overcome this limitation. However there was no spectroscopy in the literature which analyzes for the different scattering channels of atoms for the same initial states. Through analysis of the initial atomic states, this thesis studies how the spin state of the atoms affects what photo-association resonances are available to the colliding atoms based on their scattering channel and how this affects the optical Feshbach resonances. From this analysis a prediction is made for the extent of alteration of spin-mixing achievable as well as the impact on the atom loss rate.

The physics of ultracold atomic gases has been the subject of a long standing theoretical and experimental research over the last half century. The development of evaporative cooling techniques and the realization of the first Bose-Einstein Condensate (BEC) in 1995 gave a great advantage to the field. A great experimental knowledge of the fundamental properties of BECs, such as long-range coherence, superfluidity and topological excitations, has now been acquired. On top of these advances, current research on ultracold atoms is also focusing on quantum simulations, which aim at building analogue models of otherwise difficult to compute physical systems in the lab. In this context, BECs, with their enhanced coherence, many-body dynamics and superfluid character offer a powerful platform for advances in the field. Shortly after the first realization of a BEC, research started also investigating the physics of quantum mixtures of a BECs, either composed of different atomic species or isotopes, or of atoms occupying different hyperfine states. The latter are known as spin mixtures, or spinor condensates. The presence of multiple components interacting through mutual contact interactions enriches the physics of the condensate, introducing ground states with magnetic ordering as well as spin dynamics, which can be order of magnitudes less energetic than the density one. On top of this, hyperfine states can be coherently coupled with an external resonant radiation. Interesting physics arises when the strength of the coupling is comparable with the energy of spin excitations, an example of which is given by the emergence of the internal Josephson effect. This regime has been the subject of intense theoretical studies in the past twenty years, however its experimental realization on ultracold atomic platforms have been proven to be challenging, with experiments strongly limited by coherence times of few tens of milliseconds. In fact, the small energy scale of spin excitations reflects in a high sensitivity coupling to environmental magnetic noise, which affects the resonant condition. The experimental apparatus on which I worked during my Ph.D. solve this problem employing a magnetic shield that surrounds the science chamber, attenuating external magnetic fields by 6 orders of magnitudes. During my Ph.D., I investigated the properties of a coherently coupled mixture of BEC of Sodium 23, performing different experiments in two atomic configurations. The first configuration consist of a mixture of hyperfine states, namely the |F=1, mF = -1> and |F=1, mF = +1>, coupled by a two-photon transition, which is characterized by miscibility in the ground state. Another configuration was instead realized working with a strongly immiscible mixture of |F=1, mF=-1> and |F=2, mF = -2>, realized through with a one photon transition. My first experiment was devoted to the characterization of different methods of manipulation of the coupled miscible mixture in an elongated quasi-1D geometry. In Local Density Approximation (LDA), The dynamics of the system, depends on the atom number difference, the relative phase, and coupling to mean field energy ratio, can be fully described as an internal Josephson junction. We characterized this dynamics on a sample an inhomogeneous spatial profile, developing three different protocols for state manipulations. In a second experiment, I developed a protocol to generate Faraday waves in an unpolarized miscible mixture. Faraday waves are classical non-linear waves characterized by a regular pattern, that originate in classical and quantum fluids via a parametric excitation in the fluid. Interestingly enough, this process resembles the phase of reheating of the early universe, where the oscillation of the inflaton field is thought to have excited particles out of the vacuum. In analogy with this phenomenon, the oscillation of the inflaton field can be simulated with the periodic modulation of the trapping potential. On top of this, in a spin mixture, the parametric modulation can excite either in-phase (density) modes or out-of-phase (spin) modes, as two possible elementary excitations are present in the system. By extracting the spatial periodicity of the generated pattern at different modulation frequencies, I was then able to measure the dispersion relations for both density and spin modes of the system. In the presence of the coherent coupling, when spin excitations becomes gapped, we further demonstrate the scaling of the gap with the strength of the coupling radiation. The third experiment I realized concerned the characterization of the magnetic ground state of a spatially extended immiscible mixture in the presence of the coherent coupling. The Hamiltonian of such a system is formally equivalent to a continuous version of the transverse field Ising model, which describes magnetic materials at zero temperature. In this mapping, a nonlinear interaction term arises from the ratio between the self-interaction energy and the strength of the coupling, which acts as the transverse field. As the ratio between the two quantities is varied above and below one, the ground state of the system spontaneously changes from a paramagnetic phase to an ordered ferromagnetic phase, featuring two equivalent and opposite magnetizations, a signature of the occurrence of a second order quantum phase transition (QPT). Furthermore, in the magnetic model, the degeneracy between the two ferromagnetic ground states can be broken by introducing an additional longitudinal field. In the atomic case, the role of this additional field is taken by the detuning between the coupling radiation and the resonant transition frequency of non-interacting atoms. I characterized the QPT developing protocols to manipulate the spin mixture in its spatially extended ground state, varying the longitudinal field. Leveraging on the inhomogeneity of a BEC trapped in the harmonic potential, a smooth variation of the spin self-interaction energy occurs spontaneously in space, introducing different magnetic regimes at fixed coupling strength. These protocols gave access to a characterization of static properties typical of magnetic materials, such as the presence of an hysteresis cycle. The occurrence of the phase transition was instead validated by a measurement of the magnetic susceptibility and corresponding fluctuations, which both show a divergence when crossing the QPT critical point. At last, I developed a protocol to smoothly manipulate the position of magnetic domain walls, the least energetic excitations in a ferromagnet. While the previous study focused on static properties, the last experimental investigation presented in this thesis was devoted to the study of the dynamics of the metastable ferromagnetic region of the BEC. As a result of the presence of an hysteresis cycle, it is possible to engineer states of the ferromagnetic energy landscape that are homogeneously prepared either in the global minimum, with trivial dynamics, or in the metastable, higher energy, local minima. In the latter case, a classical system should eventually decay towards the global minimum, driven by temperature fluctuations which overtop the energy barrier separating the two minima. For a quantum system described by a field theory, such as a ferromagnetic BEC, the decay towards the global minimum occurs by tunneling through the barrier, triggered by quantum fluctuations. The event of tunneling is known as False Vacuum Decay (FVD), and is of outstanding relevance also for high energy physics and cosmology, were the first theoretical models were developed. In the FVD model, the decay towards the global minimum, the true vacuum, is a stochastic process that occurs only if a resonant bubble of true vacuum is formed. Once formed, the bubble will eventually expand throughout the whole system, as the true vacuum is energetically favorable. The probability for such a bubble to form can be approximately calculated analytically in 1D, and should depend exponentially on the height of the barrier the field has to tunnel through. Due to the exponentially long time scale of the process, experimental observations of FVD were still lacking. Thanks to the enhanced coherence time of the superfluid ferromagnetic mixture, and to the precise control of the barrier height through the detuning from atomic resonance, we were able to observe the event of bubble nucleation in a ferromagnetic BEC. To corroborate the observation, I measured the characteristic timescale of the decay for different values of the control parameters. Results were successfully compared first with numerical simulation, and then validated by instanton theory.

The quantum properties of matter waves, in particular quantum correlations and entanglement are an important frontier in atom optics with applications in quantum metrology and quantum information. In this thesis, we report the first observation of sub-Poissonian fluctuations in the magnetization of a spinor 87Rb condensate. The fluctuations in the magnetization are reduced up to 10 dB below the classical shot noise limit. This relative number squeezing is indicative of the predicted pair-correlations in a spinor condensate and lay the foundation for future experiments involving spin-squeezing and entanglement measurements. We have investigated the limits of the imaging techniques used in our lab, absorption and fluorescence imaging, and have developed the capability to measure atoms numbers with an uncertainly < 10 atoms. Condensates as small as ≈ 10 atoms were imaged and the measured fluctuations agree well with the theoretical predictions. Furthermore, we implement a reliable calibration method of our imaging system based on quantum projection noise measurements. We have resolved the individual lattice sites of a standing-wave potential created by a CO2 laser, which has a lattice spacing of 5.3 µm. Using microwaves, we site-selectively address and manipulate the condensate and therefore demonstrate the ability to perturb the lattice condensate of a local level. Interference between condensates in adjacent lattice sites and lattice sites separated by a lattice site are observed.