Order-by-disorder is the phenomenon of the selection of a long-ranged ordered state by fluctuations in a many-body system. This mechanism, at first sight, seems paradoxical, since fluctuations (disorder) intuitively tend to suppress the order. However, when ObD happens, disorder works in favour of a particular ordered phase. Order-by-disorder can happen where an accidental degeneracy occurs in classical or mean field treatment of a system. These degeneracies, which are not due to exact symmetries of the system, can then be lifted by quantum or thermal fluctuations. The ObD phenomenon is ubiquitous in condensed matter systems with competing or frustrated interactions. Traditionally, the ObD is studied at $T = 0^+$ where the ground state of the system can be selected by quantum fluctuations. The study of ObD at temperature regimes near criticality, $T \lesssim Tc$ where transition happens from the paramagnetic phase to an ordered phase, however, have not received as much attention. In this thesis, we study the ObD phenomenon in three dimensional frustrated sys- tems close to criticality. We consider 3-component classical Heisenberg spins on pyrochlore lattice and FCC lattice. In the former, the spins interact via a Hamiltonian that can include the most general nearest-neighbour symmetry allowed bilinear interactions, long- range magnetostatic dipole-dipole interaction and second and/or third nearest-neighbour exchange interactions. However, in the latter, the Hamiltonian only consists of long-range magnetostatic dipole-dipole interactions. These two systems, correspond to insulating rare- earth pyrochlore oxides and rare-earth FCC salts. The mean field treatment shows, that accidental $O(4)$ and $U(1)$ symmetries emerge in two different regions of the parameter space of the Hamiltonian of pyrochlore system. While in the FCC system, an accidental $O(3)$ symmetry emerges at the mean field level. We show that fluctuations break these symmetries by respectively introducing cubic (in 4-vector and 3-vector models) and hexag- onal anisotropies to the free-energy of the system. To study these system beyond mean field approximation, we use Monte Carlo simulations, spin wave theory and we develop the E-TAP method which is an extended version of the method originally proposed by Thouless, Anderson and Palmer to study fluctuations in spin glasses.

In this thesis, we investigate strongly correlated phenomena in quantum spin systems. In the first part of this work, we study geometrically frustrated antiferromagnets (AFMs). Generalizing the SU(2) Heisenberg Hamiltonian to Sp(N) symmetry, we obtain, in the large-N limit, the mean-field phase diagrams for the planar pyrochlore and cubic AFMs. We then use gauge theories to consider fluctuation effects about their respective mean-field configurations. We find, in addition to conventional Neel states, a plethora of novel magnetically disordered phases: two kinds of spin liquids, Z2 in 2+1D and U(1)in 3+1D, and several valence bond solids such as two and three-dimensional plaquette and columnar singlet states. We use the same approach to study the diamond lattice AFM which possesses extended classical ground state degeneracy. We demonstrate that quantum and entropic fluctuations lift this degeneracy in different ways. In the second part of the thesis, we study ultracold spinor atoms confined in optical lattices. We first demonstrate the feasibility of experimental realization of rotor models using ultracold spin-one Bose atoms in a spin-dependent and disordered optical lattice. We show that the ground state of such disordered rotor models with quadrupolar interactions can exhibit biaxial nematic ordering in the disorder-averaged sense, and suggest an imaging experiment to detect the biaxial nematicity in such systems. Finally, using variational wavefunction methods, we study the Mott phases and superfluid-insulator transition of spin-three bosons in an optical lattice with an anisotropic two dimensional optical trap. We chart out the phase diagrams for Mott states with n = 1 and n = 2 atoms per lattice site. We show that the long-range dipolar interaction stabilizes a state characterized by antiferromagnetic chains made of ferromagnetically aligned spins. We also obtain the mean-field phase boundary for the superfluid-insulator transition, and show that inside the superfluid phase and near the superfluid-insulator phase boundary, the system undergoes a first order antiferromagnetic-ferromagnetic spin ordering transition.

In this thesis, we investigate strongly correlated phenomena in quantum spin systems. In the first part of this work, we study geometrically frustrated antiferromagnets (AFMs). Generalizing the SU(2) Heisenberg Hamiltonian to Sp(N) symmetry, we obtain, in the large-N limit, the mean-field phase diagrams for the planar pyrochlore and cubic AFMs. We then use gauge theories to consider fluctuation effects about their respective mean-field configurations. We find, in addition to conventional Neel states, a plethora of novel magnetically disordered phases: two kinds of spin liquids, Z2 in 2+1D and U(1)in 3+1D, and several valence bond solids such as two and three-dimensional plaquette and columnar singlet states. We use the same approach to study the diamond lattice AFM which possesses extended classical ground state degeneracy. We demonstrate that quantum and entropic fluctuations lift this degeneracy in different ways. In the second part of the thesis, we study ultracold spinor atoms confined in optical lattices. We first demonstrate the feasibility of experimental realization of rotor models using ultracold spin-one Bose atoms in a spin-dependent and disordered optical lattice. We show that the ground state of such disordered rotor models with quadrupolar interactions can exhibit biaxial nematic ordering in the disorder-averaged sense, and suggest an imaging experiment to detect the biaxial nematicity in such systems. Finally, using variational wavefunction methods, we study the Mott phases and superfluid-insulator transition of spin-three bosons in an optical lattice with an anisotropic two dimensional optical trap. We chart out the phase diagrams for Mott states with n = 1 and n = 2 atoms per lattice site. We show that the long-range dipolar interaction stabilizes a state characterized by antiferromagnetic chains made of ferromagnetically aligned spins. We also obtain the mean-field phase boundary for the superfluid-insulator transition, and show that inside the superfluid phase and near the superfluid-insulator phase boundary, the system undergoes a first order antiferromagnetic-ferromagnetic spin ordering transition.