Condensed matter physics provides us with an opportunity to explore a large variety of systems with diverse properties. Central to the understanding of these systems is a characterization of the nature of their ground states and low energy excitation. Often, such systems show various forms of emergent properties that are absent in the microscopic level. Identification of such emergent phases of condensed matter form an important avenue of research in the field. In this thesis example of such phases and their associated phase transitions have been studied. The work presented here may be broadly divided into two themes: construction of the theoretical framework for understanding materials already studied experimentally, and, trying to provide new theoretical avenues which may be relevant for understanding future experiments. In these studies we shall explore some unconventional phases and phase transitions that may occur in condensed matter systems. A comprehensive understanding of the properties of such unconventional phases and phase transitions is important in the context of the large array of experimentally studied materials that regularly defy conventional wisdom in more than one way. The thesis consists of two distinct parts. In the first part we study three problems in frustrated magnets. The second part consists of studies of the tunnelling spectroscopy of metal-insulator-superconductor junctions in graphene. Studies in frustrated magnets have opened up the possibility of existence of a whole range of phases beyond the already known magnetically ordered ones. Some of these new phases, like the spin nematic or the valence bond solid, display some other conventional order themselves. Others, like the much sort after spin liquid phases displays a whole new kind of order that cannot be captured through the celebrated Landau’s classification of phases on the basis of symmetry breaking and associated order parameters. The phase transitions in these systems are also equally interesting and lead to intriguing possibilities that demand new modes of analysis. In this part of the thesis we shall study the different properties of three magnets with spin-1/2, 1 and 3/2 respectively. We start by providing an introduction to frustrated spin systems in Chapter [1]. The origin of antiferromagnetic interactions in Mott insulators is discussed and the concept of frustration of magnetic interaction is explained. We also point out the causes that may destroy magnetic order in spin systems, particularly the role of quantum fluctuations in presence or absence of magnetic frustration. This is followed with a brief outline of various magnetically ordered and disordered ground states with particular emphasis on the description of the later. We also give a brief outline of various properties of such phases and associated quantum phase transitions particularly noting the influences of quantum interferences encoded in the Berry phase terms. A brief description of the finite temperature properties is also provided. We end an outline of various experimentally relevant compounds that requires comprehensive understanding, some of which have been addressed in this thesis. In Chapter [2] we study the properties of a spin-nematic state in context of the recently discovered spin-1 Mott insulator Nickel Gallium Sulphide (NiGa2S4). This isotropic triangular lattice compound shows no spin ordering till low temperatures. We propose that it may have a particular type of spin-nematic ground state and explain the experimentally observed properties of the compound on the basis of our proposal. Starting from a two band Hubbard model description, relevant for the compound, we derive the Bilinear Biquadratic spin Hamiltonian. We then show, within mean field theory, that this Hamiltonian describes a transition from the spiral state to a ferro-nematic state as a function of the ratio of bilinear and biquadratic couplings. We also study the possible effects of small pinning disorder andmagnetic field and suggest experiments that can possibly distinguish the proposed nematic state from others. In Chapter [3] we explore the effects of the magneto-elastic coupling in the spin-3/2 B-site chromite spinel Cadmium Chromite (CdCr2O4). In this compound the spins form a pyrochlore lattice. Nearest neighbour spins interact antiferromagnetically. Due to frustration the system does not order at low temperatures and instead goes into a classical spin liquid state. Such a cooperative paramagnet is very susceptible to external perturbations which may relieve their frustration. In CdCr2O4, at lower temperatures the magnetic frustration is relieved by distorting the lattice through a first order magnetoelastic transition. Thus the compound presents a case where the relevant perturbation to the frustrated spin interactions is provided by spin-phonon coupling. An effect of such perturbations on a cooperative paramagnet is of general interest and all aspects of this are not understood presently. We take the initial step of characterizing the spin-phonon interaction in detail. Based on recent sound velocity experiments, we construct a microscopic theory for the sound velocity renormalization due to the spin-phonon coupling and explain the recent experimental data obtained by S. Zherlitsyn et al. using our theory we can explain the dependence of the sound velocity on temperature as well as magnetic field. We also construct a Landau theory to explain (qualitatively) the behaviour of sound velocity across the magneto-structural transition. Further, we discuss the effects due to the small Dzyaloshinskii-Moriya interaction that may be present in these compounds. In Chapter [4] we study the possibility of a direct second order quantum phase transition from spiral to dimer phase in two dimensional antiferromagnets. Such transitions between phases with incompatible symmetries are forbidden within conventional Landau Ginzburg-Wilson paradigm of critical phenomena. Early works showed that when the spiral is destroyed by long wavelength fluctuations a fractionalized Z2 spin liquid is obtained. In this work we show an alternative way–the quantum destruction of the spiral magnet. We argue that, when the defects of the spiral phase proliferate and condense, their associated Berry phase automatically leads to dimerization. We apply our theory to study concrete lattice models where such transitions may be observed. This transition is an example of a Landau forbidden deconfined quantum phase transition. The proposed critical theory is naturally written in terms of fractional degrees of freedom which emerge right at the critical point. These fractional particles interact with each other through emergent gauge fields and are deconfined right at the critical point (but are confined in either of the two adjoining phases). We argue, based on existing results, that the monopoles of the gauge field are dangerously irrelevant right at the critical point rendering the later noncompact. The critical point is characterized by an emergent global U (1) conservation law that is absent in the microscopic model, a typical feature of a deconfined quantum critical point. The resultant field theory belongs to the class of anisotropic NCCP3 class which may be studied numerically in future to understand its critical properties. In modern condensed matter physics the emergence of new and novel phases of matter have often been associated with the presence of strong correlations. Indeed, strongly correlated systems seem to harbour in them the potential to realize some of the most unconventional and exotic emergent phases of matter. However in graphene, which is a single layer of graphite, the emergence of novel properties, as present experiments suggest, is due to its unique band structure and not a fallout of intricate correlation effects. Band structure studies of graphene suggest that the material is a zero gap semiconductor with the low energy excitations resembling massless Dirac quasi-particles. The consequence of this is immediate and interesting. It has lead to the possibility of exploring the physics of relativistic fermions in two spatial dimensions and much of this has been studied with great vigour in the last five years. In our studies, presented in Chapter [5], we explore one of the many consequence of this emergent Dirac structure of the low energy quasi-particles, namely the properties of metal-insulator-superconductor junctions of graphene. The twin effect of Klein tunneling of Dirac fermions (and associated transmission resonances) and Andreev reflection (both specular and retro) sets them aside from their conventional counterparts. The graphene normal metal-insulator-superconductor (NIS) junctions show strikingly different properties like oscillations in the sub-gap tunneling conductance as a function of both barrier strength and width. We make a detailed study of this for arbitrary barrier strengths and widths with and without Fermi-surface mismatch between the normal and the superconducting sides. The amplitude of these oscillations are maximum for aligned Fermi surface and vanishes for large Fermi surface mismatch. We provide an understanding for this unconventional behaviour of graphene NIS junctions. We also suggest experimental tests for our theory. Such experimental verification will reveal one more remarkable emergent property in a condensed matter system.

Dans ce manuscrit, nous présentons une étude expérimentale d'un gaz de Bose de spin 1 avec des interactions antiferromagnétiques avec des atomes de sodium ultra-froids dans l'état hyperfin F=1. Les trois composantes Zeeman sont piégées simultanément dans des pièges dipolaires optiques. Nous obtenons un condensat de Bose-Einstein spineur par refroidissement évaporatif et nous étudions ses propriétés magnétiques. Il y a deux types d’interactions dans le système: des interactions de contact qui ne changent pas les populations des composantes Zeeman et des interactions d'échange de spin qui les modifient. Une compétition entre l'énergie Zeeman et l'énergie d'échange impose l'ordre magnétique dans le système.Nous étudions dans un premier temps les phases magnétiques de condensats de Bose-Einstein spineurs a température quasi nulle. L'état fondamental comporte deux phases qui sont observées en variant le champ magnétique (donc l'énergie Zeeman quadratique) et la magnétisation de l'échantillon. Dans la phase antiferromagnétique, le spin de l'échantillon est simplement selon l'axe du champ magnétique. Dans la phase polaire, une composante transverse apparait pour minimiser l'énergie Zeeman. Pour une magnétisation nulle, le condensat spineur forme un nématique de spin. Cet état, nommé par analogie avec la phase nématique dans les cristaux liquides, est caractérisée par des fluctuations de spin orthogonales à un axe particulier, mais sans préférer une des deux direction sur cet axe. Dans chacune des deux phases, l'ordre nématique se manifeste par un minimisation de la longueur du spin transverse en imposant une valeur particulière ($pi$) de la phase relative des composantes Zeeman ${theta = phi_{+1} + phi_{-1} - 2 phi_{0}}$. Nous mesurons la longueur du spin transverse en analysant le bruit de spin après une rotation.Dans un second temps, nous étudions la thermodynamique d'un gaz de Bose de spin 1 près de la température critique pour la condensation de Bose-Einstein. Nous mesurons plusieurs scénarios de condensation séquentiels en fonction de la magnétisation et du champ magnétique. La température critique mesurée révèle que les interactions ont un effet important quand la condensation d'une composante se fait en présence d'un condensat dans une autre composante. Nous utilisons une théorie d'Hartree-Fock simplifiée, en négligeant les interactions d’échange de spin. Nous constatons que les résultats expérimentaux sont en bon accord. Cependant, pour de bas champs magnétiques, le diagramme de phase thermodynamique est largement modifié par les interactions d'échange de spin, ce qui pose de nouvelles questions sur leur rôle a température finie. / In this manuscript, we present an experimental study of a Spin 1 Bose gas with antiferromagnetic interactions with ultracold sodium atoms in the F=1 manifold. The three Zeeman components are trapped simultaneously in optical dipole traps. By performing evaporative cooling, we obtain quasi-pure spinor Bose-Einstein condensates of which we study the magnetic properties. There are two types of interactions between the constituents of the system: Contact interactions that do not change the Zeeman populations and spin-exchange contact interactions that do. A competition between Zeeman energy and the spin-exchange energy sets the magnetic ordering in the system.We first study the magnetic phases of spinor Bose-Einstein condensates near zero temperature. The ground state present two phases that are observed by varying the magnetic field (hence the quadratic Zeeman energy) and the magnetization of the sample. In the antiferromagnetic phase, the spin of the sample is purely along the direction of the magnetic field. In the broken-axisymmetry phase, a transverse component appears in order to minimize the Zeeman energy. For zero magnetization, the spinor condensate forms a spin nematic. This state, named in analogy with the liquid crystal nematic phase, is characterized by spin fluctuations orthogonal to a particular axis, with no preferred direction along that axis. In both phases, spin nematic order manifests as a minimization of the transverse spin length that is realized by enforcing a particular value ($pi$) of the relative phase of the Zeeman components $theta = phi_{+1} + phi_{-1} - 2 phi_0$. We measure the transverse spin length by analyzing spin noise after a spin rotation.Second, we study the thermodynamics of an antiferromagnetic spin 1 Bose gas next to the critical temperature for Bose-Einstein condensation. We measure several sequential condensation scenarii depending on the magnetization and the magnetic field. The measured critical temperatures reveal a large effect of interactions when one of the Zeeman component condenses in presence of a condensate in another component. We use a simplified Hartree-Fock theory, neglecting the spin exchange interactions and note a good agreement with our data. However, for low magnetic fields, the thermodynamic phase diagram is strongly modified which raises new open questions about the role of spin exchange interactions at finite temperatures.