We discuss the appearance of quantum orders in the Hubbard model for interacting electrons, at half-filling. Such phases do not have local order parameters and need to be characterized by the quantum mechanical properties of their ground state. On one hand, we study the Mott transition from a metal to a spin liquid insulator in two dimensions, of potential relevance to some layered organic compounds. The correlation-driven transition occurs at fixed filling and involves fractionalization of the electron: upon entering the insulator, a Fermi surface of neutral spinons coupled to an internal gauge field emerges. We focus on the transport properties near the quantum critical point and find that the emergent gauge fluctuations play a key role in determining the universal scaling. Second, motivated by a class of three-dimensional transition metal oxides, the pyrochlore iridates, we study the interplay of non-trivial band topology and correlations. Building on the strong spin orbit coupling in these compounds, we construct a general microscopic Hubbard model and determine its mean-field phase diagram, which contains topological insulators, Weyl semimetals, axion insulators and various antiferromagnets. We also discuss the effects many-body correlations on theses phases. We close by examining a fractionalized topological insulator that combines the two main themes of the thesis: fractionalization and non-trivial band topology. Specifically, we study how the two-dimensional protected surface states of a topological Mott insulator interact with a three-dimensional emergent gauge field. Various correlation effects on observables are identified.

In this PhD thesis, we study the interplay between symmetry-breaking order and quantum-disordered phases in the milieu of frustrated quantum magnets, and further show how the excitation process of long-wavelength (semi-)classical modes in spin-orbit coupled antiferromagnets crucially depends on the nature and interactions of the underlying quantum quasiparticles. First, we focus on Kitaev's exactly solvable model for a Z2 spin liquid as a building block for constructing novel phases of matter, utilizing Majorana mean-field theory (MMFT) to map out phase diagrams and study occurring phases. In the Kitaev Kondo lattice, conduction electrons couple via a Kondo interaction to the local moments in the Kitaev model. We find at small Kondo couplings a fractionalized Fermi liquid (FL*) phase, a stable non-Fermi liquid where conventional electronic quasiparticles coexist with the deconfined excitations of the spin liquid. The transition between FL* and a conventional Fermi liquid is masked by an exotic (confining) superconducting phase which exhibits nematic triplet pairing, which we argue to be mediated by the Majorana fermions in the Kitaev spin liquid. We moreover study bilayer Kitaev models, where two Kitaev honeycomb spin liquids are coupled via an antiferromagnetic Heisenberg interaction. Varying interlayer coupling and Kitaev coupling anisotropy, we find both direct transitions from the spin liquid to a trivial dimer paramagnet as well as intermediate 'macrospin' phases, which can be studied by mappings to effective transverse-field Ising models. Further, we find a novel interlayer coherent pi-flux phase. Second, we consider the stuffed honeycomb Heisenberg antiferromagnet, where recent numerical studies suggest the coexistence of collinear Néel order and a correlated paramagnet, dubbed 'partial quantum disorder'. We elucidate the mechanism which drives the disorder in this model by perturbatively integrating out magnons to derive an effective model for the disordered sublattice. This effective model is close to a transition between two competing ground states, and we conjecture that strong fluctuations associated with this transition lead to disorder. Third, we study the generation of coherent low-energy magnons using ultrafast laser pulses in the spin-orbit coupled antiferromagnet Sr2IrO4, inspired by recent pump-probe experiments. While the relaxation dynamics of the system at long time scales can be well described semi-classically, the ultrafast excitation process is inherently non-classical. Using symmetry analysis to write down the most general coupling between electric field and spin operators, we subsequently integrate out high-energy spin fluctuations to derive induced effective fields which act to excite the low-energy magnon, constituting a generalized 'inverse Faraday effect'. Our theory reveals a tight relationship between induced fields and the two-magnon density of states.:1 Introduction 1.1 Frustrated antiferromagnets 1.2 Quantum spin liquids 1.3 Fractionalization and topological order 1.4 Spin-orbit coupling 1.5 Outline I Novel phases by building on Kitaev’s honeycomb model 2 Kitaev honeycomb spin liquid 2.1 Microscopic spin model and constants of motion 2.2 Majorana representation of spin algebra 2.3 Exact solution 2.3.1 Ground state 2.3.2 Correlations and dynamics 2.3.3 Thermodynamic properties 2.4 Z2 gauge structure 2.5 Toric code 2.6 Topological order 2.6.1 Superselection sectors and ground-state degeneracy 2.6.2 Topological entanglement entropy 2.6.3 Symmetry-enriched and symmetry-protected topological phases 3 Mean-field theory 3.1 Generalized spin representations 3.1.1 Parton constructions 3.1.2 SO(4) Majorana representation 3.2 Projective symmetry groups 3.3 Mean-field solution of the Kitaevmodel 3.4 Comparisonwithexactsolution 3.4.1 Spectral properties 3.4.2 Correlation functions 3.4.3 Thermodynamic properties 3.5 Generalized decoupling 3.6 Comparison to previous Abrikosov fermion mean-field theories of the Kitaev model 3.7 Discussion 4 Fractionalized Fermi liquids and exotic superconductivity in the Kitaev Kondo lattice 4.1 Metals with frustration 4.2 Local-moment formation and Kondo effect 4.2.1 Single Kondo impurity 4.2.2 Kondo lattices and heavy Fermi liquids 4.3 Fractionalized Fermi liquids 4.4 Construction of the Kitaev Kondo lattice 4.4.1 Hamiltonian 4.4.2 Symmetries 4.5 Mean-field decoupling of Kondo interaction 4.5.1 Solution of self-consistency conditions 4.6 Overview of mean-field phases 4.7 Fractionalized Fermi liquid 4.7.1 Results from mean-field theory 4.7.2 Perturbation theory beyond mean-field theory 4.8 Heavy Fermi liquid 4.9 Superconducting phases 4.9.1 Spontaneously broken U(1) phase rotation symmetry 4.9.2 Excitation spectrum and nematicity 4.9.3 Topological triviality 4.9.4 Group-theoretical classification 4.9.5 Pairing glue 4.10 Comparison with a subsequent study 4.11 Discussion and outlook 5 Bilayer Kitaev models 5.1 Model and stacking geometries 5.1.1 Hamiltonian 5.1.2 Symmetries and conserved quantities 5.2 Previous results 5.3 Mean-field decoupling and phase diagrams 5.3.1 AA stacking 5.3.2 AB stacking 5.3.3 σAC stacking 5.3.4 σ ̄AC stacking 5.4 Quantum phase transition in the AA stacking 5.4.1 Perturbative analysis 5.5 Phase transition in the σAC stacking 5.6 Macro-spin phases 5.6.1 KSL-MAC transition: Effective model for Kitaev dimers 5.6.2 DIM-MAC transition: Effective theory for triplon condensation 5.6.3 Macro-spin interactions and series expansion results 5.6.4 Antiferromagnet in the AB stacking 5.7 Stability of KSL and the interlayer-coherent π-flux phase 5.7.1 Perturbative stability of the Kitaev spin liquid 5.7.2 Spontaneous interlayer coherence near the isotropic point 5.8 Summary and discussion II Partial quantum disorder in the stuffed honeycomb lattice 6 Partial quantum disorder in the stuffed honeycomb lattice 6.1 Definition of the stuffed honeycomb Heisenberg antiferromagnet 6.2 Previous numerical results 6.3 Derivation of an effective model 6.3.1 Spin-wave theory for the honeycomb magnons 6.3.2 Magnon-central spin vertices 6.3.3 Perturbation theory 6.3.4 Instantaneous approximation 6.3.5 Truncation of couplings 6.3.6 Single-ion anisotropy 6.3.7 Discussion of most dominant interactions 6.4 Analysis of effective model 6.4.1 Classical ground states 6.4.2 Stability of classical ground states in linear spin-wave theory 6.4.3 Minimal model for incommensurate phase 6.4.4 Discussion of frustration mechanism in the effective model 6.5 Partial quantum disorder beyond the effectivemodel 6.5.1 Competition between PD and the (semi-)classical canted state 6.5.2 Topological aspects 6.5.3 Experimental signatures 6.6 Discussion 6.6.1 Directions for further numerical studies 6.6.2 Experimental prospects III Optical excitation of coherent magnons 7 Ultrafast optical excitation of magnons in Sr2IrO4 7.1 Pump-probe experiments 7.2 Previous approaches to the inverse Faraday effect and theory goals 7.3 Sr2IrO4 as a spin-orbit driven Mott insulator 7.4 Spin model for basal planes in Sr2IrO4 7.4.1 Symmetry analysis 7.4.2 Classical ground state and linear spin-wave theory 7.4.3 Mechanism for in-plane anisotropy 7.5 Pump-induced dynamics 7.5.1 Coupling to the electric field: Symmetry analysis 7.5.2 Keldysh path integral 7.5.3 Low-energy dynamics 7.5.4 Driven low-energy dynamics 7.6 Derivation of the induced fields 7.6.1 Perturbation theory 7.6.2 Evaluation of loop diagram 7.6.3 Analytical momentum integration in the continuum limit 7.6.4 Numerical evaluation of effective fields 7.7 Analysis of induced fields 7.7.1 Polarization and angular dependence 7.7.2 Two-magnon spectral features 7.8 Applications to experiment 7.8.1 Predictions for experiment 7.8.2 Magnetoelectrical couplings 7.9 Discussion and outlook 8 Conclusion and outlook 8.1 Summary 8.2 Outlook IV Appendices A Path integral methods B Spin-wave theory B.1 Holstein-Primakoff bosons B.2 Linear spin-wave theory B.2.1 Diagonalization via Bogoliubov transformation B.2.2 Applicability of linear approximation B.3 Magnon-magnon interactions B.3.1 Dyson's equation and 1/S consistency B.3.2 Self-energy from quartic interactions in collinear states on bipartite lattices C Details on the SO(4) Majorana mean-field theory C.1 SO(4) Matrix representation of SU(2) subalgebras C.2 Generalized SO(4) Majorana mean-field theory for a Heisenberg dimer (Chapter 3) C.3 Dimerization of SO(4) Majorana mean-field for the Kitaev model (Chapter3) C.4 Mean-field Hamiltonian in the Kitaev Kondo lattice (Chapter 4) C.5 Example solutions in the superconducting phase for symmetry analysis (Chapter4) D Linear spin-wave theory for macrospin phase in the bilayer Kitaev model (Chapter 5) D.1 Spin-wave Hamiltonian and Bogoliubov rotation D.2 Results and discussion E Extrapolation of the effective couplings for the staggered field h -> 0 (Chapter 6) E.1 xy interaction E.1.1 Leadingorder ~ S0 E.1.2 Subleadingorder ~ S^(−1) E.2 z-Ising interaction F Light-induced fields by analytical integration (Chapter 7) F.1 Method F.2 Results Bibliography

Dans la description à basse énergie de systèmes fortement corrélés, les champs de jauge peuvent émerger comme excitations collectives couplées à des quasiparticules fractionalisées. En particulier, certains aimants bidimensionnels dits frustrés sont décrits par un liquide de spin de Dirac comportant une symétrie de jauge U(1) compacte. La description infrarouge est donnée par une théorie conforme des champs, soit l'électrodynamique quantique en 2+1 dimensions avec 2N saveurs de fermions sans masse. Dans les aimants typiques, N=2 ou 4. L'aspect compact du champ de jauge implique également l'existence d'excitations topologiques, soit des instantons créés, dans ce contexte, par des opérateurs monopôles. Cette thèse porte sur les transitions de phase quantiques à partir d'un liquide de spin de Dirac et les propriétés des monopôles aux points critiques correspondants. Ces transitions sont induites en activant diverses interactions de type Gross-Neveu. Dans tous les cas à l'étude, la dimension d'échelle des monopôles est obtenue grâce à la correspondance état-opérateur et à un développement en 1/N. L'accent est d'abord mis sur une transition de confinement-déconfinement vers une phase antiferromagnétique décrite par la condensation d'un monopôle. Une levée de dégénérescence est observée au point critique alors que certaines dimensions d'échelle de monopôles sont réduites par rapport à leur valeur dans le liquide de spin de Dirac. Cette hiérarchie est caractérisée quantitativement en comparant les dimensions d'échelle dans des secteurs distincts du spin magnétique à l'ordre dominant en 1/N, puis qualitativement par une analyse en théorie des représentations. Des exposants critiques pour d'autres observables dans la théorie non compacte sont également obtenus. Enfin, deux transitions vers des liquides de spin topologiques, soit le liquide de spin chiral et le liquide de spin Z2, sont considérées. Les dimensions anormales des monopôles sont obtenues à l'ordre sous-dominant en 1/N. Ces résultats permettent de vérifier une dualité conjecturée avec un modèle bosonique et la valeur d'un coefficient universel pour les théories de jauge U(1) / In strongly correlated systems, gauge fields can emerge as collective excitations coupled to fractionalized quasiparticles. In particular, certain frustrated two-dimensional quantum magnets are described by a Dirac spin liquid which has a U(1) gauge symmetry. The infrared description is given by a conformal field theory, namely quantum electrodynamics in 2+1 dimensions with 2N flavours of massless fermions. In typical magnets, N=2 or 4. The compact aspect of the gauge field also implies the existence of topological excitations corresponding to instantons, which are created by monopole operators in this context. This thesis focuses on quantum phase transitions out of a Dirac spin liquid and the properties of monopoles at the corresponding critical points. These transitions are driven by activating various types of Gross-Neveu interactions. In all the cases studied, the scaling dimension of monopoles are obtained using the state-operator correspondence and a 1/N expansion. The confinement-deconfinement transition to an antiferromagnetic order produced by a monopole condensate is first studied. A degeneracy lifting is observed at the critical point, as certain monopoles have their scaling dimension reduced in comparison with the value in the Dirac spin liquid. This hierarchy is charactized quantitatively by comparing monopole scaling dimensions in distinct magnetic spin sector at leading-order in 1/N, and qualitatively by an analysis in representation theory. Critical exponents of various other operators are obtained in the non-compact model. Transitions to two topological spin liquids, namely a chiral spin liquid and a Z2 spin liquid, are also considered. Anomalous dimensions of monopoles are obtained at sub-leading order in 1/N. These results allow the verification of a conjectured duality with a bosonic model and the value of a universal coefficient in U(1) gauge theories.