In narrow electron bands in which the Coulomb interaction energy becomes comparable to the bandwidth, interactions can drive new quantum phases. In this dissertation, we achieve narrow bands by twisting two atomically thin layers of the semiconducting van der Waals material WSe₂. The resulting moiré potential from the twist angle modulates the electronic bands, yielding minibands of tens of meV on the valence band. We perform transport measurements at cryogenic temperatures and observe signatures of collective phases over twist angles that range from 4 to 5.1°. At half-band filling, a correlated insulator appeared that is tunable with both twist angle and displacement field. Near the boundary between ordered and disordered quantum phases, several experiments have demonstrated metallic behaviour that defies the Landau Fermi paradigm. We find that the metal-insulator transition as a function of both density and displacement field is continuous. At the metal–insulator boundary, the resistivity displays strange metal behaviour at low temperatures, with dissipation comparable to that at the Planckian limit. Further into the metallic phase, Fermi liquid behaviour is recovered at low temperature, and this evolves into a quantum critical fan at intermediate temperatures, before eventually reaching an anomalous saturated regime near room temperature. An analysis of the residual resistivity indicates the presence of strong quantum fluctuations in the insulating phase. We further show via magnetotransport measurements that new correlated electronic phases can exist independent of moiré commensurability, and are instead driven by weak interactions in twisted WSe₂. The first of these phases is an antiferromagnetic metal that is driven by proximity to the van Hove singularity (vHS), which trails a range of incommensurate dopings. The temperature, magnetic field and density dependence of the Hall effect carry signatures of the reconstructed Fermi surface due to itinerant magnetic ordering. The second is an excitonic metal-insulator phase that exists at high external magnetic field in the vicinity of half-filling of the moiré superlattice. For a 4.2° sample, magnetic field dependence of the longitudinal resistance shows metallic behavior at fields above 5 T, but transitions to an insulating state above ∼ 24 T. A detailed analysis of of the Landau fans and the high field 𝝆_𝜘𝛾 near the gap rules out the possibility of a trivial insulator. We propose an Ising excitonic insulator as the most likely scenario. Moreover, in the electron-imbalanced excitonic metal, a set of correlated Landau levels emerge. The observation of tunable collective phases in a simple band, which hosts only two holes per unit cell at full filling, establishes twisted bilayer transition metal dichalcogenides as an ideal platform to study correlated physics in two dimensions on a triangular lattice.

In this PhD thesis, we study the interplay between symmetry-breaking order and quantum-disordered phases in the milieu of frustrated quantum magnets, and further show how the excitation process of long-wavelength (semi-)classical modes in spin-orbit coupled antiferromagnets crucially depends on the nature and interactions of the underlying quantum quasiparticles. First, we focus on Kitaev's exactly solvable model for a Z2 spin liquid as a building block for constructing novel phases of matter, utilizing Majorana mean-field theory (MMFT) to map out phase diagrams and study occurring phases. In the Kitaev Kondo lattice, conduction electrons couple via a Kondo interaction to the local moments in the Kitaev model. We find at small Kondo couplings a fractionalized Fermi liquid (FL*) phase, a stable non-Fermi liquid where conventional electronic quasiparticles coexist with the deconfined excitations of the spin liquid. The transition between FL* and a conventional Fermi liquid is masked by an exotic (confining) superconducting phase which exhibits nematic triplet pairing, which we argue to be mediated by the Majorana fermions in the Kitaev spin liquid. We moreover study bilayer Kitaev models, where two Kitaev honeycomb spin liquids are coupled via an antiferromagnetic Heisenberg interaction. Varying interlayer coupling and Kitaev coupling anisotropy, we find both direct transitions from the spin liquid to a trivial dimer paramagnet as well as intermediate 'macrospin' phases, which can be studied by mappings to effective transverse-field Ising models. Further, we find a novel interlayer coherent pi-flux phase. Second, we consider the stuffed honeycomb Heisenberg antiferromagnet, where recent numerical studies suggest the coexistence of collinear Néel order and a correlated paramagnet, dubbed 'partial quantum disorder'. We elucidate the mechanism which drives the disorder in this model by perturbatively integrating out magnons to derive an effective model for the disordered sublattice. This effective model is close to a transition between two competing ground states, and we conjecture that strong fluctuations associated with this transition lead to disorder. Third, we study the generation of coherent low-energy magnons using ultrafast laser pulses in the spin-orbit coupled antiferromagnet Sr2IrO4, inspired by recent pump-probe experiments. While the relaxation dynamics of the system at long time scales can be well described semi-classically, the ultrafast excitation process is inherently non-classical. Using symmetry analysis to write down the most general coupling between electric field and spin operators, we subsequently integrate out high-energy spin fluctuations to derive induced effective fields which act to excite the low-energy magnon, constituting a generalized 'inverse Faraday effect'. Our theory reveals a tight relationship between induced fields and the two-magnon density of states.:1 Introduction 1.1 Frustrated antiferromagnets 1.2 Quantum spin liquids 1.3 Fractionalization and topological order 1.4 Spin-orbit coupling 1.5 Outline I Novel phases by building on Kitaev’s honeycomb model 2 Kitaev honeycomb spin liquid 2.1 Microscopic spin model and constants of motion 2.2 Majorana representation of spin algebra 2.3 Exact solution 2.3.1 Ground state 2.3.2 Correlations and dynamics 2.3.3 Thermodynamic properties 2.4 Z2 gauge structure 2.5 Toric code 2.6 Topological order 2.6.1 Superselection sectors and ground-state degeneracy 2.6.2 Topological entanglement entropy 2.6.3 Symmetry-enriched and symmetry-protected topological phases 3 Mean-field theory 3.1 Generalized spin representations 3.1.1 Parton constructions 3.1.2 SO(4) Majorana representation 3.2 Projective symmetry groups 3.3 Mean-field solution of the Kitaevmodel 3.4 Comparisonwithexactsolution 3.4.1 Spectral properties 3.4.2 Correlation functions 3.4.3 Thermodynamic properties 3.5 Generalized decoupling 3.6 Comparison to previous Abrikosov fermion mean-field theories of the Kitaev model 3.7 Discussion 4 Fractionalized Fermi liquids and exotic superconductivity in the Kitaev Kondo lattice 4.1 Metals with frustration 4.2 Local-moment formation and Kondo effect 4.2.1 Single Kondo impurity 4.2.2 Kondo lattices and heavy Fermi liquids 4.3 Fractionalized Fermi liquids 4.4 Construction of the Kitaev Kondo lattice 4.4.1 Hamiltonian 4.4.2 Symmetries 4.5 Mean-field decoupling of Kondo interaction 4.5.1 Solution of self-consistency conditions 4.6 Overview of mean-field phases 4.7 Fractionalized Fermi liquid 4.7.1 Results from mean-field theory 4.7.2 Perturbation theory beyond mean-field theory 4.8 Heavy Fermi liquid 4.9 Superconducting phases 4.9.1 Spontaneously broken U(1) phase rotation symmetry 4.9.2 Excitation spectrum and nematicity 4.9.3 Topological triviality 4.9.4 Group-theoretical classification 4.9.5 Pairing glue 4.10 Comparison with a subsequent study 4.11 Discussion and outlook 5 Bilayer Kitaev models 5.1 Model and stacking geometries 5.1.1 Hamiltonian 5.1.2 Symmetries and conserved quantities 5.2 Previous results 5.3 Mean-field decoupling and phase diagrams 5.3.1 AA stacking 5.3.2 AB stacking 5.3.3 σAC stacking 5.3.4 σ ̄AC stacking 5.4 Quantum phase transition in the AA stacking 5.4.1 Perturbative analysis 5.5 Phase transition in the σAC stacking 5.6 Macro-spin phases 5.6.1 KSL-MAC transition: Effective model for Kitaev dimers 5.6.2 DIM-MAC transition: Effective theory for triplon condensation 5.6.3 Macro-spin interactions and series expansion results 5.6.4 Antiferromagnet in the AB stacking 5.7 Stability of KSL and the interlayer-coherent π-flux phase 5.7.1 Perturbative stability of the Kitaev spin liquid 5.7.2 Spontaneous interlayer coherence near the isotropic point 5.8 Summary and discussion II Partial quantum disorder in the stuffed honeycomb lattice 6 Partial quantum disorder in the stuffed honeycomb lattice 6.1 Definition of the stuffed honeycomb Heisenberg antiferromagnet 6.2 Previous numerical results 6.3 Derivation of an effective model 6.3.1 Spin-wave theory for the honeycomb magnons 6.3.2 Magnon-central spin vertices 6.3.3 Perturbation theory 6.3.4 Instantaneous approximation 6.3.5 Truncation of couplings 6.3.6 Single-ion anisotropy 6.3.7 Discussion of most dominant interactions 6.4 Analysis of effective model 6.4.1 Classical ground states 6.4.2 Stability of classical ground states in linear spin-wave theory 6.4.3 Minimal model for incommensurate phase 6.4.4 Discussion of frustration mechanism in the effective model 6.5 Partial quantum disorder beyond the effectivemodel 6.5.1 Competition between PD and the (semi-)classical canted state 6.5.2 Topological aspects 6.5.3 Experimental signatures 6.6 Discussion 6.6.1 Directions for further numerical studies 6.6.2 Experimental prospects III Optical excitation of coherent magnons 7 Ultrafast optical excitation of magnons in Sr2IrO4 7.1 Pump-probe experiments 7.2 Previous approaches to the inverse Faraday effect and theory goals 7.3 Sr2IrO4 as a spin-orbit driven Mott insulator 7.4 Spin model for basal planes in Sr2IrO4 7.4.1 Symmetry analysis 7.4.2 Classical ground state and linear spin-wave theory 7.4.3 Mechanism for in-plane anisotropy 7.5 Pump-induced dynamics 7.5.1 Coupling to the electric field: Symmetry analysis 7.5.2 Keldysh path integral 7.5.3 Low-energy dynamics 7.5.4 Driven low-energy dynamics 7.6 Derivation of the induced fields 7.6.1 Perturbation theory 7.6.2 Evaluation of loop diagram 7.6.3 Analytical momentum integration in the continuum limit 7.6.4 Numerical evaluation of effective fields 7.7 Analysis of induced fields 7.7.1 Polarization and angular dependence 7.7.2 Two-magnon spectral features 7.8 Applications to experiment 7.8.1 Predictions for experiment 7.8.2 Magnetoelectrical couplings 7.9 Discussion and outlook 8 Conclusion and outlook 8.1 Summary 8.2 Outlook IV Appendices A Path integral methods B Spin-wave theory B.1 Holstein-Primakoff bosons B.2 Linear spin-wave theory B.2.1 Diagonalization via Bogoliubov transformation B.2.2 Applicability of linear approximation B.3 Magnon-magnon interactions B.3.1 Dyson's equation and 1/S consistency B.3.2 Self-energy from quartic interactions in collinear states on bipartite lattices C Details on the SO(4) Majorana mean-field theory C.1 SO(4) Matrix representation of SU(2) subalgebras C.2 Generalized SO(4) Majorana mean-field theory for a Heisenberg dimer (Chapter 3) C.3 Dimerization of SO(4) Majorana mean-field for the Kitaev model (Chapter3) C.4 Mean-field Hamiltonian in the Kitaev Kondo lattice (Chapter 4) C.5 Example solutions in the superconducting phase for symmetry analysis (Chapter4) D Linear spin-wave theory for macrospin phase in the bilayer Kitaev model (Chapter 5) D.1 Spin-wave Hamiltonian and Bogoliubov rotation D.2 Results and discussion E Extrapolation of the effective couplings for the staggered field h -> 0 (Chapter 6) E.1 xy interaction E.1.1 Leadingorder ~ S0 E.1.2 Subleadingorder ~ S^(−1) E.2 z-Ising interaction F Light-induced fields by analytical integration (Chapter 7) F.1 Method F.2 Results Bibliography