In the search for novel magnetic materials, systems with strong spin-orbit coupling are a focus. 5d Ir-oxides and 4d Ru-halide, in particular, are associated in this context with a flurry of new theoretical concepts, models, and predictions, and more recently to various exotic topological states. In this thesis, we use computational quantum-chemistry methods to determine nearest-neighbor (NN) magnetic interactions in such systems. We also explore different routes to tune NN exchange couplings and provide guidelines for material design. In the first chapter, an introduction to concepts of electron correlations, spin-orbit coupling and magnetic interactions is provided. Many-body quantum chemistry methods used to determine electronic and magnetic properties of the transition metal systems in this work are outlined in the second chapter. In chapter 3, we determine multiplet-structure, magnetic g factors as well as NN magnetic interaction for the edge-shared 4d5 honeycomb lattice-based system, i.e., α-RuCl3. We find that the the magnetic anisotropy shows up in the form of bond-dependent Kitaev couplings, which defines the largest superexchange energy scale in this system. Magnetic couplings obtained by mapping the ab initio data onto an effective spin Hamiltonian are then used in the the subsequent exact diagonalization calculation to retrieve the magnetic phase diagram as a function of second and third NN coupling. Further, in chapter 4, we investigate the effects of uniform pressure and strain on the magnetic interactions in honeycomb and related lattice-based systems. We find that the Heisenberg and Kitaev terms are affected differently: for strain, in particular, the Heisenberg component decreases more rapidly than the Kitaev counterpart. This suggests a scenario where strain can stabilize a spin liquid state in such materials. In chapter 5, we discuss another factor that allows to modify magnetic couplings, i.e., the electrostatics between layered stackings with different metallic species. We examine magnetic interactions between Ir moments in H3LiIr2O6, a recently proposed Kitaev spin liquid candidate, and clarify the effect of interlayer electrostatics on the anisotropic Kitaev exchange . We show that the precise position of H+ cations between magnetically active [LiIr2O6]3− honeycomb-like layers has a strong impact on the magnitude of Kitaev interactions. In the last chapter, we examine Ir-oxides on the pyrochlore lattice. In these corner-sharing systems the NN anisotropic exchange occurs in the form of antisymmetric exchange, also known as Dzyaloshinskii-Moriya (DM) coupling. Our calculations predict that a highly unusual regime can be realized in such systems due to the vanishing NN Heisenberg interaction, making the antisymmetric DM exchange to be the dominant interaction in the oxides where the Ir-O-Ir links show bond-angles less than 125◦. We also confirm the accuracy of the employed quantum-chemistry methods by reproducing experimental data for Sm2Ir2O7.:Table of contents 1 Introduction 1 1.1 Electronic correlations 2 1.2 Crystal fields and d-level splitting 5 1.3 Spin-orbit Coupling 8 1.4 Magnetic interactions 10 1.5 Conclusions 13 2 Quantum Chemistry Methods 15 2.1 Introduction 15 2.2 Motivation for using quantum chemical approach 17 2.3 The Hartree-Fock approach 19 2.4 Multiconfigurational approach 22 2.5 Multireference configuration interaction 26 2.5.1 Recent developments towards performing FCI 27 2.6 Embedded cluster approach 28 2.7 Conclusions 30 3 Anisotropic spin interactions in α-RuCl3 31 3.1 Introduction 31 3.2 Spin-orbit ground state and excitations 33 3.2.1 Structural details .34 3.2.2 Computational details 37 3.2.3 Results and Discussions 40 3.3 Intersite exchange interactions for j=1/2 moments 44 3.3.1 Kitaev-Heisenberg model and symmetric anisotropies 45 3.3.2 Computational details 49 3.3.3 Results and Discussion 53 3.4 Conclusions 61 x Table of contents 4 Strain and pressure tuned magnetic interactions in Kitaev materials 63 4.1 Introduction 64 4.2 Qualitative analysis: Kitaev-Heisenberg model 65 4.3 Quantitative analysis: ab initio results 66 4.3.1 Computational approach 69 4.3.2 Results and discussion 70 4.4 Experimental results for pressurized α-RuCl3 74 4.4.1 Pressure induced dimerization 75 4.4.2 Ab initio calculations 76 4.5 Conclusions 78 5 Impact of inter-layer species on in-plane magnetism in H3LiIr2O6 79 5.1 Introduction 79 5.2 Structural details 81 5.3 Computational approach 82 5.4 Results and discussion 85 5.4.1 Magnetic couplings 85 5.4.2 Phase diagram and longer-range interactions 86 5.4.3 Position of H cations and effect on in-plane interactions 88 5.4.4 Angle dependence, the Kitaev limit 91 5.5 Conclusions 92 6 Anisotropic spin interactions in pyrochlore iridates 95 6.1 Introduction 95 6.2 Structural details 97 6.3 Computational details 98 6.3.1 Embedded cluster and basis sets 98 6.3.2 Quantum chemistry calculations 99 6.3.3 Effective spin model Hamiltonian 99 6.4 Results and Discussion 101 6.4.1 Magnetic couplings 101 6.4.2 Spin Dynamics 103 6.4.3 Magnetic ground state 105 6.5 Conclusions 109 Summary 111

We discuss the appearance of quantum orders in the Hubbard model for interacting electrons, at half-filling. Such phases do not have local order parameters and need to be characterized by the quantum mechanical properties of their ground state. On one hand, we study the Mott transition from a metal to a spin liquid insulator in two dimensions, of potential relevance to some layered organic compounds. The correlation-driven transition occurs at fixed filling and involves fractionalization of the electron: upon entering the insulator, a Fermi surface of neutral spinons coupled to an internal gauge field emerges. We focus on the transport properties near the quantum critical point and find that the emergent gauge fluctuations play a key role in determining the universal scaling. Second, motivated by a class of three-dimensional transition metal oxides, the pyrochlore iridates, we study the interplay of non-trivial band topology and correlations. Building on the strong spin orbit coupling in these compounds, we construct a general microscopic Hubbard model and determine its mean-field phase diagram, which contains topological insulators, Weyl semimetals, axion insulators and various antiferromagnets. We also discuss the effects many-body correlations on theses phases. We close by examining a fractionalized topological insulator that combines the two main themes of the thesis: fractionalization and non-trivial band topology. Specifically, we study how the two-dimensional protected surface states of a topological Mott insulator interact with a three-dimensional emergent gauge field. Various correlation effects on observables are identified.

We discuss the appearance of quantum orders in the Hubbard model for interacting electrons, at half-filling. Such phases do not have local order parameters and need to be characterized by the quantum mechanical properties of their ground state. On one hand, we study the Mott transition from a metal to a spin liquid insulator in two dimensions, of potential relevance to some layered organic compounds. The correlation-driven transition occurs at fixed filling and involves fractionalization of the electron: upon entering the insulator, a Fermi surface of neutral spinons coupled to an internal gauge field emerges. We focus on the transport properties near the quantum critical point and find that the emergent gauge fluctuations play a key role in determining the universal scaling. Second, motivated by a class of three-dimensional transition metal oxides, the pyrochlore iridates, we study the interplay of non-trivial band topology and correlations. Building on the strong spin orbit coupling in these compounds, we construct a general microscopic Hubbard model and determine its mean-field phase diagram, which contains topological insulators, Weyl semimetals, axion insulators and various antiferromagnets. We also discuss the effects many-body correlations on theses phases. We close by examining a fractionalized topological insulator that combines the two main themes of the thesis: fractionalization and non-trivial band topology. Specifically, we study how the two-dimensional protected surface states of a topological Mott insulator interact with a three-dimensional emergent gauge field. Various correlation effects on observables are identified.