Thesis advisor: Fazel Tafti / The search for an ideal quantum spin-liquid (QSL) material which can host a QSL ground state as well as exotic excitations has been one of the leading research topics in condensed matter physics over the past few decades. Out of all the proposals to realize the physics of a QSL, the Kitaev model is the most promising proposal with a QSL ground state. The Kitaev Hamiltonian is exactly solvable via fractionalization of its spin degrees of freedom into Majorana excitations, and it can be engineered in real materials. Among all the proposed Kitaev candidates, α-Li2IrO3, Na2IrO3, Li2RhO3, and α-RuCl3 are the most promising candidates. During my Ph.D. research I explored new physics related to Kitaev materials via modification of the symmetry and structural properties of these known Kitaev candidates. First, I studied how modification of the inter-layer chemistry can alter the thermodynamic properties of Kitaev candidate α-Li2IrO3 via an enhancement of the spin-orbit coupling (SOC) effect. The light, octahedrally-coordinated inter-layer Li atoms are replaced with heavier, linearly-coordinated Ag atoms to synthesize Ag3LiIr2O6. In addition to these structural modifications to the parent compound α-Li2IrO3, having heavier elements between the honeycomb layers in the Ag compound increased the effect of SOC in the honeycomb layers and led to a decrease in the long-range ordering temperature in Ag3LiIr2O6 compared to its parent compound. Second, I studied the effect of local crystal distortion in the presence of a weak SOC effect to explore a new spin-orbital state different from the Jeff=1/2 state. Based on theoretical predictions, the ground states of Kitaev materials can be tuned to other exotic spin-orbital states such as an Ising spin-1/2 state. To provide the proper conditions for a competition between the trigonal crystal distortion and the SOC effect, I modified the crystal environment around the magnetic elements in the parent compound Li2RhO3 via a topo-chemical method and synthesized Ag3LiRh2O6. An increase in the strength of trigonal distortion in Ag3LiRh2O6, in the presence of weak SOC, led to a transition from the Jeff=1/2 ground state (Kitaev limit) in the parent compound to an Ising spin-1/2 ground state (Ising limit) in the product. This change in spin-orbital state resulted in a dramatic change in magnetic behavior. Whereas Li2RhO3 shows a spin-freezing transition at 6 K, Ag3LiRh2O6 reveals a robust long-range antiferromagnetic transition at 94 K. This is the first realization of a change of ground state between the Kitaev and Ising limits in the same structural family. Lastly, I studied how the crystal symmetry can be an important factor in the physics of Kitaev materials. Honeycomb layered materials can be crystallized in space groups C2/m, C2/c, and P6_322. However, the crystal symmetry of most Kitaev candidates is described by the C2/m space group. We successfully synthesized a polymorph of a 3d Kitaev candidate, hexagonal Na2Co2TeO6 (P6_322 space group) in space group C2/m. The change in crystal symmetry of this cobalt tellurate replaced three anti-ferromagnetic (AFM) orders at 27, 15, 7 K in the hexagonal polymorph by a single AFM peak at 9.6 K in the monoclinic Na2Co2TeO6. / Thesis (PhD) — Boston College, 2023. / Submitted to: Boston College. Graduate School of Arts and Sciences. / Discipline: Physics.