Particle accelerators are machines to accelerate and store charged particle beams, such as electrons or protons, to the energy levels for various scientific applications. There are three basic types of particle accelerators: linear accelerators (linac), storage-ring (or circular) accelerators, and recirculating accelerators. The third type, also the most recent one, is designed to accelerate a particle beam in a short section of linac, circulate and then continue to accelerate it for energy boost or decelerate it for energy recovery. The modern recirculating machines possess the advantages to both accelerate and preserve the beam with high beam quality, as well as efficiently reuse the accelerating components. As modern accelerators push toward the high-brightness or high-intensity frontier by demanding particles in a highly charged bunch to concentrate in an ever-decreasing beam phase space, the interaction amongst particles via their self-generated electromagnetic fields can potentially lead to coherent instabilities of the beam and thus pose significant challenges to the machine design and operation. Microbunching instability (MBI) has been one of the most challenging issues for such high-brightness or high-intensity beam transport, as it would degrade lasing performance in the fourth-generation light sources, reduce cooling efficiency in electron cooling facilities, and eventually compromise the luminosity of colliding beams in lepton or lepton-hadron colliders. The dissertation work will focus on the MBI in modern recirculating electron accelerators. The research attempts to develop a comprehensive theoretical formulation of MBI with aspects including among various degrees of freedoms the beam itself, the beamline lattice optics, and incorporation of all relevant collective effects that the beam encounters, for example the coherent synchrotron radiation (CSR) and the longitudinal space charge (LSC) effects. This dissertation includes the following seven themes: 1) Development and generalization of MBI theory to arbitrary linear lattices and coupled beams with constant and varying energies; 2) Construction of CSR impedance models from steady state to transient state and from high to low energy regime; 3) Numerical implementation of the developed theory as a fast and numerical-noise-free Vlasov solver and benchmarking with massive particle tracking simulation; 4) Exploration of multistage cascaded amplification mechanism of CSR microbunching development; 5) Control of CSR-induced MBI in multi-bend transport or recirculation arcs; 6) Study of more aspects of microbunched structures in beam phase spaces; and 7) Study of MBI for magnetized beams and confirming the suppression of MBI for a recent cooler design for Jefferson Lab Electron Ion Collider project. / Ph. D. / Particle accelerators are machines to accelerate and store charged particle beams, such as electrons or protons, to the energy levels for various scientific applications. There are three basic types of particle accelerators: linear accelerators (linac), storage-ring (or circular) accelerators, and recirculating accelerators. The third type, also the most recent one, is designed to accelerate a particle beam in a short section of linac, circulate and then continue to accelerate it for energy boost or decelerate it for energy recovery. The modern recirculating machines possess the advantages to both accelerate and preserve the beam with high beam quality, as well as eciently reuse the accelerating components. As modern accelerators push toward the high-brightness or high-intensity frontier by demanding particles in a highly charged bunch to concentrate in an ever-decreasing beam phase space, the interaction amongst particles via their self-generated electromagnetic fields can potentially lead to coherent instabilities of the beam and thus pose significant challenges to the machine design and operation. Microbunching instability (MBI) has been one of the most challenging issues for such high-brightness or high-intensity beam transport, as it would degrade lasing performance in the fourth-generation light sources, reduce cooling eciency in electron cooling facilities, and eventually compromise the luminosity of colliding beams in lepton or lepton-hadron colliders. The dissertation work will focus on the MBI in modern recirculating electron accelerators. The research attempts to develop a comprehensive theoretical formulation of MBI with aspects including among various degrees of freedoms the beam itself, the beamline lattice optics, and incorporation of all relevant collective e↵ects that the beam encounters, for example the coherent synchrotron radiation (CSR) and the longitudinal space charge (LSC) e↵ects. This dissertation includes the following seven themes: 1) Development and generalization of MBI theory to arbitrary linear lattices and coupled beams with constant and varying energies; 2) Construction of CSR impedance models from steady state to transient state and from high to low energy regime; 3) Numerical implementation of the developed theory as a fast and numerical-noise-free Vlasov solver and benchmarking with massive particle tracking simulation; 4) Exploration of multistage cascaded amplification mechanism of CSR microbunching development; 5) Control of CSR-induced MBI in multi-bend transport or recirculation arcs; 6) Study of more aspects of microbunched structures in beam phase spaces; and 7) Study of MBI for magnetized beams and confirming the suppression of MBI for a recent cooler design for Je↵erson Lab Electron Ion Collider project.