In the last few decades, the study of many-body quantum systems far from equilibrium has risen to prominence, with exciting developments on both experimental and theoretical physics fronts. In this dissertation, we will focus particularly on the adiabatic gauge potential (AGP), which is the generator of adiabatic deformations between quantum eigenstates and also related to "fidelity susceptibility", as our lens into the general phenomenon. In the first two projects, the AGP is studied in the context of counter-diabatic driving protocols which present a way of generating adiabatic dynamics at an arbitrary pace. This is quite useful as adiabatic evolution, which is a common strategy for manipulating quantum states, is inherently a slow process and is, therefore, susceptible to noise and decoherence from the environment. However, obtaining and implementing the AGP in many-body systems is a formidable task, requiring knowledge of the spectral properties of the instantaneous Hamiltonians and control of highly nonlocal multibody interactions. We show how an approximate gauge potential can be systematically built up as a series of nested commutators, remaining well-defined in the thermodynamic limit. Furthermore, the resulting counter-diabatic driving protocols can be realized up to arbitrary order without leaving the available control space using tools from periodically-driven (Floquet) systems. In the first project, this driving protocol was successfully implemented on the electronic spin of a nitrogen vacancy in diamond as a proof of concept and in the second project, it was extended to many-body systems, where it was shown the resulting Floquet protocols significantly suppress dissipation and provide a drastic increase in fidelity. In the third project, the AGP is studied in the context of quantum chaos wherein it is found to be an extremely sensitive probe. We are able to detect transitions from non-ergodic to ergodic behavior at perturbation strengths orders of magnitude smaller than those required for standard measures. Using this alternative probe in two generic classes of spin chains, we show that the chaotic threshold decreases exponentially with system size and that one can immediately detect integrability-breaking (chaotic) perturbations by analyzing infinitesimal perturbations even at the integrable point. In some cases, small integrability-breaking is shown to lead to anomalously slow relaxation of the system, exponentially long in system size. This work paves the way for further studies in various areas such as quantum computation, quantum state preparation and quantum chaos.