This thesis considers quantum impurity models that exhibit a quantum phase transition (QPT) between a Fermi liquid strong coupling (SC) phase, and a doubly-degenerate non-Fermi liquid local moment (LM) phase. We focus on what can be said from exact analytic arguments about the LM phase of these models, where the system is characterized by an SU(2) spin degree of freedom in the entire system. Conventional perturbation theory about the non-interacting limit does not hold in the non-Fermi liquid LM phase. We circumvent this problem by reformulating the perturbation theory using a so-called `two self-energy' (TSE) description, where the two self-energies may be expressed as functional derivatives of the Luttinger-Ward functional. One particular paradigmatic model that possesses a QPT between SC and LM phases is the pseudogap Anderson impurity model (PAIM). We use infinite-order perturbation theory in the interaction, U, to self-consistently deduce the exact low-energy forms of both the self-energies and propagators in each of the distinct phases of the model. We analyse the behaviour of the model approaching the QPT from each phase, focusing on the scaling of the zero-field single-particle dynamics using both analytical arguments and detailed numerical renormalization group (NRG) calculations. We also apply two `conserving' approximations to the PAIM. First, second-order self-consistent perturbation theory and second, the fluctuation exchange approximation (FLEX). Within the FLEX approximation we develop a numerical algorithm capable of self-consistently and coherently describing the QPT coming from both distinct phases. Finally, we consider a range of static spin susceptibilities that each probe the underlying QPT in response to coupling to a magnetic field.