Selective robustness is a key feature of biochemical networks, conferring a fitness benefit to organisms living in dynamic environments. The (in-)sensitivity of a network to external perturbations results from the interaction between network dynamics, design, and enzyme kinetics. In this work, we focus on the subtle interplay between robustness and fragility. We describe a quantitative method for defining the fragility and robustness of system fluxes and metabolite concentrations to perturbations in enzyme activity. We find that for many mathematical models of metabolic pathways, the robustness is captured by a broad distribution of the robustness coefficients and demonstrate that, unlike fragility, robustness is not a conserved process. Using a combination of existing in silico models and novel sets of models, designed to allow specific network features of interest to be studied in isolation, we examine the effect of various network properties on the robustness of such pathways. We discuss the question of how to measure, in a meaningful way, the robustness of a pathway as a whole, defining several summary metrics which, in combination, can be used to compare the robustness of different pathways. We show that networking increases robustness, but that robustness is affected differently by varying aspects of complexity. The effect of system control loops on robustness is analysed and we find that, in general, the addition of such regulation increases pathway robustness. The evolution of flux robustness is also examined. We show that robustness in metabolic pathways is unlikely to simply be a by-product of selection for other pathway traits, highlighting several trade-offs that result from the evolution of robust systems. Finally, we extend our definition of robustness, defining robustness coefficients for cellular properties other than flux or metabolite concentration, and to perturbations other than changes in enzyme activity. Using the effect of benzoic acid on glycolysis as a case study, we show how such robustness coefficients can be used to give novel insights from experimental data.