Advanced process control solutions are oftentimes inadequate in their handling of uncertainty and disturbances. The main contribution of this work is to address this issue by providing solutions of immediate relevance to process control practitioners. To meet increasing performance demands, this work considers a Hidden Markov Model-based framework for describing non-stationary disturbance signals of practical interest such as intermittent drifts and abrupt jumps. The result is a more sophisticated model used by the state estimator for jump systems. At the expense of slightly higher computational costs (due to the state estimator), the proposed HMM disturbance model provides better tracking compared to a state estimator based on the commonly employed (in process control) integrated white noise disturbance model. Better tracking performance translates to superior closed loop performance without any redesign of the controller, through the typical assumption of separation and certainty equivalence. As a result, this provides a tool that can be readily adopted by process control practitioners. In line with this, the second aim is to develop approximate dynamic programming techniques for the rigorous control of nonlinear stochastic jump systems. The contribution is the creation of a framework that treats uncertainty in a systematic manner whilst leveraging existing off-the-shelf optimization solvers commonly employed by control practitioners.