Tracking multiple targets with uncertain target dynamics is a difficult problem, especially with nonlinear state and/or measurement equations. Random finite set theory provides a rigorous foundation to multitarget tracking problems. It provides a framework to represent the full multitarget posterior in contrast to other conventional approaches. However, the computational complexity of performing multitarget recursion grows exponentially with the number of targets. The Probability Hypothesis Density (PHD) filter, which only propagates the first moment of the multitarget posterior, requires much less computational complexity. This thesis addresses some of the essential issues related to practical multitarget tracking problems such as tracking target maneuvers, stealthy targets, multitarget tracking in a distributed framework. With maneuvering targets, detecting and tracking the changes in the target motion model also becomes important and an effective solution for this problem using multiple-model based PHD filter is proposed. The proposed filter has the advantage over the other methods in that it can track a timevarying number of targets in nonlinear/ non-Gaussian systems. Recent developments in stealthy military aircraft and cruise missiles have emphasized the need to t rack low SNR targets. The conventional approach of thresholding the measurements throws away potential information and thus results in poor performance in tracking dim targets. The problem becomes even more complicated when multiple dim targets are present in the surveillance region. A PHD filter based recursive track-before-detect approach is proposed in this thesis to track multiple dim targets in a computationally efficient way. This thesis also investigates multiple target tracking using a network of sensors. Generally, sensor networks have limited energy, communication capability and computational power. The crucial consideration is what information needs to be transmitted over the network in order to perform online estimation of the current state of the monitored system, whilst attempting to minimize communication overhead. Finally, a novel continuous approximation approach for nonlinear/ non-Gaussian Bayesian tracking system based on spline interpolation is presented. The resulting filter has the advantages over the widely-known discrete particle based approximation approach in that it does not suffer from degeneracy problems and retains accurate density over the state space. The filter is general enough to be applicable to nonlinear/non-Gaussian system and the density could even be multi-modal. / Thesis / Candidate in Philosophy