Multiple-Input Multiple-Output (MIMO) systems have gained an enormous amount of attention as one of the most promising research areas in wireless communications. However, while MIMO systems have been extensively explored over the past decade, few schemes acknowledge the nonlinearity caused by the use of high power amplifiers (HPAs) in the communication chain. When HPAs operate near their saturation points, nonlinear distortions are introduced in the transmitted signals, and the resulting MIMO channel will be nonlinear. The nonlinear distortion is further exacerbated by the fading caused by the propagation channel. The goal of this thesis is: 1) to use neural networks (NNs) to model and identify nonlinear MIMO channels; and 2) to employ the proposed NN model in designing efficient detection techniques for these types of MIMO channels. In the first part of the thesis, we follow a previous work on modeling and identification of nonlinear MIMO channels, where it has been shown that a proposed block-oriented NN scheme allows not only good identification of the overall MIMO input-output transfer function but also good characterization of each component of the system. The proposed scheme employs an ordinary gradient descent based algorithm to update the NN weights during the learning process and it assumes only real-valued inputs. In this thesis, natural gradient (NG) descent is used for training the NN. Moreover, we derive an improved variation of the previously proposed NN scheme to avoid the input type restriction and allow for complex modulated inputs as well. We also investigate the scheme tracking capabilities of time-varying nonlinear MIMO channels. Simulation results show that NG descent learning significantly outperforms the ordinary gradient descent in terms of convergence speed, mean squared error (MSE) performance, and nonlinearity approximation. Moreover, the NG descent based NN provides better tracking capabilities than the previously proposed NN. The second part of the thesis focuses on signal detection. We propose a receiver that employs the neural network channel estimator (NNCE) proposed in part one, and uses the Zero-Forcing Vertical Bell Laboratories Layered Space-Time (ZF V-BLAST) detection algorithm to retrieve the transmitted signals. Computer simulations show that in slow time-varying environments the performance of our receiver is close to the ideal V-BLAST receiver in which the channel is perfectly known. We also present a NN based linearization technique for HPAs, which takes advantage of the channel information provided by the NNCE. Such linearization technique can be used for adaptive data predistortion at the transmitter side or adaptive nonlinear equalization at the receiver side. Simulation results show that, when higher modulation schemes (>16-QAM) are used, the nonlinear distortion caused by the use of HPAs is greatly minimized by our proposed NN predistorter and the performance of the communication system is significantly improved. / Thesis (Master, Electrical & Computer Engineering) -- Queen's University, 2007-08-08 14:55:50.489