The increasing demand for high-mobility and high data rate in wireless communications results in constraints and problems in the limited radio spectrum, multipath fading, and delay spread. The multiple-input multiple-output (MIMO) system has been generally considered as one of the key technologies for the next generation wireless communication systems. MIMO systems which utilize multiple antennas in both the transmit side and the receive side can overcome the abovementioned challenges since they are able to increase the channel capacity and the spectrum usage efficiency without the need for additional channel bandwidth. The detection algorithm is a big bottleneck in MIMO systems. Generally, it is expected to fulfill two main goals simultaneously: low computational complexity and good error rate performance. However, the existing detection algorithms are either too complicated or suffering from very bad error-rate performance. The purpose of this thesis is to comprehensively investigate the detection algorithms of MIMO systems, and based on that, to develop new methods which can reduce the computational complexity while retain good system performance. Firstly, the background and the principle of MIMO systems and the previous work on the MIMO decoding algorithms conducted by other researchers are thoroughly reviewed. Secondly, the geometrical analysis of the signal detection is investigated, and a geometric decoding algorithm which can offer the optimum BLER performance is proposed. Thirdly, the semidefinite relaxation (SDR) detection algorithms are extended to high-order modulation MIMO systems, and a novel SDR detector for 256-QAM constellations is proposed. The theoretical analysis on the tightness and the complexity are conducted. It demonstrates that the proposed SDR detector can offer better BLER performance, while its complexity is in between those of its two counterparts. Fourthly, we combine the SDR detection algorithms with the sphere decoding. This is helpful for reducing the computational complexity of the traditional sphere decoding since shorter initial radius of the hyper sphere can be obtained. Finally, the novel lattice-reduction-aided SDR detectors are proposed. They can provide near-optimum error rate performance and achieve the full diversity gain with very little computational complexity added compared with the stand-alone SDR detectors. / published_or_final_version / Electrical and Electronic Engineering / Doctoral / Doctor of Philosophy