Digital satellite communication systems are equipped with nonlinear amplifiers such as travelling wave tube (TWT) amplifiers at or near saturation for better efficiency. The TWT exhibits nonlinear distortion in both amplitude and phase (AM/AM and AM/PM) conversion, respectively. That is, in the digital satellite communication the transmission is disturbed not only by the non-linearity of transmitter amplifier, but also by the inter-symbol interference (ISI) with additive white Gaussian noise. To compensate the non-linearity of the transmitter amplifier and ISI, in this thesis, a new nonlinear compensation scheme consists of the predistorter and adaptive third-order Volterra-based equalizer, with the inverse QRD-RLS (IQRD-RLS) algorithm, which are located before and after the nonlinear channel, is proposed respectively. The third-order Volterra filter (TVF) equalizer based on the IQRD-RLS algorithm achieve superior performance, in terms of convergence rate, steady-state mean-squared error (MSE), and numerically stable. They are highly amenable to parallel implementation using array architectures, such as systolic arrays. The computer simulation results using the M-ary PSK modulation scheme are carried out the signal¡¦s constellation diagrams, the learning curve of the MSE and the bit error rate (BER) are compared with conventional least mean square (LMS), gradient adaptive lattice (GAL) and adaptive LMS with lattice pre-filter algorithms.