In the current transmission system, the information bandwidth of the optical fiber communication system is limited by optical amplifier bandwidth, and more efficient use of bandwidth is a very important issue. Amplitude and phase shift keying (APSK) is an advanced modulation scheme to improve the spectral efficiency and can effectively increase the transmission capacity. Certainly, APSK format has a good potential for development. This master thesis is focusing on that to study the transmission performance of the APSK format using digital coherent receiver. As the extinction ratio (ER) of the amplitude shift keying (ASK) signal affects the performances of the ASK signal and phase shift keying (PSK) signal simultaneously, the effect of the ER on the APSK transmission performance was studied. The APSK format has the trade-off between the performances of both the ASK signal and the PSK signal through the ER of the ASK signal. To overcome this issue, a method named zero-nulling method had been proposed, and this method solved the trade-off issue properly. At first, the amendment is to modify the digital coherent receiver program, confirmed that the digital coherent receiver program can correctly resolve APSK signal. Second, in this master thesis, I set up a 500km long optical fiber to measure the transmission performance under APSK format, and try to use recirculating loop system to further increase the transmission distance to several thousand kilometers. Finally, by further modify the receiver program to achieve APSK modulation of the zero-nulling method, and can prove its feasibility.