Amplitude and Phase Shift Keying (APSK) format is one of the most attractive advanced modulation formats because of its good spectral efficiency. As the information bandwidth of the current optical fiber communication system is limited by the optical amplifier bandwidth, it is important to utilize the limited bandwidth effectively. This master thesis focuses on to study the transmission performance of the APSK format both theoretically and experimentally. At first, a theoretical study was conducted using a numerical simulation. As the Extinction Ratio (ER) of the Amplitude Shift Keying (ASK) signal affects the performances of both the ASK and the Phase Shift Keying (PSK) signals, the effect of the ER upon the transmission performance of the APSK format was studied. A clear trade-off between the performance of the ASK signal and the PSK signal due to the change of the ER was observed. Then, in order to improve the performance of the APSK format, a method to improve the transmission performance was proposed. This method was named as ¡§zero-nulling method¡¨, and it solved the trade-off issue caused by the ER. The effectiveness of this method was confirmed through the numerical simulation. Next, an experimental study was conducted. An experimental setup including 330km optical fiber transmission line was prepared, and it was used to confirm the results of the theoretical simulation. The performance trade-off between the ASK and the PSK signals due to the ER was confirmed experimentally. Finally, another experimental study was conducted. An experimental setup of 500km transmission line was used for this study. By adopting the recirculating loop experimental setup, the transmission length could be extended to a few thousand kilometers. The applicability of the ¡§zero-nulling method¡¨ was confirmed using this experimental setup.