In the recent years, the exponential Internet traffic growth projections place enormous transmission rate demand on the underlying information infrastructure at every level, from the long haul submarine transmission to optical metro networks. In recent years, optical transmission at 100 Gb/s Ethernet date rate has been standardized by ITU-T and IEEE forums and 400Gb/s and 1Tb/s rates per DWDM channel systems has been under intensive investigation which are expected to be standardized within next couple of years.To facilitate the implementation of 400GbE and 1TbE technologies, the new advanced modulation scheme combined with advanced forward error correction code should be proposed. Instead of using traditional QAM, we prefer to use some other modulation techniques, which are more suitable for current coherent optical transmission systems and can also deal with the channel impairments. In this dissertation, we target at improving the channel capacity by designing the new modulation formats. For the first part of the dissertation, we first describe the optimal signal constellation design algorithm (OSCD), which is designed by placing constellation points onto a two dimensional space. Then, we expand the OSCD onto multidimensional space and design its corresponding mapping rule. At last, we also develop the OSCD algorithm for different channel scenario in order to make the constellation more tolerant to different channel impairments. We propose the LLR-OSCD for linear phase noise dominated channel and NL-OSCD for nonlinear phase noise dominated channel including both self-phase modulation (SPM) and cross-phase modulation (XPM) cases. For the second part of the dissertation, we target at probability shaping of the constellation sets (non-uniform signaling). In the conventional data transmission schemes, the probability of each point in a given constellation is transmitted equally likely and the number of constellation sets is set to 2!. If the points with low energy are transmitted with larger probability then the others with large energy, the non- uniform scheme can achieve higher energy efficiency. Meanwhile, this scheme may be more suitable for optical communication because the transmitted points with large probabilities, which have small energy, suffer less nonlinearity. Both the Monte Carlo simulations and experiment demonstration of both OSCD and non-uniform signaling schemes indicate that our proposed signal constellation significantly outperforms QAM, IPQ, and sphere-packing based signal constellations.