As the bandwidth has become much cheaper in the recent years, video applications are more popular than before. However, the demand of high video resolution, high frame rate, or high bit-depth has continued to increase more rapidly than the cost of video transmission and storage bandwidth. It requires more efficient compression techniques, and hence many international video coding standards are developed in the past decades such as the MPEG-1/2/4 part 2, H.264/MPEG-4 part 10 AVC and the latest High Efficiency Video Coding (HEVC) standards. The main objective of this thesis is to consider the problems in analyzing the characteristics of video signals and providing efficient compression and transmission solutions in both H.264/AVC and HEVC video systems. Three main parts of this work are briey summarized below. / The first part concerns transform coding. Transform coding has been widely used to remove spatial redundancy of prediction residuals in the modern video coding standards. However, since the residual blocks exhibit diverse characteristics in a video sequence, conventional sinusoidal transforms with fixed transform kernels may result in low coding efficiency. To tackle this problem, we propose a novel content adaptive transform framework for H.264/AVC-based video coding. We propose to utilize pixel rearrangement to dynamically adjust the transform kernels to adapt to the video signals. In addition, unlike the traditional adaptive transforms, the proposed method obtains the transform kernels from the reconstructed block, and hence it consumes only one logic indicator for each transform unit. Moreover, a spiral-scanning method is developed to reorder the transform coefficients for better entropy coding. Experimental results on the Key Technical Area (KTA) platform show that the proposed method can achieve a significant bits reduction under both the all-intra and low-delay configurations. / The second part investigates the next-generation video coding. Due to increase of display resolution from High-definition (HD) to Ultra-HD, how to efficiently compress the Ultra-HD signals are essential in the development of future video compression systems. High-resolution video coding benefits from a larger prediction block size and thereof transform and quantization of prediction residues. However, in the current HEVC video coding standard, the maximum coding tree unit (CTU) size is 64x64, which can limit a possible larger prediction block in Ultra-HD video coding, and hence cause negative effects on coding efficiency. Thus, we propose to extend CTU to a super coding unit (SCU) for next-generation video coding, and two separate coding structures are designed to encode a SCU, including Direct-CTU and SCU-to-CTU modes. In Direct-CTU, an SCU is first split into a number of predefined CTUs, and then, the best encoding parameters are searched from the current CTU to the possible minimum coding unit (MCU). Similarly, in SCU-to-CTU, the best encoding parameters are searched from SCU to CTU. In addition, the adaptive loop filter (ALF) and sample adaptive offset (SAO) methods are investigated in SCU based video coding framework. We propose to change the filtering control from SCU level to the coding unit (CU) level, and an improved CU level ALF signaling method is also proposed to further improve the coding efficiency. Furthermore, an adaptive SAO block method is also proposed, and this flexibility of SAO blocks can further improve the performance of the traditional method in the Ultra HD video coding. / In the last part, we explore the bit rate control of video transmission. Rate control serves as an important technique to regulate the bit rate of video transmission over a limited bandwidth and to maximize the overall video quality. Video quality fluctuation plays a key role in human visual perception, and hence many rate control algorithms have been widely developed to maintain a consistent quality for video communication. We propose a novel rate control framework based on the Lagrange multiplier in HEVC. With the assumption of constant quality control, a new relationship between the distortion and the Lagrange multiplier is established. Based on the proposed distortion model and buffer status, we obtain a computationally feasible solution to the problem of minimizing the distortion variation across video frames at the coding tree unit level. Extensive simulation results show that our method outperforms the HEVC rate control by providing a more accurate rate regulation, lower video quality fluctuation and stabler buffer fullness. / 近些年，隨著帶寬費用變得越來越便宜，各種視頻應用比以前更為流行了。然而，人們對于高視頻分辨率，高幀率，或更高比特像素的需求增加了視頻傳輸和存儲帶寬的成本。滿足這樣的需求需要更有效的壓縮技術，因此在過去的幾十年裏，很多國際視頻編碼標准被開發出來，例如MPEG-1/2/4 part2, H264/MPEG-4 part 10 AVC和最新高效視頻編碼標准(HEVC)。本論文的主要目的是研究視頻信號的特點，在H.264和HEVC視頻系統中提供高效的壓縮和傳輸解決方案。論文分三部分，簡要總結如下。 / 第壹部分涉及變換編碼。在現代視頻編碼標准中，變換編碼已被廣泛用于消除預測殘差的空間冗余度。然而，由于在視頻序列中的預測殘差塊有著不同的特性，傳統的變換采用固定變換矩陣可能會導致低的編碼效率。為了解決這個問題，我們提出了壺種新的基于內容自適應變換方案的視頻編碼框架。我們利用重排像素，動態調整的變換矩陣以適應當前的視頻信號。此外，與傳統的自適應變換不同之處在于，我們所提出的方法得到的變換矩陣不需要傳輸到解碼端，而它僅消耗壺個邏輯單元指示當前變換矩陣。此外，我們提出了相應的變換系數掃描方法以達到更有效的熵編碼。在關鍵技術領域(KTA)平台，實驗結果表明本方法可以有效的改善幀內和低延遲的配置下的編碼效率。 / 第二部分探討了新壹代視頻編碼。由于主流顯示分辨率從高清到超高清的變化，如何有效地壓縮超高清視頻信號是未來視頻壓縮技術發展的關鍵。超高分辨率視頻編碼的好處在于可從壹個更大的預測塊對其預測殘差進行變換和量化。然而，在目前HEVC視頻編碼標準，最大編碼榭單元尺寸（CTU）是64x64，其可能限制較大的預測塊，從而影響編碼效率。因此，我們提出了擴展CTU為SCU。其中編碼壹個SCU可能用到兩個獨立的編碼模式，包括Direct-CTU和SCU-to-CTU。在Direct-CTU模式中，SCU被分割成許多預定義的CTUs，然後，最佳的編碼參數搜索範圍為CTU到MCU。同樣，在SCU-to-CTU模式中，最佳的編碼參數搜索範圍是SCU到CTU。此外，自適應環路濾波器（ALF）和自適應采偏移（SAO）在新的SCU編碼框架下進行了研究。我們提出濾波控制從SCU級別更改為CU級別，並提出了新的ALF信號傳送方法進壹步提高傳統的方法在超高清視頻編碼的中性能。 / 在最後壹部分，我們探討了視頻傳輸中的碼率控制。碼率控制作為壹種重要的技術，在有限的帶寬條件下，以最大限度地提高整體的視頻質量。視頻質量波動在人眼視覺感知中起著至關重要的作用，因此許多碼率控制方法得到了廣泛的發展，以追求提供穩定的視頻通信質量。我們提出了壹個新基于HEVC拉格日乘數碼率控制框架。在平穩視頻質量的假設下，我們提出了壹種新的失真和拉格日乘子之間的關係。基于新提出的失真模型和緩沖區的狀態，我們得到壹個計算上可行的解決方案，以最大限度地減少在編碼榭單元級的視頻幀的失真變化。大量的仿真結果表明，我們的方法優于HEVC的碼率控制，它可以提供更精確的碼率調節，降低視頻質量波動，以及維護穩定的緩沖區占有率。 / Wang, Miaohui. / Thesis Ph.D. Chinese University of Hong Kong 2015. / Includes bibliographical references (leaves 158-164). / Abstracts and acknowledgements also in Chinese. / Title from PDF title page (viewed on 11, October, 2016). / Detailed summary in vernacular field only. / Detailed summary in vernacular field only. / Detailed summary in vernacular field only. / Detailed summary in vernacular field only.