Quincunx filter banks are two-dimensional, two-channel, nonseparable filter banks. They are widely used in many signal processing applications. In this thesis, we study the design and applications of quincunx filter banks in the processing of two-dimensional digital signals. Symmetric extension algorithms for quincunx filter banks are proposed. In the one-dimensional case,symmetric extension is a commonly used technique to build nonexpansive transforms of finite-length sequences. We show how this technique can be extended to the nonseparable quincunx case. We consider three types of quadrantally-symmetric linear-phase quincunx filter banks, and for each of these types we show how nonexpansive transforms of two-dimensional sequences defined on arbitrary rectangular regions can be constructed. New optimization-based techniques are proposed for the design of high-performance quincunx filter banks for the application of image coding. The new methods yield linear-phase perfect-reconstruction systems with high coding gain, good analysis/synthesis filter frequency responses, and certain prescribed vanishing moment properties. We present examples of filter banks designed with these techniques and demonstrate their efficiency for image coding relative to existing filter banks. The best filter banks in our design examples outperformother previously proposed quincunx filter banks in approximately 80% cases and sometimes even outperform the well-known 9/7 filter bank from the JPEG-2000 standard.

Image coding is studied, with the work consisting of two distinct parts. Each part focuses on different coding paradigm. The first part of the research examines subband coding of images. An optimization-based method for the design of high-performance separable filter banks for image coding is proposed. This method yields linear-phase perfect-reconstruction systems with high coding gain, good frequency selectivity, and certain prescribed vanishing-moment properties. Several filter banks designed with the proposed method are presented and shown to work extremely well for image coding, outperforming the well-known 9/7 filter bank (from the JPEG-2000 standard) in most cases. Several families of perfect reconstruction filter banks exist, where the filter banks in each family have some common structural properties. New filter banks in each family are designed with the proposed method. Experimental results show that these new filter banks outperform previously known filter banks from the same family. The second part of the research explores normal meshes as a tool for image coding, with a particular interest in the normal-mesh-based image coder of Jansen, Baraniuk, and Lavu. Three modifications to this coder are proposed, namely, the use of a data-dependent base mesh, an alternative representation for normal/vertical offsets, and a different scan-conversion scheme based on bicubic interpolation. Experimental results show that our proposed changes lead to improved coding performance in terms of both objective and subjective image quality measures.