With the rapid development of computer and information technologies, medical imaging has become one of the major sources of information for therapy and research in medicine, biology and other fields. Along with the advancement of medical imaging techniques, computer-aided detection and diagnosis (CAD/CADx) has recently emerged to become one of the major research subjects within the area of diagnostic radiology and medical image analysis. This thesis presents two multi-level learning-based approaches for medical image understanding with applications of CAD/CADx. The so-called "multi-level learning strategy" relies on that supervised and unsupervised statistical learning techniques are utilized to hierarchically model and analyze the medical image content in a "bottom up" way. As the first approach, a learning-based algorithm for automatic medical image classification based on sparse aggregation of learned local appearance cues is proposed. The algorithm starts with a number of landmark detectors to collect local appearance cues throughout the image, which are subsequently verified by a group of learned sparse spatial configuration models. In most cases, a decision could already be made at this stage by simply aggregating the verified detections. For the remaining cases, an additional global appearance filtering step is employed to provide complementary information to make the final decision. This approach is evaluated on a large-scale chest radiograph view identification task and a multi-class radiograph annotation task, demonstrating its improved performance in comparison with other state-of-the-art algorithms. It also achieves high accuracy and robustness against images with severe diseases, imaging artifacts, occlusion, or missing data. As the second approach, a learning-based approach for automatic segmentation of ill-defined and spiculated mammographic masses is presented. The algorithm starts with statistical modeling of exemplar-based image patches. Then, the segmentation problem is regarded as a pixel-wise labeling problem on the produced mass class-conditional probability image, where mass candidates and clutters are extracted. A multi-scale steerable ridge detection algorithm is further employed to detect spiculations. Finally, a graph-cuts technique is employed to unify all outputs from previous steps to generate the final segmentation mask. The proposed method specifically tackles the challenge of inclusion of mass margin and associated extension for segmentation, which is considered to be a very difficult task for many conventional methods. / Master of Science