Given the strong association between aberrant nuclear morphology and tumor progression, changes in nuclear structure have remained the gold standard for cancer diagnosis for over 150 years. Recently, the rapid development of imaging hardware and computation power creates the opportunity for automated computer-aided diagnosis (CAD). Developing a robust and reliable pattern recognition pipeline is a pressing need to mine and analyze tons of nuclei data being captured. Among the rich studies on pattern recognition problems in pathology, automated nuclei detection, segmentation and cancer detection are the recurring tasks due to the importance and challenges of nuclei analysis. In this thesis, we propose and investigate the state-of-art methods in the CAD modules for maximizing the overall amount of information from images for decision making. We focus on nuclei segmentation and patient cancer detection in the nuclei image analysis pipeline. As the first step in nuclei analysis, we develop an unsupervised nuclei detection and segmentation approach for pathology images. Different from many supervised segmentation methods whose performances rely on the quality and quantity of training samples, the proposed method is able to automatically search for the nucleus contour by solving the shortest path problem with little user effort. We consider the cancer detection task as a set classification problem and propose a highly discriminative predictive model in the sense that it not only optimizes the classifier decision boundary but also transfers discriminative information to set representation learning. The innovation of the model is the integration of set representation learning and classifier training into one objective function for boosting the cancer detection performance. Experimental results showed that the new model provides significant improvements compared with state-of-art methods in the diagnostic challenges. In addition, we showed that the predictive model enables visual interpretation of discriminative nuclear characteristics representing the whole nuclei set. We believe the proposed model is quite general and provide experimental validations in several extended pattern recognition problems.