One of the most active research areas in computer vision is image classification. Although there have been many research efforts in this area, it remains a difficult problem, especially when the number of categories is large. Most of the previous work in image classification uses low-level image features. We believe low-level features ignore a lot of the semantic structures of the image classes. In this thesis, we go beyond simple low-level features and propose new approaches for constructing mid-level visual features for image classification. We represent an image using the outputs of a collection of binary classifiers. These binary classifiers are trained to differentiate pairs of object classes in an object hierarchy. Our feature representations implicitly capture the hierarchical structure in object classes. We show that our proposed approach outperforms other baseline methods in image classification.