The field of Computer Vision continues to be revolutionized by advances in Convolutional Neural Networks. These networks are well suited for the regular grid structure of image data. However, there are many irregular image types that do not fit within such a framework, such as multi-view images, spherical images, superpixel representations, and texture maps for 3D meshes. These kinds of representations usually have specially designed networks that only operate and train on that unique form of data, thus requiring large datasets for each data domain. This dissertation aims to bridge the gap between standard convolutional networks and specialized ones. It proposes selection-based convolution. This technique operates on graph representations, giving it the flexibility to represent many irregular image domains, but maintains the spatially-oriented nature of an image convolution. Thus, it is possible to train a network on standard images, then use those same network weights for any kind graph-based representation. The effectiveness of this technique is evaluated on image types such as spherical images and 3D meshes for tasks such as segmentation and style transfer. Improvements to the selection mechanism through various forms of interpolation are also presented. Finally, this work demonstrates the generality of selection and its ability to be applied to various forms of graph networks and graph data, not just those specific to the image domain.