The visual processing of a stimulus is facilitated by attention when it is at an attended location compared to an unattended location. However, whether attentional selection operates on the basis of visual features (e.g., color) independently of spatial locations is less clear. Six experiments were designed to examine how color information as well as location information affected attentional selection. In Experiment 1, the color of the targets and the spatial distance between them were both manipulated. Stimuli were found to be grouped based on color similarity. Additionally, the evidence suggested direct selection on the basis of color groups, rather than selection that was mediated by location. By varying the probabilities of target location and color, Experiments 2, 3 and 4 demonstrated that the use of color in perceptual grouping and in biasing the priority of selection is not automatic, but is modulated by task demands. Experiments 5 and 6 further investigated the relationship between using color and using location as the selection basis under exogenous and endogenous orienting. The results suggest that the precise nature of the interaction between color and location varies according to the mode of attentional control. Collectively, these experiments contribute to an understanding of how different types of information are used in selection and suggest a greater degree of flexibility of attentional control than previously expected. The flexibility is likely to be determined by a number of factors, including task demands and the nature of attentional control.