Studies assessing the evolution of plant traits frequently focus on pollinators as the primary drivers of floral trait evolution. However, herbivores can also play a role, and, under some circumstances, may even impose stronger selection on floral display than pollinators. This is especially true when the traits under selection are linked to anti-herbivore defense. Here I describe a study in which I quantified the relative role of herbivores and pollinators in selection for floral traits in the North American invasive plant, purple loosestrife (Lythrum salicaria). Because L. salicaria responds to leaf-chewing herbivores by producing compensatory tissue growth, and this in turn alters the architecture of the floral display, I further tested whether herbivores can indirectly modify pollinator-mediated selection through this pathway. Using a split-plot design, I measured pollen limitation and reproductive output in experimentally manipulated plants in the presence and absence of simulated herbivore damage in order to quantify the effects of damage and pollination on natural selection for floral display. My results showed that damage significantly increased direct selection (beta-i) for earlier flowering time and decreased selection on the number of inflorescences, even more than pollinators did. Because damage did not modify pollinator-mediated selection for floral display traits, the selection imposed by herbivores is likely only having direct effects. My findings demonstrate the importance of considering multiple agents of selection and their potential interactions when quantifying natural selection in a study system. In particular, it is important to consider that the agent most frequently studied may not be imposing the brunt of selection.