Traditional theories of vision assume that object segregation occurs before access to object memories. Yet, behavioral evidence shows that familiar configuration is a prior for segregation, and electrophysiological experiments demonstrate these memories are accessed rapidly. A behavioral index of the speed of access is lacking, however. Here we asked how quickly behavior is influenced by object memories that are accessed in the course of object segregation. We investigated whether access to object memories on the groundside of a border can slow behavior during a rapid categorization task. Participants viewed two silhouettes that depicted a real-world and a novel object. Their task was to saccade toward the real-world object as quickly as possible. Half of the nontarget novel objects were ambiguous in that a portion of a real-world object was suggested, but not consciously perceived, on the groundside of their borders. The rest of the nontargets were unambiguous. We tested whether saccadic reaction times were perturbed by the real-world objects suggested on the groundside of ambiguous novel silhouettes. In Experiments 1 and 2, saccadic reaction times were slowed when nontargets were ambiguous rather than unambiguous. Experiment 2 set an upper limit of 190 ms on the time required for object memories in grounds to influence behavior. Experiment 3 ruled out factors that could have produced longer latencies other than access to object memories. These results provide the first behavioral index of how quickly memories of objects suggested in grounds can influence behavior, placing the upper limit at 190 ms.