The science of behavior analysis relies heavily on direct observation. Human observers are typically used to measure behavior in applied settings. Although the use of human observers is beneficial in many regards, it also presents challenges. Of primary concern is the extent to which the data generated by observers actually corresponds to the behavioral events of interest, and the implications this may have in terms of replication. This study assessed the effects that labels, definitions, and examples and non-examples of two different modalities had on observer accuracy, consistency, and agreement. Results showed that current practices in observer training may require refinement to ensure high observer accuracy, consistency, and agreement. Suggestions for how to improve the desired stimulus control of observers are provided.