In second language (L2) listening assessment, various factors have the potential to impact the validity of listening test items (Brindley & Slatyer, 2002; Buck & Tatsuoka, 1998; Freedle & Kostin, 1999; Nissan, DeVincenzi, & Tang, 1996; Read, 2002; Shohamy & Inbar, 1991). One relatively unexplored area to date is who controls the aural input. In traditional standardized listening tests, an administrator controlled recording is played once or twice. In real world or classroom listening, however, listeners can sometimes request repetition or clarification. Allowing listeners to control the aural input thus has the potential to add test authenticity but requires careful design of the input and expected response as well as an appropriate computer interface. However, if candidates feel less anxious, allowing control of listening input may enhance examinees' experience and still reflect their listening proficiency. Comparing traditional and self paced (i.e., examinees having the opportunity to start, stop, and move the audio position) delivery of multiple choice comprehension items, my research inquiry is whether self paced listening can be a sufficiently reliable and valid measure of examinees' listening ability. Data were gathered from 100 prospective and current university ESL students. They were administered computer based multiple choice listening tests: 10 identical once played items, followed by 33 items in three different conditions: 1) administrator paced input with no audio player visible, 2) self paced with a short time limit, and 3) self paced with a longer time limit. Many facet Rasch (1960/1980) modeling was used to compare the difficulty and discrimination of the items across conditions. Results indicated items on average were similar difficulty overall but discriminated best in self paced conditions. Furthermore, the vast majority of examinees reported they preferred self paced listening. The quantitative results were complemented by follow up stimulated recall interviews with eight participants who took 22 additional test items using screen capture software to explore whether and when they paused and/or repeated the input. Frequency of and reasons for self pacing did not follow any particular pattern by proficiency level. Examinees tended to play more than once but not two full times through, even without limited time. Implications for listening instruction and classroom assessment, as well as standardized testing, are discussed.