BACKGROUND: The success of flipped classrooms is dependent upon students' preparation prior to class, the lack of which is the most common challenge associated with this teaching methodology. To mitigate this limitation, it is important to develop and assess methods of engaging learners during pre-class activities. OBJECTIVE: To determine if quiz delivery method (embedded throughout versus after pre-class videos) affects students' knowledge retention, grades, and video viewing behavior. METHODS: Participating students were randomized to take quizzes, either during pre-class videos via Panopto (EQV) or after pre-class videos in the traditional manner via Moodle (TMM). Outcomes assessed included students' knowledge retention, scores on pre- and post-class quizzes, and pre-class video viewing behavior (total views and minutes viewed per student) during a three-week period. Having experienced both quizzing modalities during the semester, the perceptions from students in the EQV group were surveyed. RESULTS: Baseline assessment results of both groups (n = 27 per group) were comparable with a median score of 33% (IQR: 17, 50) in both groups. Performance was also similar on knowledge retention [TMM: 67% (50, 83) vs. EQV: 83% (50, 83)], pre-class quiz scores [TMM: 90% (87, 97) vs. EQV: 93% (90, 95)], and post-class quiz scores [TMM: 93% (80, 100) vs. EQV: 87% (80, 100)], while students in the EQV group had more total views [10 (8, 12)] vs. [5 (2, 11)] and minutes viewed [71 (36, 108) vs. 35 (15, 81)]. Results from the perception survey administered to students in the EQV group (74.1% response rate) indicated a preference for embedded quizzes overall (58%) and for class preparation (75%) when compared with post-video quizzes. CONCLUSION: Students' knowledge retention and performance were similar in both EQV and TMM groups, though students in the EQV group were more engaged with videos and most of them preferred this quiz delivery. Using embedded quizzes for formative rather than summative assessment might be an appropriate mechanism to encourage students' viewing of pre-class videos and their preparation for flipped classes.