Video data has increased dramatically in recent years due to the prevalence of handheld cameras. Such videos, however, are usually shakier compared to videos shot by tripod-mounted cameras or cameras with mechanical stabilizers. In addition, most handheld cameras use CMOS sensors. In a CMOS sensor camera, different rows in a frame are read/reset sequentially from top to bottom. When there is fast relative motion between the scene and the video camera, a frame can be distorted because each row was captured under a different 3D-to-2D projection. This kind of distortion is known as rolling shutter effect. Digital video stabilization and rolling shutter rectification seek to remove the unwanted frame-to-frame jitter and rolling shutter effect, in order to generate visually stable and pleasant videos. In general, we need to (1) estimate the camera motion, (2) regenerate camera motion, and (3) synthesize new frames. This dissertation aims at improving the first two steps of video stabilization and rolling shutter rectification. It has been shown that the inertial sensors in handheld devices can provide more accurate and robust motion estimation compared to vision-based methods. This dissertation proposes an online camera-gyroscope calibration method for sensor fusion while a user is capturing video. The proposed method uses an implicit extended Kalman filter and is based on multiple-view geometry in a rolling shutter camera model. It is able to estimate the needed calibration parameters online with all kinds of camera motion. Given the camera motion estimated from inertial sensors after the pro- posed calibration method, this dissertation first proposes an offline motion smoothing algorithm based on a 3D rotational camera motion model. The offline motion smoothing is formulated as a geodesic-convex regression problem on the manifold of rotation matrix sequences. The formulated problem is solved by an efficient two-metric projection algorithm on the manifold. The geodesic-distance-based smoothness metric better exploits the manifold structure of sequences of rotation matrices. Then this dissertation proposes two online motion smoothing algorithms that are also based on a 3D rotational camera motion model. The first algorithm extends IIR filtering from Euclidean space to the nonlinear manifold of 3D rotation matrices. The second algorithm uses unscented Kalman filtering on a constant angular velocity model. Both offline and online motion smoothing algorithms are constrained to guarantee that no black borders intrude into the stabilized frames. / text

Frame rate is a fundamental element of creating the illusion of movement in video based media. For almost a century film has been produced in agreement with a standard frame rate of 24 frames per second, originally established due to technical limitations. This number lives on for films today, despite many technological innovations and other video based media formats straying from this standard. With contemporary video technology, content cannot only be recorded in higher frame rate; frames can also be artificially interpolated. So called Frame Interpolation technology now comes as a pre-installed feature on most televisions. As a consequence, this has formed a debate on how video based media should be presented, not least when it is artificially generated outside of the creators’ control. This study therefore aims to explore how manipulation of a video clip’s frame rate influences the viewer experience and thereby if the use of Frame Interpolation technology in televisions is justified. A study was conducted wherein participants were shown video clips in their original frame rate and compared them to artificially manipulated copies. The results showed that there is no definitive frame rate that is preferred by all participants and that some participants did not perceive any difference at all. It is also shown that the artificial manipulation of frame rate is generally not appreciated, and that criticisms against its use are misguided in terms of content shown. It is then discussed how television manufacturers should reconsider the use of Frame Interpolation technology. Lastly, we affirm how the results of this study are limited in accuracy by its scope. Further exploration of the subject is suggested to further consider these results found here and the results of earlier papers. / Bildhastigheten i videobaserad media är en fundamental aspekt i hur vi översätter stillbild till rörlig bild. Sedan ett sekel tillbaka produceras film enligt en standard bildhastighet på 24 bilder per sekund, på grund av tekniska begränsningar. Den siffran lever kvar än idag, trots tekniska innovationer samt andra videobaserade medier som töjt på denna standard. Med modern teknik kan media inte bara spelas in i högre bildhastigheter; bilder kan också artificiellt interpoleras. Frame Interpolation-teknik som den kallas kommer numera förinställd på de flesta tv-apparater. Som konsekvens har det förts debatt för och emot högre bildhastigheter, inte minst när de manipuleras av tv-tillverkare utöver skaparnas kontroll. Den här studien vill ta reda på hur manipulering av ett videoklipps bildhastighet påverkar människors känslomässiga tittarupplevelse och därigenom om bruk av Frame Interpolation-teknik i tv-apparater är motiverad. Undersökningen testade deltagarna genom att visa klipp i sin ursprungliga bildhastighet i jämförelse med artificiellt manipulerade kopior. Studien visade att det inte binärt går att bestämma en bildhastighet som deltagarna fann definitivt bäst och att skillnaden inte är uppenbar för alla. Resultatet visar också att artificiell manipulering av bildhastighet inte uppskattas, och att kritiken riktar sig mot fel innehåll. Det diskuteras därför om tv-tillverkare bör överväga användningen av Frame Interpolation-teknik. Slutligen klargörs det varför man ska ställa sig kritisk inför resultaten utifrån studiens begränsningar. Vidare forskning föreslås som kan stödja studiens och liknande studiers slutsatser.