Video frame rate up-conversion is an important issue for multimedia systems in achieving better video quality and motion portrayal. Motion-compensated methods offer better quality interpolated frames since the interpolation is performed along the motion trajectory. In addition, computational complexity, regularity, and memory bandwidth are important for a real-time implementation. Motion-compensated frame rate up-conversion (MC-FRC) is composed of two main parts: motion estimation (ME) and motion-compensated frame interpolation (MCFI). Since ME is an essential part of MC-FRC, a new fast motion estimation (FME) algorithm capable of producing sub-sample motion vectors at low computational-complexity has been developed. Unlike existing FME algorithms, the developed algorithm considers the low complexity sub-sample accuracy in designing the search pattern for FME. The developed FME algorithm is designed in such a way that the block distortion measure (BDM) is modeled as a parametric surface in the vicinity of the integer-sample motion vector; this modeling enables low computational-complexity sub-sample motion estimation without pixel interpolation. MC-FRC needs more accurate motion trajectories for better video quality; hence, a novel true-motion estimation (TME) algorithm targeting to track the projected object motion has been developed for video processing applications, such as motion-compensated frame interpolation (MCFI), deinterlacing, and denoising. Developed TME algorithm considers not only the computational complexity and regularity but also memory bandwidth. TME is obtained by imposing implicit and explicit smoothness constraints on block matching algorithm (BMA). In addition, it employs a novel adaptive clustering algorithm to keep the low-complexity at reasonable levels yet enable exploiting more spatiotemporal neighbors. To produce better quality interpolated frames, dense motion field at the interpolation instants are obtained for both forward and backward motion vectors (MVs); then, bidirectional motion compensation using forward and backward MVs is applied by mixing both elegantly.

Video streaming is an important field of global communications and data processing. It is divided into server and client sides connected via network. Video streaming is concerned with delivering video data from server to client over the network as fast and with as little loss as possible. In this study the possibilities to minimize the amount of data transferred over the network in video streaming are investigated and a video streaming technique comprised of server and client sides is proposed. To expand the flexibility and adaptability of the proposed video streaming technique an operational parameter system was constructed and the parameter value ranges were defined. The proposed video streaming technique was then applied to three sample videos. Before streaming the server side of the proposed technique reduced the frame count of input videos based on operational parameter values while the client side reconstructed the skipped frames. Then the quality of the resulting videos was measured and evaluated. To evaluate the reconstructed frames and videos the PSNR measurement method was used. The study concludes that by using the proposed video streaming technique it is possible to reduce the amount of transfer data by dropping frames on the server side and reconstructing them on the client side.

The ability to understand videos at the level of pixels plays a key role in a wide range of computer vision applications. For example, a robot or autonomous vehicle relies on classifying each pixel in the video stream into semantic categories to holistically understand the surrounding environment, and video editing software needs to exploit the spatiotemporal context of video pixels to generate various visual effects. Despite the great progress of Deep Learning (DL) techniques, applying DL-based vision models to process video pixels remains practically challenging, due to the high volume of video data and the compute-intensive design of DL approaches. In this thesis, we aim to design efficient and robust deep models for pixel-level video understanding of high-level semantics, mid-level grouping, and low-level interpolation. Toward this goal, in Part I, we address the semantic analysis of video pixels with the task of Video Semantic Segmentation (VSS), which aims to assign pixel-level semantic labels to video frames. We introduce methods that utilize temporal redundancy and context to efficiently recognize video pixels without sacrificing performance. Extensive experiments on various datasets demonstrate our methods' effectiveness and efficiency on both common GPUs and edge devices. Then, in Part II, we show that pixel-level motion patterns help to differentiate video objects from their background. In particular, we propose a fast and efficient contour-based algorithm to group and separate motion patterns for video objects. Furthermore, we present learning-based models to solve the tracking of objects across frames. We show that by explicitly separating the object segmentation and object tracking problems, our framework achieves efficiency during both training and inference. Finally, in Part III, we study the temporal interpolation of pixels given their spatial-temporal context. We show that intermediate video frames can be inferred via interpolation in a very efficient way, by introducing the many-to-many splatting framework that can quickly warp and fuse pixels at any number of arbitrary intermediate time steps. We also propose a dynamic refinement mechanism to further improve the interpolation quality by reducing redundant computation. Evaluation on various types of datasets shows that our method can interpolate videos with state-of-the-art quality and efficiency. To summarize, we discuss and propose efficient pipelines for pixel-level video understanding tasks across high-level semantics, mid-level grouping, and low-level interpolation. The proposed models can contribute to tackling a wide range of real-world video perception and understanding problems in future research.

Frame rate is a fundamental element of creating the illusion of movement in video based media. For almost a century film has been produced in agreement with a standard frame rate of 24 frames per second, originally established due to technical limitations. This number lives on for films today, despite many technological innovations and other video based media formats straying from this standard. With contemporary video technology, content cannot only be recorded in higher frame rate; frames can also be artificially interpolated. So called Frame Interpolation technology now comes as a pre-installed feature on most televisions. As a consequence, this has formed a debate on how video based media should be presented, not least when it is artificially generated outside of the creators’ control. This study therefore aims to explore how manipulation of a video clip’s frame rate influences the viewer experience and thereby if the use of Frame Interpolation technology in televisions is justified. A study was conducted wherein participants were shown video clips in their original frame rate and compared them to artificially manipulated copies. The results showed that there is no definitive frame rate that is preferred by all participants and that some participants did not perceive any difference at all. It is also shown that the artificial manipulation of frame rate is generally not appreciated, and that criticisms against its use are misguided in terms of content shown. It is then discussed how television manufacturers should reconsider the use of Frame Interpolation technology. Lastly, we affirm how the results of this study are limited in accuracy by its scope. Further exploration of the subject is suggested to further consider these results found here and the results of earlier papers. / Bildhastigheten i videobaserad media är en fundamental aspekt i hur vi översätter stillbild till rörlig bild. Sedan ett sekel tillbaka produceras film enligt en standard bildhastighet på 24 bilder per sekund, på grund av tekniska begränsningar. Den siffran lever kvar än idag, trots tekniska innovationer samt andra videobaserade medier som töjt på denna standard. Med modern teknik kan media inte bara spelas in i högre bildhastigheter; bilder kan också artificiellt interpoleras. Frame Interpolation-teknik som den kallas kommer numera förinställd på de flesta tv-apparater. Som konsekvens har det förts debatt för och emot högre bildhastigheter, inte minst när de manipuleras av tv-tillverkare utöver skaparnas kontroll. Den här studien vill ta reda på hur manipulering av ett videoklipps bildhastighet påverkar människors känslomässiga tittarupplevelse och därigenom om bruk av Frame Interpolation-teknik i tv-apparater är motiverad. Undersökningen testade deltagarna genom att visa klipp i sin ursprungliga bildhastighet i jämförelse med artificiellt manipulerade kopior. Studien visade att det inte binärt går att bestämma en bildhastighet som deltagarna fann definitivt bäst och att skillnaden inte är uppenbar för alla. Resultatet visar också att artificiell manipulering av bildhastighet inte uppskattas, och att kritiken riktar sig mot fel innehåll. Det diskuteras därför om tv-tillverkare bör överväga användningen av Frame Interpolation-teknik. Slutligen klargörs det varför man ska ställa sig kritisk inför resultaten utifrån studiens begränsningar. Vidare forskning föreslås som kan stödja studiens och liknande studiers slutsatser.