Optical flow is widely used in image and video processing. This paper describes the definition of optical flow and some basic methods of estimating optical flow. There are three main optical flow methods described in this paper: Horn–Schunck method, Lucas–Kanade method and Anandan method. It will select the most appropriate method to measure the flow in videos containing smoke and gas by comparing those three methods with different criteria. The results are as below 1) Horn–Schunck method is good at measuring flow in infrared video and in horizontal direction, when 2) Anandan method is adept in estimating upward optical flow but not suitable for infrared video. 3) Lucas-Kanade method can be used not only in smoke videos, but also in methane gas videos, and it can detect flow in both horizontal and vertical direction.

From the last decade, studies on human facial emotion recognition revealed that computing models based on regression modelling can produce applicable performance. In this study, an automatic facial expression real-time system was built and tested. The method is used in this study has been used widely in different areas such as Local Binary Pattern method, which has been used in many research projects in machine vision, and the K-Nearest Neighbour algorithm is method utilized for regression modelling. In this study, these two techniques has been used and implemented on the FPGA for the first time, on the side and joined together to great the model in such way to display a continues and automatic emotional state detection model on the monitor. To evaluate the effectiveness of the classifier technique for human emotion recognition from video, the model was designed and tested on MATLAB environment and then MATLAB Simulink environment that is capable of recognizing continuous facial expression in real time with a rate of 1 frame per second and implemented on a desktop PC. It has been evaluated in a testing dataset and the experimental results were promising with the accuracy of 51.28%. The datasets and labels used in this study are made from videos which, recorded twice from 5 participants while watching a video. In order to implement it in real-time in faster frame rate, the facial expression recognition system was built on FPGA. The model was built on Atlys™ Spartan-6 FPGA Development Board. It can perform continuously emotional state recognition in real time at a frame rate of 30 with the accuracy of 47.44%. A graphic user interface was designed to display the participant video in real time and also two dimensional predict labels of the emotion at the same time. This is the first time that automatic emotional state detection has been successfully implemented on FPGA by using LBP and K-NN techniques in such way to display a continues and automatic emotional state detection model on the monitor.

In high speed applications of visual servoing, latency from the recognition algorithm can cause significant degradation of in response time. Hardware acceleration allows for recognition algorithms to be applied directly during the raster scan from the image sensor, thereby removing virtually all video processing latency. This paper examines one such method, along with an analysis of design decisions made to optimize for use during high speed airborne object tracking tests for the US military. Designing test equipment for defense use involves working around unique challenges that arise from having many details being deemed classified or highly sensitive information. Designing tracking system without knowing any exact numbers for speeds, mass, distance or nature of the objects being tracked requires a flexible control system that can be easily tuned after installation. To further improve accuracy and allow rapid tuning to a yet undisclosed set of parameters, a machine learning powered auto-tuner is developed and implemented as a control loop optimizer.

In the modern society, mobile is gradually going to become all about video streaming. The main reasons of video growth are mobile devices such as smartphones and tablets which enable people to have access to videos they would like to watch at anywhere and anytime. However, due to the large video data size and intensive computation, video processing leads to a huge power consumption. Mobile system designers typically focus on hardware-level power optimization techniques without considering how hardware performance interfaces with viewer experience. In my research, I investigated how viewing context factors affect mobile viewing experience. Furthermore, a viewer-aware intelligent mobile video system was designed to optimize power efficiency automatically in real-time according to the viewing context and maintain the same viewing experience. Our research opened a door for developments of future viewer-aware mobile system design, accelerating low-cost mobile devices with longer battery life.

The purpose of this diploma thesis is to analyze possibilities of non-photorealistic rendering focused on area of image transformation and afterwards design algorithm for video postprocessing to it's cartoon representation. The thesis describes problem analysis, design of the algorithm and it's implementation.

For applications such as live video processing, there is a high demand for high performance and low latency solutions. The configurable logic in FPGAs allows for custom hardware to be tailored to a specific video application. These FPGA designs require technical expertise and lengthy implementation times by vendor tools for each unique solution. This thesis presents a dynamically configurable topology as an FPGA overlay to deploy custom hardware processing pipelines during run-time by utilizing dynamic partial reconfiguration. Within the FPGA overlay, a configurable topology with a routable switch allows video streams to be copied and mixed to create complex data paths. This work demonstrates a dynamic video processing pipeline with 11 reconfigurable regions and 16 unique processing cores, allowing for billions of custom run-time configurations.

This paper deals with image and video processing on Symbian OS smartphones. Description of required development tools is given, and pros and cons of existing image processing applications are discussed. Afterwards, a new application, fast image viewer and editor, is designed keeping disadvantages of similar applications in mind. Purpose of this work is to make simple and fast tool for easy manipulation with integrated camera and captured images. Results and future development directions are at the end.

Lecture videos are good sources for people to learn new things. Students commonly use online videos to explore various domains. However, some recorded videos are posted on online platforms without being post-processed due to technology and resource limitations. In this work, we focus on the research of developing an intelligent system to automatically extract essential information, including the main instructor and screen, in a lecture video in several scenarios by using modern deep learning techniques. This thesis aims to combine the extracted essential information to render the videos and generate a new layout with a smaller file size than the original one. Another benefit of using this approach is that the users may save video post-processing time and costs. State-of-the-art object detection models, an algorithm to correct screen display, tracking the instructor, and other deep learning techniques were adopted in the system to detect both the main instructor and the screen in given videos without much of the computational burden. There are four main contributions: 1. We built an intelligent video analysis and post-processing system to extract and reframe detected objects from lecture videos. 2. We proposed a post-processing algorithm to localize the frontal human torso position in processing a sequence of frames in the videos. 3. We proposed a novel deep learning approach to distinguish the main instructor from other instructors or audiences in several complex situations. 4. We proposed an algorithm to extract the four edge points of a screen at the pixel level and correct the screen display in various scenarios.

The processing of digital video has long been a significant computational task for modern x86 processors. With every video frame composed of one to three planes, each consisting of a two-dimensional array of pixel data, and a video clip comprising of thousands of such frames, the sheer volume of data is significant. With the introduction of new high definition video formats such as 4K or stereoscopic 3D, the volume of uncompressed frame data is growing ever larger. Modern CPUs offer performance enhancements for processing digital video through SIMD instructions such as SSE2 or AVX. However, even with these instruction sets, CPUs are limited by their inherently sequential design, and can only operate on a handful of bytes in parallel. Even processors with a multitude of cores only execute on an elementary level of parallelism. GPUs provide an alternative, massively parallel architecture. GPUs differ from CPUs by providing thousands of throughput-oriented cores, instead of a maximum of tens of generalized “good enough at everything” x86 cores. The GPU’s throughput-oriented cores are far more adept at handling large arrays of pixel data, as many video filtering operations can be performed independently. This computational independence allows for pixel processing to scale across hun- dreds or even thousands of device cores. This thesis explores the utilization of GPUs for video processing, and evaluates the advantages and caveats of porting the modern video filtering framework, Vapoursynth, over to running entirely on the GPU. Compute heavy GPU-enabled video processing results in up to a 108% speedup over an SSE2-optimized, multithreaded CPU implementation.

Machine-driven analysis of visual data is the hard core of intelligent surveillance systems. Its main goal is to recognize di erent objects in the video sequence and their behaviour. Such operation is very challenging due to the dynamic nature of the scene and the lack of semantic-comprehension for visual data in machines. The general ow of the recognition process starts with the object extraction task. For so long, this task has been performed using image segmentation. However, recent years have seen the emergence of another contender, image matting. As a well-known process, matting has a very rich literature, most of which is designated to interactive approaches for applications like movie editing. Thus, it was conventionally not considered for visual data analysis operations. Following the new shift toward matting as a means to object extraction, two methods have stood out for their foreground-extraction accuracy and, more importantly, their automation potential. These methods are Closed-Form Matting (CFM) and Spectral Matting (SM). They pose the matting process as either a constrained optimization problem or a segmentation-like component selection process. This di erence of formulation stems from an interesting di erence of perspective on the matting process, opening the door for more automation possibilities. Consequently, both of these methods have been the subject of some automation attempts that produced some intriguing results. For their importance and potential, this thesis will provide detailed discussion and analysis on two of the most successful techniques proposed to automate the CFM and SM methods. In the beginning, focus will be on introducing the theoretical grounds of both matting methods as well as the automatic techniques. Then, it will be shifted toward a full analysis and assessment of the performance and implementation of these automation attempts. To conclude the thesis, a brief discussion on possible improvements will be presented, within which a hybrid technique is proposed to combine the best features of the reviewed two techniques. / Thesis / Master of Applied Science (MASc)