Téma doktorské práce je zaměřeno na analýzu zpracování signálů v širokopásmových multimediálních sítích a systémech budoucnosti, kde se předpokládají systémy s ultra vysokým rozlišením (UHDTV), vysokým snímkovým kmitočtem (HFR) a stereoskopické systémy (3D). Tyto systémy budou umožňovat vysoce účinnou zdrojovou kompresi obrazu, zvuku a dat a také jejich vysoce účinný přenos, a to jak při volném vysílání (např. DVB-T2), tak ve službách placené televize (např. IPTV). Cílem práce je analýza a vyhodnocení kvality obrazu a služeb v těchto systémech na základě objektivních metrik a subjektivních testů. Práce se dále zaměřuje na analýzu vnímané kvality u stereoskopické televize, kódovací účinnost moderních stereoskopických enkoderů a vlivu sekvencí na uživatelský komfort.

Diploma thesis is focused on real-time 4K video encoding using NVENC technology. First chapter describes the most used video codecs H.264 and HEVC. There is an explanation of the principle of graphic cards and their programmable units. Analysis of the solution of open source Video Codec SDK is also part of the thesis. The main focus of the thesis is an implementation of an application which can handle 4K video encoding from multiple cameras in real time. Performance and qualitative tests were performed for application. Results of these tests were analyzed and discussed.

CU-tree (Coding Unit tree) is an algorithm for adaptive QP (quantization parameter). It runs in the lookahead and decreases the QP of blocks that are heavily referenced by future blocks, taking into account the quality of the prediction and the complexity of the future blocks, approximated by the inter and intra residual. In this study, CU-tree is implemented in c65, an experimental HEVC encoder used internally by Ericsson. The effects of CU-tree are evaluated on the video clips in the HEVC Common test conditions and the performance is compared across c65, x265 and x264. The results are similar across all encoders, with average PSNR (peak signal-to-noise ratio) improvements of 3-10% depending on the fixed QP offsets that are replaced. The runtime is not impaired and improvements to visual quality are expected to be even greater. The algorithm works better at slow speed modes, low bitrates and with source material that is well suited for inter prediction. / CU-tree är en algoritm för adaptiv QP. Den körs under framåtblicken (lookahead) och minskar QP för block som refereras av många framtida block, med hänsyn tagen till prediktionens kvalitet och de framtida blockens komplexitet, approximerat av inter- och intra-skillnaden. I denna studie implementeras CU-tree i c65, en experimentell videokodare som används internt på Ericsson. Effekterna av algoritmen utvärderas på videoklippen i HEVC Common test conditions och prestandan jämförs mellan c65, x265 och x264. Resultaten är liknande i alla videokodare, med genomsnittliga PSNR-förbättringar på 3-10% beroende på vilka fasta QP-offsets som algoritmen ersätter. Körtiden påverkas inte nämnvärt och den subjektiva kvaliteten förbättras troligen ännu mer. Algoritmen fungerar bättre med långsamma hastighetsinställningar, låg bitrate samt videoinnehåll som lämpar sig väl för inter-prediktion.

Digital video is becoming extremely important nowadays and its importance has greatly increased in the last two decades. Due to the rapid development of information and communication technologies, the demand for Ultra-High Definition (UHD) video applications is becoming stronger. However, the most prevalent video compression standard H.264/AVC released in 2003 is inefficient when it comes to UHD videos. The increasing desire for superior compression efficiency to H.264/AVC leads to the standardization of High Efficiency Video Coding (HEVC). Compared with the H.264/AVC standard, HEVC offers a double compression ratio at the same level of video quality or substantial improvement of video quality at the same video bitrate. Yet, HE-VC/H.265 possesses superior compression efficiency, its complexity is several times more than H.264/AVC, impeding its high throughput implementation. Currently, most of the researchers have focused merely on algorithm level adaptations of HEVC/H.265 standard to reduce computational intensity without considering the hardware feasibility. What’s more, the exploration of efficient hardware architecture design is not exhaustive. Only a few research works have been conducted to explore efficient hardware architectures of HEVC/H.265 standard. In this dissertation, we investigate efficient algorithm adaptations and hardware architecture design of HEVC intra encoders. We also explore the deep learning approach in mode prediction. From the algorithm point of view, we propose three efficient hardware-oriented algorithm adaptations, including mode reduction, fast coding unit (CU) cost estimation, and group-based CABAC (context-adaptive binary arithmetic coding) rate estimation. Mode reduction aims to reduce mode candidates of each prediction unit (PU) in the rate-distortion optimization (RDO) process, which is both computation-intensive and time-consuming. Fast CU cost estimation is applied to reduce the complexity in rate-distortion (RD) calculation of each CU. Group-based CABAC rate estimation is proposed to parallelize syntax elements processing to greatly improve rate estimation throughput. From the hardware design perspective, a fully parallel hardware architecture of HEVC intra encoder is developed to sustain UHD video compression at 4K@30fps. The fully parallel architecture introduces four prediction engines (PE) and each PE performs the full cycle of mode prediction, transform, quantization, inverse quantization, inverse transform, reconstruction, rate-distortion estimation independently. PU blocks with different PU sizes will be processed by the different prediction engines (PE) simultaneously. Also, an efficient hardware implementation of a group-based CABAC rate estimator is incorporated into the proposed HEVC intra encoder for accurate and high-throughput rate estimation. To take advantage of the deep learning approach, we also propose a fully connected layer based neural network (FCLNN) mode preselection scheme to reduce the number of RDO modes of luma prediction blocks. All angular prediction modes are classified into 7 prediction groups. Each group contains 3-5 prediction modes that exhibit a similar prediction angle. A rough angle detection algorithm is designed to determine the prediction direction of the current block, then a small scale FCLNN is exploited to refine the mode prediction.

Cette thèse porte sur l’optimisation des techniques de compression d'images fixes et de vidéos en vue de la caractérisation des matériaux pour des applications dans le domaine de la mécanique, et s’inscrit dans le cadre du projet de recherche MEgABIt (MEchAnic Big Images Technology) soutenu par l’Université Polytechnique Hauts-de-France. L’objectif scientifique du projet MEgABIt est d’investiguer dans l’aptitude à compresser de gros volumes de flux de données issues d’instrumentation mécanique de déformations à grands volumes tant spatiaux que fréquentiels. Nous proposons de concevoir des algorithmes originaux de traitement dans l’espace compressé afin de rendre possible au niveau calculatoire l’évaluation des paramètres mécaniques, tout en préservant le maximum d’informations fournis par les systèmes d’acquisitions (imagerie à grande vitesse, tomographie 3D). La compression pertinente de la mesure de déformation des matériaux en haute définition et en grande dynamique doit permettre le calcul optimal de paramètres morpho-mécaniques sans entraîner la perte des caractéristiques essentielles du contenu des images de surface mécaniques, ce qui pourrait conduire à une analyse ou une classification erronée. Dans cette thèse, nous utilisons le standard HEVC (High Efficiency Video Coding) à la pointe des technologies de compression actuelles avant l'analyse, la classification ou le traitement permettant l'évaluation des paramètres mécaniques. Nous avons tout d’abord quantifié l’impact de la compression des séquences vidéos issues d’une caméra ultra-rapide. Les résultats expérimentaux obtenus ont montré que des taux de compression allant jusque 100 :1 pouvaient être appliqués sans dégradation significative de la réponse mécanique de surface du matériau mesurée par l’outil d’analyse VIC-2D. Finalement, nous avons développé une méthode de classification originale dans le domaine compressé d’une base d’images de topographie de surface. Le descripteur d'image topographique est obtenu à partir des modes de prédiction calculés par la prédiction intra-image appliquée lors de la compression sans pertes HEVC des images. La machine à vecteurs de support (SVM) a également été introduite pour renforcer les performances du système proposé. Les résultats expérimentaux montrent que le classificateur dans le domaine compressé est robuste pour la classification de nos six catégories de topographies mécaniques différentes basées sur des méthodologies d'analyse simples ou multi-échelles, pour des taux de compression sans perte obtenus allant jusque 6: 1 en fonction de la complexité de l'image. Nous avons également évalué les effets des types de filtrage de surface (filtres passe-haut, passe-bas et passe-bande) et de l'échelle d'analyse sur l'efficacité du classifieur proposé. La grande échelle des composantes haute fréquence du profil de surface est la mieux appropriée pour classer notre base d’images topographiques avec une précision atteignant 96%. / This PhD. thesis focuses on the optimization of fixed image and video compression techniques for the characterization of materials in mechanical science applications, and it constitutes a part of MEgABIt (MEchAnic Big Images Technology) research project supported by the Polytechnic University Hauts-de-France (UPHF). The scientific objective of the MEgABIt project is to investigate the ability to compress large volumes of data flows from mechanical instrumentation of deformations with large volumes both in the spatial and frequency domain. We propose to design original processing algorithms for data processing in the compressed domain in order to make possible at the computational level the evaluation of the mechanical parameters, while preserving the maximum of information provided by the acquisitions systems (high-speed imaging, tomography 3D). In order to be relevant image compression should allow the optimal computation of morpho-mechanical parameters without causing the loss of the essential characteristics of the contents of the mechanical surface images, which could lead to wrong analysis or classification. In this thesis, we use the state-of-the-art HEVC standard prior to image analysis, classification or storage processing in order to make the evaluation of the mechanical parameters possible at the computational level. We first quantify the impact of compression of video sequences from a high-speed camera. The experimental results obtained show that compression ratios up to 100: 1 could be applied without significant degradation of the mechanical surface response of the material measured by the VIC-2D analysis tool. Then, we develop an original classification method in the compressed domain of a surface topography database. The topographical image descriptor is obtained from the prediction modes calculated by intra-image prediction applied during the lossless HEVC compression of the images. The Support vector machine (SVM) is also introduced for strengthening the performance of the proposed system. Experimental results show that the compressed-domain topographies classifier is robust for classifying the six different mechanical topographies either based on single or multi-scale analyzing methodologies. The achieved lossless compression ratios up to 6:1 depend on image complexity. We evaluate the effects of surface filtering types (high-pass, low-pass, and band-pass filter) and the scale of analysis on the efficiency of the proposed compressed-domain classifier. We verify that the high analysis scale of high-frequency components of the surface profile is more appropriate for classifying our surface topographies with accuracy of 96 %.

Submitted by Fabio Sobreira Campos da Costa (fabio.sobreira@ufpe.br) on 2016-06-21T15:14:55Z No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) Vitor_de_Andrade_Coutinho-dissertacao_ppgee.pdf: 3622975 bytes, checksum: 01a22e0302dfc1890d745c6b1bffe327 (MD5) / Made available in DSpace on 2016-06-21T15:14:56Z (GMT). No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) Vitor_de_Andrade_Coutinho-dissertacao_ppgee.pdf: 3622975 bytes, checksum: 01a22e0302dfc1890d745c6b1bffe327 (MD5) Previous issue date: 2015-02-23 / CNPq / O presente trabalho aborda o desenvolvimento de aproximações para a transformada dis- reta do osseno (DCT) utilizando a abordagem pruning. Devido à propriedade da ompa ta- ção de energia, a DCT é empregada em diversas apli ações de ompressão de dados. Embora algoritmos rápidos permitam omputar a DCT e ientemente, operações de multipli ação são inevitáveis. Devido a res ente demanda por métodos de baixo onsumo energéti o, novos algoritmos de usto omputa ional reduzido são ne essários. Neste ontexto, aproximações para a DCT foram propostas nos últimos anos. Tais aproximações permitem algoritmos livres de multipli ação, sem a ne essidade de operações de ponto utuante, mantendo o desempe- nho de ompressão omparável ao forne ido por métodos baseados na DCT. Uma abordagem adi ional para reduzir o usto omputa ional da DCT é a utilização de pruning. Tal té ni a onsiste em não onsiderar oe ientes dos vetores de entrada e/ou saída que apresentam menor relevân ia em termos de energia on entrada. No aso da DCT, esses oe ientes são os termos de mais alta frequên ia do vetor transformado. A apli ação de pruning a aproxima- ções para a DCT é uma área pou o explorada. O objetivo deste trabalho é apli ar a té ni a a diferentes métodos aproximados para a DCT. As transformações resultantes foram apli adas no ontexto de ompressão de imagem e vídeo e os resultados mostraram desempenho ompa- rável ao de métodos exatos a um usto omputa ional bastante reduzido. Uma generalização do on eito é apresentada, assim omo uma análise da omplexidade aritméti a. / This work introdu es approximate dis rete osine transforms (DCT) based on the pruning approa h. Due to the energy ompa tion property, the DCT is employed in several data ompression appli ations. Although fast algorithms allow an e ient DCT omputation, multipli ation operations are inevitable. Due to the in reasing demand for energy e ient methods, new algorithms with redu ed omputational ost are required. In this ontext, DCT approximations have been proposed re ently. Su h approximations allow multipli ation free algorithms whi h an avoid oating point operations, while maintaining a ompetitive performan e. A further approa h to redu e the omputational ost of the DCT is pruning. The te hnique onsists of dis arding input and/or output ve tors oe ients whi h are regarded as less signi ant. In the ase of the DCT, su h oe ients are output oe ients asso iated to higher frequen y terms. Pruned DCT approximations is a relatively unexplored eld of resear h. The obje tive of this work is the ombination of approximations and pruning to derive extremely low- omplexity DCT approximations. The resulting methods were applied in the image and vídeo ompression s enario and results showed omparative performan e with exa t methods at a mu h lower omputational omplexity. A qualitative and quantitative omparison with a omprehensive list of existing methods is presented. A generalization of the pruning on ept is presented.

Les fondements théoriques pour la saillance visuelle ont été dressés, il y a 35 ans, par Treisman qui a proposé "feature-integration theory" pour le système visuel humain: dans n’importe quel contenu visuel, certaines régions sont saillantes en raison de la différence entre leurs caractéristiques (intensité, couleur, texture, et mouvement) et leur voisinage. Notre thèse offre un cadre méthodologique et expérimental compréhensif pour extraire les régions saillantes directement des flux compressés (MPEG-4 AVC et HEVC), tout en minimisant les opérations de décodage. L’extraction de la saillance visuelle à partir du flux compressé est à priori une contradiction conceptuelle. D’une part, comme suggéré par Treisman, dans un contenu vidéo, la saillance est donnée par des singularités visuelles. D’autre part, afin d’éliminer la redondance visuelle, les flux compressés ne devraient plus préserver des singularités. La thèse souligne également l’avantage pratique de l’extraction de la saillance dans le domaine compressé. Dans ce cas, nous avons démontré que, intégrée dans une application de tatouage robuste de la vidéo compressée, la carte saillance agit comme un outil d’optimisation, ce qui permet d’augmenter la transparence (pour une quantité d’informations insérées et une robustesse contre les attaques prescrites) tout en diminuant la complexité globale du calcul. On peut conclure que la thèse démontre aussi bien méthodologiquement que expérimentalement que même si les normes MPEG-4 AVC et HEVC ne dépendent pas explicitement d’aucun principe de saillance visuelle, leurs flux préservent cette propriété remarquable reliant la représentation numérique de la vidéo au mécanisme psycho-cognitifs humains / The theoretical ground for visual saliency was established some 35 years ago by Treisman who advanced the integration theory for the human visual system: in any visual content, some regions are salient (appealing) because of the discrepancy between their features (intensity, color, texture, motion) and the features of their surrounding areas. This present thesis offers a comprehensive methodological and experimental framework for extracting the salient regions directly from video compressed streams (namely MPEG-4 AVC and HEVC), with minimal decoding operations. Note that saliency extraction from compressed domain is a priori a conceptual contradiction. On the one hand, as suggested by Treisman, saliency is given by visual singularities in the video content. On the other hand, in order to eliminate the visual redundancy, the compressed streams are no longer expected to feature singularities. The thesis also brings to light the practical benefit of the compressed domain saliency extraction. In this respect, the case of robust video watermarking is targeted and it is demonstrated that the saliency acts as an optimization tool, allowing the transparency to be increased (for prescribed quantity of inserted information and robustness against attacks) while decreasing the overall computational complexity. As an overall conclusion, the thesis methodologically and experimentally demonstrates that although the MPEG-4 AVC and the HEVC standards do not explicitly rely on any visual saliency principle, their stream syntax elements preserve this remarkable property linking the digital representation of the video to sophisticated psycho-cognitive mechanisms

Les travaux effectués durant cette thèse de doctorat ont pour but d’augmenter l’efficacité de codage dans 3D-HEVC. Nous proposons des approches conventionnelles orientées vers la normalisation vidéo, ainsi que des approches en rupture basées sur le flot optique. En approches conventionnelles, nous proposons une méthode qui prédit les modes Intra de profondeur avec ceux de texture. L’héritage est conditionné par un critère qui mesure le degré de similitude entre les deux modes. Ensuite, nous proposons deux méthodes pour améliorer la prédiction inter-vue du mouvement dans 3D-HEVC. La première ajoute un vecteur de disparité comme candidat inter-vue dans la liste des candidats du Merge, et la seconde modifie le processus de dérivation de ce vecteur. Finalement, un outil de codage intercomposantes est proposé, où le lien entre les arbres quaternaires de texture et de profondeur est exploité pour réduire le temps d’encodage et le débit, à travers un codage conjoint des deux arbres. Dans la catégorie des approches en rupture, nous proposons deux méthodes basées sur l’estimation de champs denses de vecteurs de mouvement en utilisant le flot optique. La première calcule un champ au niveau d’une vue de base reconstruite, puis l’extrapole au niveau d’une vue dépendante, où il est hérité par les unités de prédiction en tant que candidat dense du Merge. La deuxième méthode améliore la synthèse de vues : quatre champs sont calculés au niveau de deux vues de référence en utilisant deux références temporelles. Ils sont ensuite extrapolés au niveau d’une vue synthétisée et corrigés en utilisant une contrainte épipolaire. Les quatre prédictions correspondantes sont ensuite combinées. / This PhD. thesis deals with improving the coding efficiency in 3D-HEVC. We propose both constrained approaches aimed towards standardization, and also more innovative approaches based on optical flow. In the constrained approaches category, we first propose a method that predicts the depth Intra modes using the ones of the texture. The inheritance is driven by a criterion measuring how much the two are expected to match. Second, we propose two simple ways to improve inter-view motion prediction in 3D-HEVC. The first adds an inter-view disparity vector candidate in the Merge list and the second modifies the derivation process of this disparity vector. Third, an inter-component tool is proposed where the link between the texture and depth quadtree structures is exploited to save both runtime and bits through a joint coding of the quadtrees. In the more innovative approaches category, we propose two methods that are based on a dense motion vector field estimation using optical flow. The first computes such a field on a reconstructed base view. It is then warped at the level of a dependent view where it is inserted as a dense candidate in the Merge list of prediction units in that view. The second method improves the view synthesis process: four fields are computed at the level of the left and right reference views using a past and a future temporal reference. These are then warped at the level of the synthesized view and corrected using an epipolar constraint. The four corresponding predictions are then blended together. Both methods bring significant coding gains which confirm the potential of such innovative solutions.

Intra-prediction is a method for coding standalone frames in video coding. Until now, this has mainly been done using linear formulae. Using an Artificial Neural Network (ANN) may improve the prediction accuracy, leading to improved coding efficiency. In this degree project, Fully Connected Networks (FCN) and Convolutional Neural Networks (CNN) were used for intra-prediction. Experiments were done on samples from different image sizes, block sizes, and block contents, and their effect on the results were compared and discussed. The results show that ANN methods have the potential to perform better or on par with the video coder High Efficiency Video Coding (HEVC) in the intra-prediction task. The proposed ANN designs perform better on smaller block sizes, but different designs could lead to better performance on larger block sizes. It was found that training one network for each HEVC mode and using the most suitable network to predict each block improved performance of the ANN approach. / Intra-prediktion är en metod för kodning av stillbilder i videokodning. Hittills har detta främst gjorts med hjälp av linjära formler. Användning av artificialla neuronnät (ANN) skulle kunna öka prediktionsnoggrannheten och ge högre effektivitet vid kodning. I detta examensarbete användes fully connected networks (FCN) och convolutional neural networks (CNN) för att utföra intra-prediktion. Experiment gjordes på prover från olika bildstorlekar, blockstorlekar och blockinnehåll, och de olika parametrarnas effekt på resultaten jämfördes och diskuterades. Resultaten visar att ANN-metoder har potential att prestera bättre eller lika bra som videokodaren High Efficiency Video Coding (HEVC) för intra-prediktion. De föreslagna ANN-designerna presterar bättre på mindre blockstorlekar, men andra ANN-designs skulle kunna ge bättre prestanda för större blockstorlekar. Det konstaterades att prestandan för ANN-metoderna kunde ökas genom att träna ett nätverk för varje HEVC-mode och använda det mest passande nätverket för varje block.

A popularização das aplicações que manipulam vídeos digitais de altas resoluções incorpora diversos desafios no desenvolvimento de novas e eficientes técnicas para manter a eficiência na compressão de vídeo. Para lidar com esta demanda, o padrão HEVC foi proposto com o objetivo de duplicar as taxas de compressão quando comparado com padrões predecessores. No entanto, para atingir esta meta, o HEVC impõe um elevado custo computacional e, consequentemente, o aumento no consumo de energia. Este cenário torna-se ainda mais preocupante quando considerados dispositivos móveis alimentados por bateria os quais apresentam restrições computacionais no processamento de aplicações multimídia. A maioria dos trabalhos relacionados com este desafio, tipicamente, concentram suas contribuições no redução e controle do esforço computacional refletido no processo de codificação. Entretanto, a literatura indica uma carência de informações com relação ao consumo de energia despendido pelo processamento da codificação de vídeo e, principalmente, o impacto energético da hierarquia de memória cache neste contexto. Esta tese apresenta uma metodologia para caracterização energética da codificação de vídeo HEVC em processador de propósito geral. O principal objetivo da metodologia proposta nesta tese é fornecer dados quantitativos referentes ao consumo de energia do HEVC. Esta metodologia é composta por dois módulos, um deles voltado para o processamento da codificação HEVC e, o outro, direcionado ao comportamento do padrão HEVC no que diz respeito à memória cache. Uma das principais vantagens deste segundo módulo é manter-se independente de aplicação ou de arquitetura de processador. Neste trabalho, diversas análises foram realizadas visando a caracterização do consumo de energia do codificador HEVC em processador de propósito geral, considerando diferentes sequências de vídeo, resoluções e parâmetros do codificador. Além disso, uma análise extensa e detalhada de diferentes configurações possíveis de memória cache foi realizada com o propósito de avaliar o impacto energético destas configurações na codificação. Os resultados obtidos com a caracterização proposta demonstram que o gerenciamento dos parâmetros da codificação de vídeo, de maneira conjunta com as especificações da memória cache, tem um alto potencial para redução do consumo energético de codificação de vídeo, mantendo bons resultados de qualidade visual das sequências codificadas. / The popularization of high-resolution digital video applications brings several challenges on developing new and efficient techniques to maintain the video compression efficiency. To respond to this demand, the HEVC standard was proposed aiming to duplicate the compression rate when compared to its predecessors. However, to achieve such goal, HEVC imposes a high computational cost and, consequently, energy consumption increase. This scenario becomes even more concerned under battery-powered mobile devices which present computational constraints to process multimedia applications. Most of the related works about encoder realization, typically concentrate their contributions on computational effort reduction and management. Therefore, there is a lack of information regarding energy consumption on video encoders, specially about the energy impact of the cache hierarchy in this context. This thesis presents a methodology for energy characterization of the HEVC video encoder in general purpose processors. The main goal of this methodology is to provide quantitative data regarding the HEVC energy consumption. This methodology is composed of two modules, one focuses on the HEVC processing and the other focuses on the HEVC behavior regarding cache memory-related consumption. One of the main advantages of this second module is to remain independent of application or processor architecture. Several analyzes are performed aiming at the energetic characterization of HEVC coding considering different video sequences, resolutions, and parameters. In addition, an extensive and detailed analysis of different cache configurations is performed in order to evaluate the energy impact of such configurations during the video coding execution. The results obtained with the proposed characterization demonstrate that the management of the video coding parameters in conjunction with the cache specifications has a high potential for reducing the energy consumption of video coding whereas maintaining good coding efficiency results.