Les vidéos omnidirectionnelles, également appelées vidéos sphériques ou vidéos360°, sont des vidéos avec des pixels enregistrés dans toutes les directions de l’espace. Un utilisateur qui regarde un tel contenu avec un Casques de Réalité Virtuelle (CRV) peut sélectionner la partie de la vidéo à afficher, usuellement nommée viewport, en bougeant la tête. Pour se sentir totalement immergé à l’intérieur du contenu, l’utilisateur a besoin de voir au moins 90 viewports par seconde en 4K. Avec les technologies de streaming traditionnelles, fournir une telle qualité nécessiterait un débit de plus de100 Mbit s−1, ce qui est bien trop élevé. Dans cette thèse, je présente mes contributions pour rendre possible le streaming de vidéos omnidirectionnelles hautement immersives sur l’Internet. On peut distinguer six contributions : une proposition d’architecture de streaming viewport adaptatif réutilisant une partie des technologies existantes ; une extension de cette architecture pour des vidéos à six degrés de liberté ; deux études théoriques des vidéos à qualité spatiale non-homogène; un logiciel open source de manipulation des vidéos 360°; et un jeu d’enregistrements de déplacements d’utilisateurs regardant des vidéos 360°. / Omnidirectional videos, also denoted as spherical videos or 360° videos, are videos with pixels recorded from a given viewpoint in every direction of space. A user watching such an omnidirectional content with a Head Mounted Display (HMD) can select the portion of the videoto display, usually denoted as viewport, by moving her head. To feel high immersion inside the content a user needs to see viewport with 4K resolutionand 90 Hz frame rate. With traditional streaming technologies, providing such quality would require a data rate of more than 100 Mbit s−1, which is far too high compared to the median Internet access band width. In this dissertation, I present my contributions to enable the streaming of highly immersive omnidirectional videos on the Internet. We can distinguish six contributions : a viewport-adaptive streaming architecture proposal reusing a part of existing technologies ; an extension of this architecture for videos with six degrees of freedom ; two theoretical studies of videos with non homogeneous spatial quality ; an open-source software for handling 360° videos ; and a dataset of recorded users’ trajectories while watching 360° videos.

La migration vers la résolution HD de la plupart des applications multimédias visuelles a nécessité la création de nouveaux standards de compression vidéo tels que le H264/AVC (Advanced Video Coding) et le HEVC (High Efficiency Video Coding). Ces standards sont caractérisés par des hautes performances de codage en termes de taux de compression et qualité vidéo par rapport aux normes précédentes. Cependant, ces performances entraînent de grandes complexités de calcul ce qui rend difficile d'assurer un encodage en temps réel pour la résolution HD sur des processeurs monocœurs programmables qui sont les plus répandus. De plus, comme actuellement les systèmes embarqués sont de plus en plus utilisés dans diverses applications multimédias, concevoir une solution logicielle embarquée pour l'encodeur H264/AVC constitue ainsi un défit très difficile puisqu'il faut répondre aux exigences de l'embarqué au niveau des ressources matérielles comme la mémoire et de la consommation d'énergie. Les récents systèmes embarqués dotés de la technologie multicœur représentent une solution attractive pour surmonter ces problèmes. Dans ce contexte, cette thèse s'intéresse à exploiter la performance de la nouvelle génération de DSP multicœurs de Texas Instruments pour concevoir un encodeur H264/AVC embarqué de résolution HD fonctionnant en temps réel. Nous visons une solution logicielle, caractérisée par une forte flexibilité, par rapport aux IPs existants, qui permet de tout paramétrer (qualité, débit etc). Cette flexibilité logicielle permet aussi l'évolutivité de système en suivant les améliorations de codage comme la migration vers la nouvelle norme HEVC, partiellement abordée dans cette thèse. Nous présentons ainsi les diverses optimisations appliquées que ce soient algorithmiques, architecturales et structurelles afin d'améliorer la vitesse d'encodage sur un seul cœur DSP avant de passer à une implémentation multicœur. Ensuite, nous proposons des implémentations parallèles de l'encodeur H264/AVC sur différentes unités de calcul en exploitant le parallélisme potentiel au sein de la chaîne d'encodage afin de satisfaire la contrainte de temps réel tout en assurant une bonne performance de codage en termes de qualité vidéo et débit binaire. Nous étudions également le problème d'allocation des ressources (ressources de calcul, ressources mémoire, ressources de communication) avec de fortes contraintes temporelles d'exécution. Finalement, cette thèse ouvre la voie vers l'implémentation de la nouvelle norme de codage vidéo HEVC sur deux systèmes embarqués monocœurs dans le but de préparer une solution logicielle embarquée pour les futurs travaux de recherche / The trend toward HD resolution in most of visual multimedia applications has involved the emergence of a large number of video compression standards such as H.264/AVC (Advanced Video Coding) and HEVC (High Efficiency Video Coding). These standards are characterized by high coding performances in terms of compression ratio and video quality compared to previous standards. However, these performances come with large computational complexities which make it difficult to meet real-time encoding for HD resolution on the most common single-core programmable processors. Moreover, as embedded systems have become increasingly used in various multimedia applications, designing an embedded software solution for the H264/AVC encoder represents another difficult challenge since we have to meet the embedded requirements in terms of hardware resources such as memory and power consumption. The new embedded systems with multicore technology represent an attractive solution to overcome these problems. In this context, this thesis is interested in exploiting the performance of the new generation of Texas Instruments multicore DSPs to design an embedded real-time H264/AVC high definition video encoder. We aim a software solution, characterized by high flexibility that allows setting all parameters (quality, bitrate etc) compared to existing IPs. This software flexibility allows also the system scalability by following the coding enhancements as the migration to the newest HEVC standard. Thus, we present the algorithmic, architectural, and structural optimizations which are applied to improve the encoding speed on a single DSP core before moving to a multicore implementation. Then, we propose parallel implementations of the H264/AVC encoder exploiting the multicore architecture of our platform and the potential parallelism in the encoding chain in order to meet real-time constraints while ensuring a good performance in terms of bitrate and video quality. We also explore the problem of resources allocation (computing resources, storage resources, communication resources) with hard execution time constraints. Finally, this thesis opens the way towards the implementation of the new HEVC video coding standard on two embedded systems in order to prepare a software solution for future research

Cette thèse de doctorat étudie le nouveau concept de décodeur intelligent (SDec) dans lequel le décodeur est doté de la possibilité de simuler l’encodeur et est capable de mener la compétition R-D de la même manière qu’au niveau de l’encodeur. Cette technique vise à réduire la signalisation des modes et des paramètres de codage en compétition. Le schéma général de codage SDec ainsi que plusieurs applications pratiques sont proposées, suivis d’une approche en amont qui exploite l’apprentissage automatique pour le codage vidéo. Le schéma de codage SDec exploite un décodeur complexe capable de reproduire le choix de l’encodeur calculé sur des blocs de référence causaux, éliminant ainsi la nécessité de signaler les modes de codage et les paramètres associés. Plusieurs applications pratiques du schéma SDec sont testées, en utilisant différents modes de codage lors de la compétition sur les blocs de référence. Malgré un choix encore simple et limité des blocs de référence, les gains intéressants sont observés. La recherche en amont présente une méthode innovante qui permet d’exploiter davantage la capacité de traitement d’un décodeur. Les techniques d’apprentissage automatique sont exploitées pour but de réduire la signalisation. Les applications pratiques sont données, utilisant un classificateur basé sur les machines à vecteurs de support pour prédire les modes de codage d’un bloc. La classification des blocs utilise des descripteurs causaux qui sont formés à partir de différents types d’histogrammes. Des gains significatifs en débit sont obtenus, confirmant ainsi le potentiel de l’approche. / This Ph.D. thesis studies the novel concept of Smart Decoder (SDec) where the decoder is given the ability to simulate the encoder and is able to conduct the R-D competition similarly as in the encoder. The proposed technique aims to reduce the signaling of competing coding modes and parameters. The general SDec coding scheme and several practical applications are proposed, followed by a long-term approach exploiting machine learning concept in video coding. The SDec coding scheme exploits a complex decoder able to reproduce the choice of the encoder based on causal references, eliminating thus the need to signal coding modes and associated parameters. Several practical applications of the general outline of the SDec scheme are tested, using different coding modes during the competition on the reference blocs. Despite the choice for the SDec reference block being still simple and limited, interesting gains are observed. The long-term research presents an innovative method that further makes use of the processing capacity of the decoder. Machine learning techniques are exploited in video coding with the purpose of reducing the signaling overhead. Practical applications are given, using a classifier based on support vector machine to predict coding modes of a block. The block classification uses causal descriptors which consist of different types of histograms. Significant bit rate savings are obtained, which confirms the potential of the approach.

La compression vidéo HEVC standardisée en 2013 offre des gains de compression dépassant les 50% par rapport au standard de compression précédent MPEG4-AVC/H.264. Ces gains de compression se paient par une augmentation très importante de la complexité de codage. Si on ajoute à cela l’augmentation de complexité générée par l’accroissement de résolution et de fréquence image du signal vidéo d’entrée pour passer de la Haute Définition (HD) à l’Ultra Haute Définition (UHD), on comprend vite l’intérêt des techniques de réduction de complexité pour le développement de codeurs économiquement viables. En premier lieu, un effort particulier a été réalisé pour réduire la complexité des images Intra. Nous proposons une méthode d'inférence des modes de codage à partir d'un pré-codage d'une version réduite en HD de la vidéo UHD. Ensuite, nous proposons une méthode de partitionnement rapide basée sur la pré-analyse du contenu. La première méthode offre une réduction de complexité d'un facteur 3 et la deuxième, d'un facteur 6, contre une perte de compression proche de 5%. En second lieu, nous avons traité le codage des images Inter. En mettant en œuvre une solution d'inférence des modes de codage UHD à partir d'un pré-codage au format HD, la complexité de codage est réduite d’un facteur 3 en considérant les 2 flux produits et d’un facteur 9.2 sur le seul flux UHD, pour une perte en compression proche de 3%. Appliqué à une configuration de codage proche d'un système réellement déployé, l'apport de notre algorithme reste intéressant puisqu'il réduit la complexité de codage du flux UHD d’un facteur proche de 2 pour une perte de compression limitée à 4%. Les stratégies de réduction de complexité mises en œuvre au cours de cette thèse pour le codage Intra et Inter offrent des perspectives intéressantes pour le développement de codeurs HEVC UHD plus économes en ressources de calculs. Elles sont particulièrement adaptées au domaine de la WebTV/OTT qui prend une part croissante dans la diffusion de la vidéo et pour lequel le signal vidéo est codé à des résolutions multiples pour adresser des réseaux et des terminaux de capacités variées. / The High Efficiency Video Coding (HEVC) standard was released in 2013 which reduced network bandwidth by a factor of 2 compared to the prior standard H.264/AVC. These gains are achieved by a very significant increase in the encoding complexity. Especially with the industrial demand to shift in format from High Definition (HD) to Ultra High Definition (UHD), one can understand the relevance of complexity reduction techniques to develop cost-effective encoders. In our first contribution, we attempted new strategies to reduce the encoding complexity of Intra-pictures. We proposed a method with inference rules on the coding modes from the modes obtained with pre-encoding of the UHD video down-sampled in HD. We, then, proposed a fast partitioning method based on a pre-analysis of the content. The first method reduced the complexity by a factor of 3x and the second one, by a factor of 6, with a loss of compression efficiency of 5%. As a second contribution, we adressed the Inter-pictures. By implementing inference rules in the UHD encoder, from a HD pre-encoding pass, the encoding complexity is reduced by a factor of 3x when both HD and UHD encodings are considered, and by 9.2x on just the UHD encoding, with a loss of compression efficiency of 3%. Combined with an encoding configuration imitating a real system, our approach reduces the complexity by a factor of close to 2x with 4% of loss. These strategies built during this thesis offer encouraging prospects for implementation of low complexity HEVC UHD encoders. They are fully adapted to the WebTV/OTT segment that is playing a growing part in the video delivery, in which the video signal is encoded with different resolution to reach heterogeneous devices and network capacities.

In recent years, 3D camera products and prototypes based on Integral imaging (II) technique have gradually emerged and gained broad attention. II is a method that spatially samples the natural light (light field) of a scene, usually using a microlens array or a camera array and records the light field using a high resolution 2D image sensor. The large amount of data generated by II and the redundancy it contains together lead to the need for an efficient compression scheme. During recent years, the compression of 3D integral images has been widely researched. Nevertheless, there have not been many approaches proposed regarding the compression of integral videos (IVs). The objective of the thesis is to investigate efficient coding methods for integral videos. The integral video frames used are captured by the first consumer used light field camera Lytro. One of the coding methods is to encode the video data directly by an H.265/HEVC encoder. In other coding schemes the integral video is first converted to an array of sub-videos with different view perspectives. The sub-videos are then encoded either independently or following a specific reference picture pattern which uses a MVHEVC encoder. In this way the redundancy between the multi-view videos is utilized instead of the original elemental images. Moreover, by varying the pattern of the subvideo input array and the number of inter-layer reference pictures, the coding performance can be further improved. Considering the intrinsic properties of the input video sequences, a QP-per-layer scheme is also proposed in this thesis. Though more studies would be required regarding time and complexity constraints for real-time applications as well as dramatic increase of number of views, the methods proposed inthis thesis prove to be an efficient compression for integral videos.

As recentes inovações em técnicas de Aprendizado de Máquina levaram a uma ampla utilização de modelos inteligentes para resolver problemas complexos que são especialmente difíceis de computar com algoritmos e estruturas de dados convencionais. Em particular, pesquisas recentes em Processamento de Imagens e Vídeo mostram que é possível desenvolver modelos de Aprendizado de Máquina que realizam reconhecimento de objetos e até mesmo de ações com altos graus de confiança. Além disso, os últimos avanços em algoritmos de treinamento para Redes Neurais Profundas (Deep Learning Neural Networks) estabeleceram um importante marco no estudo de Aprendizado de Máquina, levando a descobertas promissoras em Visão Computacional e outras aplicações. Estudos recentes apontam que também é possível desenvolver modelos inteligentes capazes de reduzir drasticamente o espaço de otimização do modo de decisão em codificadores de vídeo com perdas irrelevantes em eficiência de compressão. Todos esses fatos indicam que Aprendizado de Máquina para redução de complexidade em aplicações de vídeo é uma área promissora para pesquisa. O objetivo desta tese é investigar técnicas baseadas em aprendizado para reduzir a complexidade das decisões da codificação HEVC, com foco em aplicações de codificação e transcodificação rápidas. Um perfilamento da complexidade em codificadores é inicialmente apresentado, a fim de identificar as tarefas que requerem prioridade para atingir o objetivo dessa tese. A partir disso, diversas variáveis e métricas são extraídas durante os processos de codificação e decodificação para avaliar a correlação entre essas variáveis e as decisões de codificação associadas a essas tarefas. Em seguida, técnicas de Aprendizado de Máquina são empregadas para construir classificadores que utilizam a informação coletada para prever o resultado dessas decisões, eliminando o custo computacional necessário para computá-las. As soluções de codificação e transcodificação foram desenvolvidas separadamente, pois o tipo de informação é diferente em cada caso, mas a mesma metologia foi aplicada em ambos os casos. Além disso, mecanismos de complexidade escalável foram desenvolvidos para permitir o melhor desempenho taxa-compressão para um dado valor de redução de complexidade. Resultados experimentais apontam que as soluções desenvolvidas para codificação rápida atingiram reduções de complexidade entre 37% e 78% na média, com perdas de qualidade entre 0.04% e 4.8% (medidos em Bjontegaard Delta Bitrate – BD-BR). Já as soluções para trancodificação rápida apresentaram uma redução de 43% até 67% na complexidade, com BD-BR entre 0.34% e 1.7% na média. Comparações com o estado da arte confirmam a eficácia dos métodos desenvolvidos, visto que são capazes de superar os resultados atingidos por soluções similares. / The recent innovations in Machine Learning techniques have led to a large utilization of intelligent models to solve complex problems that are especially hard to compute with traditional data structures and algorithms. In particular, the current research on Image and Video Processing shows that it is possible to design Machine Learning models that perform object recognition and even action recognition with high confidence levels. In addition, the latest progress on training algorithms for Deep Learning Neural Networks was also an important milestone in Machine Learning, leading to prominent discoveries in Computer Vision and other applications. Recent studies have also shown that it is possible to design intelligent models capable of drastically reducing the optimization space of mode decision in video encoders with minor losses in coding efficiency. All these facts indicate that Machine Learning for complexity reduction in visual applications is a very promising field of study. The goal of this thesis is to investigate learning-based techniques to reduce the complexity of the HEVC encoding decisions, focusing on fast video encoding and transcoding applications. A complexity profiling of HEVC is first presented to identify the tasks that must be prioritized to accomplish our objective. Several variables and metrics are then extracted during the encoding and decoding processes to assess their correlation with the encoding decisions associated with these tasks. Next, Machine Learning techniques are employed to construct classifiers that make use of this information to accurately predict the outcome of these decisions, eliminating the timeconsuming operations required to compute them. The fast encoding and transcoding solutions were developed separately, as the source of information is different on each case, but the same methodology was followed in both cases. In addition, mechanisms for complexity scalability were developed to provide the best rate-distortion performance given a target complexity reduction. Experimental results demonstrated that the designed fast encoding solutions achieve time savings of 37% up to 78% on average, with Bjontegaard Delta Bitrate (BD-BR) increments between 0.04% and 4.8%. In the transcoding results, a complexity reduction ranging from 43% to 67% was observed, with average BD-BR increments from 0.34% up to 1.7%. Comparisons with state of the art confirm the efficacy of the designed methods, as they outperform the results achieved by related solutions.

All multimedia devices now incorporate video CODECs that comply with international video coding standards such as H.264 / MPEG4-AVC and the new High Efficiency Video Coding Standard (HEVC) otherwise known as H.265. Although the standard CODECs have been designed to include algorithms with optimal efficiency, large number of coding parameters can be used to fine tune their operation, within known constraints of for e.g., available computational power, bandwidth, consumer QoS requirements, etc. With large number of such parameters involved, determining which parameters will play a significant role in providing optimal quality of service within given constraints is a further challenge that needs to be met. Further how to select the values of the significant parameters so that the CODEC performs optimally under the given constraints is a further important question to be answered. This thesis proposes a framework that uses machine learning algorithms to model the performance of a video CODEC based on the significant coding parameters. Means of modelling both the Encoder and Decoder performance is proposed. We define objective functions that can be used to model the performance related properties of a CODEC, i.e., video quality, bit-rate and CPU time. We show that these objective functions can be practically utilised in video Encoder/Decoder designs, in particular in their performance optimisation within given operational and practical constraints. A Multi-objective Optimisation framework based on Genetic Algorithms is thus proposed to optimise the performance of a video codec. The framework is designed to jointly minimize the CPU Time, Bit-rate and to maximize the quality of the compressed video stream. The thesis presents the use of this framework in the performance modelling and multi-objective optimisation of the most widely used video coding standard in practice at present, H.264 and the latest video coding standard, H.265/HEVC. When a communication network is used to transmit video, performance related parameters of the communication channel will impact the end-to-end performance of the video CODEC. Network delays and packet loss will impact the quality of the video that is received at the decoder via the communication channel, i.e., even if a video CODEC is optimally configured network conditions will make the experience sub-optimal. Given the above the thesis proposes a design, integration and testing of a novel approach to simulating a wired network and the use of UDP protocol for the transmission of video data. This network is subsequently used to simulate the impact of packet loss and network delays on optimally coded video based on the framework previously proposed for the modelling and optimisation of video CODECs. The quality of received video under different levels of packet loss and network delay is simulated, concluding the impact on transmitted video based on their content and features.

Actuellement les communications véhiculaires sont devenues une réalité guidée par diverses applications. Notamment, la diffusion de vidéo de qualité élevée avec des contraintes de faible latence requises par les applications temps réel. Grâce au niveau de compression jamais atteint auparavant, l’encodeur H.265/HEVC est très prometteur pour la diffusion de vidéos en temps réel dans les réseaux ad hoc véhiculaire (VANET). Néanmoins, la qualité de la vidéo reçue est pénalisée par les mauvaises caractéristiques du canal de transmission (disponibilité, non stationnarité, rapport signal à bruit, etc.). Afin d’améliorer et d’assurer une qualité vidéo minimale à la réception nous proposons dans ce travail une optimisation conjointe source-canal-protocole de la transmission en tenant compte à la fois des paramètres de transmission et d’encodage vidéo. Dans un premier temps, nous montrons l’intérêt et le gain apporté par les solutions dites inter-couches « cross-layer ». Par la suite, nous développons deux approches l’une exploitant un « cross-layer » entre la couche application et la couche MAC et une seconde exploitant les protocoles de transports dans l’adaptation du flux vidéo. En ce qui concerne la première approche nous proposons une solution utilisant une gestion hiérarchique des trames au niveau des files d’attentes de la couche MAC, basée sur l’importance des images du flux vidéo. Dans une seconde solution, nous retenons le codage par descriptions multiples comme solution de protection à la source. Les résultats de simulations obtenus pour plusieurs types de scénarios véhiculaires réalistes montrent que les différents schémas de transmission véhiculaire proposés offrent des améliorations significatives en termes de qualité vidéo à la réception et de retard de bout en bout par rapport aux schémas classiques. / At present, vehicular communications have become a reality guided by various applications. In particular, high-quality video delivery with low latency constraints is required for real-time applications. The new state-of-the-art high-effciency video coding (HEVC) standard is very promising for real-time video streaming in vehicular ad hoc networks (VANET). Nevertheless, these networks have variable channel quality and a limited bandwidth that penalizes the overall performances of end-to-end video transmission. In order to meet these constraints, we proposed in this work to consider both transmission and video encoding parameters through a joint source-channel-protocol coding approach to provide an improvement in video transmission. First, we have shown the interest and the gain brought by the "cross-layer" solutions. Then, we developed two approaches. The first one exploits a "crosslayer" solution between the application layer and the Medium Access Control (MAC) layer while the second exploits the transport layer protocols in the adaptation of the video stream. Regarding the first approach, we have proposed solutions to allocate the frames to the most appropriate Access Category (AC) queue on the MAC layer based on the image importance in the video stream. In another solution, we chose multiple descriptions source coding as an error resilient solution. Thus, the simluation results obtained for different realistic vehicular scenarios demonstrate that the proposed transmission schemes offer significant video quality improvements and end-to-end delay reduction compared to conventional transmission schemes.

The current trend of digital convergence creates the need for the video encoder and decoder system, known as codec in short, that should support multiple video standards on a single platform. In a modern video codec, quantization is a key unit used for video compression. In this thesis, a generalized quantization algorithm and hardware implementation is presented to compute quantized coefficient for six different video codecs including the new developing codec High Efficiency Video Coding (HEVC). HEVC, successor to H.264/MPEG-4 AVC, aims to substantially improve coding efficiency compared to AVC High Profile. The thesis presents a high performance circuit shared architecture that can perform the quantization operation for HEVC, H.264/AVC, AVS, VC-1, MPEG- 2/4 and Motion JPEG (MJPEG). Since HEVC is still in drafting stage, the architecture was designed in such a way that any final changes can be accommodated into the design. The proposed quantizer architecture is completely division free as the division operation is replaced by multiplication, shift and addition operations. The design was implemented on FPGA and later synthesized in CMOS 0.18 μm technology. The results show that the proposed design satisfies the requirement of all codecs with a maximum decoding capability of 60 fps at 187.3 MHz for Xilinx Virtex4 LX60 FPGA of a 1080p HD video. The scheme is also suitable for low-cost implementation in modern multi-codec systems.

When distributing multiple TV programs on a fixed bandwidth channel, the bit rate of each video stream is often constant. Since video sent at a constant quality is typically wildly varying, this is a very unoptimal solution. By instead sharing the total bit rate among all programs, the video quality can be increased by allocating bit rate where it is needed. This thesis explores the statistical multiplexing problem for a specific hardware platform with the limitations and advantages of that platform. A solution for statistical multiplexing is proposed and evaluated using the major codecs used for TV distribution today. The main advantage of the statistical multiplexer is a lot more even quality and a higher minimum quality achieved across all streams. While the solution will need a faster method for bit rate approximation for a more practical solution in terms of performance, the solution is shown to work as intended.