La migration vers la résolution HD de la plupart des applications multimédias visuelles a nécessité la création de nouveaux standards de compression vidéo tels que le H264/AVC (Advanced Video Coding) et le HEVC (High Efficiency Video Coding). Ces standards sont caractérisés par des hautes performances de codage en termes de taux de compression et qualité vidéo par rapport aux normes précédentes. Cependant, ces performances entraînent de grandes complexités de calcul ce qui rend difficile d'assurer un encodage en temps réel pour la résolution HD sur des processeurs monocœurs programmables qui sont les plus répandus. De plus, comme actuellement les systèmes embarqués sont de plus en plus utilisés dans diverses applications multimédias, concevoir une solution logicielle embarquée pour l'encodeur H264/AVC constitue ainsi un défit très difficile puisqu'il faut répondre aux exigences de l'embarqué au niveau des ressources matérielles comme la mémoire et de la consommation d'énergie. Les récents systèmes embarqués dotés de la technologie multicœur représentent une solution attractive pour surmonter ces problèmes. Dans ce contexte, cette thèse s'intéresse à exploiter la performance de la nouvelle génération de DSP multicœurs de Texas Instruments pour concevoir un encodeur H264/AVC embarqué de résolution HD fonctionnant en temps réel. Nous visons une solution logicielle, caractérisée par une forte flexibilité, par rapport aux IPs existants, qui permet de tout paramétrer (qualité, débit etc). Cette flexibilité logicielle permet aussi l'évolutivité de système en suivant les améliorations de codage comme la migration vers la nouvelle norme HEVC, partiellement abordée dans cette thèse. Nous présentons ainsi les diverses optimisations appliquées que ce soient algorithmiques, architecturales et structurelles afin d'améliorer la vitesse d'encodage sur un seul cœur DSP avant de passer à une implémentation multicœur. Ensuite, nous proposons des implémentations parallèles de l'encodeur H264/AVC sur différentes unités de calcul en exploitant le parallélisme potentiel au sein de la chaîne d'encodage afin de satisfaire la contrainte de temps réel tout en assurant une bonne performance de codage en termes de qualité vidéo et débit binaire. Nous étudions également le problème d'allocation des ressources (ressources de calcul, ressources mémoire, ressources de communication) avec de fortes contraintes temporelles d'exécution. Finalement, cette thèse ouvre la voie vers l'implémentation de la nouvelle norme de codage vidéo HEVC sur deux systèmes embarqués monocœurs dans le but de préparer une solution logicielle embarquée pour les futurs travaux de recherche / The trend toward HD resolution in most of visual multimedia applications has involved the emergence of a large number of video compression standards such as H.264/AVC (Advanced Video Coding) and HEVC (High Efficiency Video Coding). These standards are characterized by high coding performances in terms of compression ratio and video quality compared to previous standards. However, these performances come with large computational complexities which make it difficult to meet real-time encoding for HD resolution on the most common single-core programmable processors. Moreover, as embedded systems have become increasingly used in various multimedia applications, designing an embedded software solution for the H264/AVC encoder represents another difficult challenge since we have to meet the embedded requirements in terms of hardware resources such as memory and power consumption. The new embedded systems with multicore technology represent an attractive solution to overcome these problems. In this context, this thesis is interested in exploiting the performance of the new generation of Texas Instruments multicore DSPs to design an embedded real-time H264/AVC high definition video encoder. We aim a software solution, characterized by high flexibility that allows setting all parameters (quality, bitrate etc) compared to existing IPs. This software flexibility allows also the system scalability by following the coding enhancements as the migration to the newest HEVC standard. Thus, we present the algorithmic, architectural, and structural optimizations which are applied to improve the encoding speed on a single DSP core before moving to a multicore implementation. Then, we propose parallel implementations of the H264/AVC encoder exploiting the multicore architecture of our platform and the potential parallelism in the encoding chain in order to meet real-time constraints while ensuring a good performance in terms of bitrate and video quality. We also explore the problem of resources allocation (computing resources, storage resources, communication resources) with hard execution time constraints. Finally, this thesis opens the way towards the implementation of the new HEVC video coding standard on two embedded systems in order to prepare a software solution for future research

<p>To provide universal multimedia experience, multimedia streaming services need to transparently handle the variation and heterogeneity in operating environment. From the standpoint of streaming application, video adaptation techniques are intended to cope with the environmental variations by utilizing manipulations of the video content itself. Scalable video coding (SVC) schemes, like that suggested by the standards H.264 and its SVC extension, is highly attractive for designing a self-adaptive video streaming system. When SVC is employed in streaming system, the produced video stream can be then easily truncated or tailored to form several sub-streams which can be decoded separately to obtain a range of preferable picture size, quality and frame rate. However, questions about how to perform the adaptation using SVC and how much adaptation SVC enables are still remaining research issues. We still lack a thorough understanding of how to automate the scaling procedure in order to achieve an optimal video Quality-of-Experience for end users.</p><p>Video QoE, depends highly on human perception. In this thesis, we introduce several video QoE studies around the usability of H.264 SVC. Several factors that contribute significantly to the overall QoEs have been identified and evaluated in these studies. As an example of application usage related factor, playback smoothness and application response time are critical performance measures which can benefit from temporal scalability. Targeting on applications that requires frequent interactivity, we propose a transcoding scheme that fully utilizes the benefits of Switching P and Switching I frames specified in H.264 to enhance video stream's temporal scalability.  Focusing on visual quality related factors, a series of carefully designed subjective quality assessment tests have been performed on mobile devices to investigate the effects of multi-dimensional scalability on human quality perception. Our study reveals that QoE degrades non-monotonically with bitrate and that scaling order preferences are content-dependent. Another study find out that the flickering effect caused by frequent switching between layers in SVC compliant bit-streams is highly related to the switching period. When the period is above a certain threshold, the flickering effect will disappear and layer switching should not be considered as harmful. We have also examined user perceived video quality in 3D virtual worlds. Our results show that the avatars' distance to the virtual screen in 3D worlds contribute significant to the video QoE, i.e., for a wide extent of distortion, there exists always a feasible virtual distance from where the distortion is not detectable for most of people, which makes sense to perform video adaptation.</p><p>The work presented in this thesis is supposed to help improving the design of self adaptive video streaming services that can deliver video content independently of network technology and end-device capability while seeking the best possible experience for video.</p> / Ardendo småföretagsdoktorand

To provide universal multimedia experience, multimedia streaming services need to transparently handle the variation and heterogeneity in operating environment. From the standpoint of streaming application, video adaptation techniques are intended to cope with the environmental variations by utilizing manipulations of the video content itself. Scalable video coding (SVC) schemes, like that suggested by the standards H.264 and its SVC extension, is highly attractive for designing a self-adaptive video streaming system. When SVC is employed in streaming system, the produced video stream can be then easily truncated or tailored to form several sub-streams which can be decoded separately to obtain a range of preferable picture size, quality and frame rate. However, questions about how to perform the adaptation using SVC and how much adaptation SVC enables are still remaining research issues. We still lack a thorough understanding of how to automate the scaling procedure in order to achieve an optimal video Quality-of-Experience for end users. Video QoE, depends highly on human perception. In this thesis, we introduce several video QoE studies around the usability of H.264 SVC. Several factors that contribute significantly to the overall QoEs have been identified and evaluated in these studies. As an example of application usage related factor, playback smoothness and application response time are critical performance measures which can benefit from temporal scalability. Targeting on applications that requires frequent interactivity, we propose a transcoding scheme that fully utilizes the benefits of Switching P and Switching I frames specified in H.264 to enhance video stream's temporal scalability.  Focusing on visual quality related factors, a series of carefully designed subjective quality assessment tests have been performed on mobile devices to investigate the effects of multi-dimensional scalability on human quality perception. Our study reveals that QoE degrades non-monotonically with bitrate and that scaling order preferences are content-dependent. Another study find out that the flickering effect caused by frequent switching between layers in SVC compliant bit-streams is highly related to the switching period. When the period is above a certain threshold, the flickering effect will disappear and layer switching should not be considered as harmful. We have also examined user perceived video quality in 3D virtual worlds. Our results show that the avatars' distance to the virtual screen in 3D worlds contribute significant to the video QoE, i.e., for a wide extent of distortion, there exists always a feasible virtual distance from where the distortion is not detectable for most of people, which makes sense to perform video adaptation. The work presented in this thesis is supposed to help improving the design of self adaptive video streaming services that can deliver video content independently of network technology and end-device capability while seeking the best possible experience for video. / Ardendo småföretagsdoktorand

Uma sequencia de vídeo pode ser adquirida de forma progressiva ou entrelaçada. No padrão de codificação de vídeo H.264/AVC os campos de uma imagem entrelaçada podem ser codificados em modo frame (campos top e bottom entrelaçados) ou em modo field (campos top e bottom agrupados separadamente). Quando a escolha é adaptativa para cada par de macro blocos a codificação é chamada de Macroblock Adaptive Frame- Field (MBAFF). Inovações na predição inter-quadro do H.264/AVC contribuíram significantemente para a performance do padrão alcançar o dobro da taxa de compressão do seu antecessor (ITU, 1994), ao custo de um grande aumento de complexidade computacional do CODEC. Dentro da predição inter-quadro, o bloco de compensação de movimento (MC) é responsável pela reconstrução de um bloco de pixels. No decodificador apresentado em (BONATTO, 2012) está integrada uma solução em hardware para o MC que suporta a maior parte do conjunto de ferramentas do perfil Main do H.264/AVC. A compensação de movimento pode ser dividida em predição de vetores e processamento de amostras. No processamento de amostras é realizada a interpolação e a ponderação de amostras. O módulo de ponderação de amostras, ou predição ponderada, utiliza fatores de escala para escalonar as amostras na saída do MC. Isso é muito útil quando há esvanecimento no vídeo. Inicialmente este trabalho apresenta um estudo do processo de compensação de movimento, segundo o padrão de codificação de vídeo H.264/AVC. São abordadas todas as ferramentas da predição inter-quadro, incluindo o tratamento de vídeo entrelaçado e todos os possíveis modos de codificação para o mesmo. A seguir é apresentada uma arquitetura em hardware para a predição ponderada do MC. Esta arquitetura atende o perfil main do H.264/AVC, que prevê a decodificação de imagens frame, field ou MBAFF. A arquitetura apresentada é baseada no compensador de movimento contido no decodificador apresentado em (BONATTO, 2012), que não tem suporte a predição ponderada e a vídeo entrelaçado. A arquitetura proposta é composta por dois módulos: Scale Factor Prediction (SFP) e Weighted Samples Prediction (WSP) . A arquitetura foi desenvolvida em linguagem VHDL e a simulação temporal mostrou que a mesma pode decodificar imagens MBAFF em tempo real @60i. Dessa forma, tornando-se uma ferramenta muito útil ao desenvolvimento de sistemas de codificação e decodificação em HW. Não foi encontrada, na literatura atual, uma solução em hardware para compensação de movimento do padrão H.264/AVC com suporte a codificação MBAFF. / A video sequence can be acquired in a progressive or interlaced mode. In the video coding H.264/AVC standard an interlaced picture can be encoded in frame mode (top and bottom fields interlaced) or field mode (top and bottom fields combined separately). When the choice for each pair of macro-blocks coding is adaptive, it is called Macroblock Adaptive Frame-Field (MBAFF). The innovations in the inter-frame prediction of H.264/AVC contributed significantly to the performance of the standard that achieved twice the compression ratio of its predecessor (ITU, 1994), at the cost of a large increase in computational complexity of the CODEC. In the inter-frame prediction, the motion compensation (MC) module is responsible for the reconstruction of a pixel's block. In the decoder shown in (BONATTO 2012) an integrated hardware solution to the MC is included which can decode most of the H.264/AVC main profile tools. The motion compensation can be divided into motion vectors prediction and sample processing. In the sample processing part, samples interpolation and weighting are performed. The weighted samples prediction module uses scale factors to weight the samples for generating the output pixels. This is useful in video fading. Initially, this work presents a study of the motion compensation process, according to the H.264/AVC standard. It covers all of inter-frame prediction tools, including all possible coding modes for interlaced video. A hardware architecture for the weighted samples prediction of MC is shown next. It is in compliance with the main profile of H.264/AVC standard, therefore it can decode frame, field and MBAFF pictures. The architecture presented is based on the motion compensator used in the (BONATTO, 2012) decoder, which does not support the weighted prediction and interlaced video. The purposed architecture is composed by two modules: Scale Factor Prediction (SFP) and Weighted Samples Prediction (WSP). The hardware implementation was described using VHDL and the timing simulation has shown that it can decode MBAFF pictures in real time @60i. Therefore, this is an useful tool for hardware CODEC development. Similar hardware solution for H.264/AVC weighted prediction that supports MBAFF coding was not found is previous works.

Uma sequencia de vídeo pode ser adquirida de forma progressiva ou entrelaçada. No padrão de codificação de vídeo H.264/AVC os campos de uma imagem entrelaçada podem ser codificados em modo frame (campos top e bottom entrelaçados) ou em modo field (campos top e bottom agrupados separadamente). Quando a escolha é adaptativa para cada par de macro blocos a codificação é chamada de Macroblock Adaptive Frame- Field (MBAFF). Inovações na predição inter-quadro do H.264/AVC contribuíram significantemente para a performance do padrão alcançar o dobro da taxa de compressão do seu antecessor (ITU, 1994), ao custo de um grande aumento de complexidade computacional do CODEC. Dentro da predição inter-quadro, o bloco de compensação de movimento (MC) é responsável pela reconstrução de um bloco de pixels. No decodificador apresentado em (BONATTO, 2012) está integrada uma solução em hardware para o MC que suporta a maior parte do conjunto de ferramentas do perfil Main do H.264/AVC. A compensação de movimento pode ser dividida em predição de vetores e processamento de amostras. No processamento de amostras é realizada a interpolação e a ponderação de amostras. O módulo de ponderação de amostras, ou predição ponderada, utiliza fatores de escala para escalonar as amostras na saída do MC. Isso é muito útil quando há esvanecimento no vídeo. Inicialmente este trabalho apresenta um estudo do processo de compensação de movimento, segundo o padrão de codificação de vídeo H.264/AVC. São abordadas todas as ferramentas da predição inter-quadro, incluindo o tratamento de vídeo entrelaçado e todos os possíveis modos de codificação para o mesmo. A seguir é apresentada uma arquitetura em hardware para a predição ponderada do MC. Esta arquitetura atende o perfil main do H.264/AVC, que prevê a decodificação de imagens frame, field ou MBAFF. A arquitetura apresentada é baseada no compensador de movimento contido no decodificador apresentado em (BONATTO, 2012), que não tem suporte a predição ponderada e a vídeo entrelaçado. A arquitetura proposta é composta por dois módulos: Scale Factor Prediction (SFP) e Weighted Samples Prediction (WSP) . A arquitetura foi desenvolvida em linguagem VHDL e a simulação temporal mostrou que a mesma pode decodificar imagens MBAFF em tempo real @60i. Dessa forma, tornando-se uma ferramenta muito útil ao desenvolvimento de sistemas de codificação e decodificação em HW. Não foi encontrada, na literatura atual, uma solução em hardware para compensação de movimento do padrão H.264/AVC com suporte a codificação MBAFF. / A video sequence can be acquired in a progressive or interlaced mode. In the video coding H.264/AVC standard an interlaced picture can be encoded in frame mode (top and bottom fields interlaced) or field mode (top and bottom fields combined separately). When the choice for each pair of macro-blocks coding is adaptive, it is called Macroblock Adaptive Frame-Field (MBAFF). The innovations in the inter-frame prediction of H.264/AVC contributed significantly to the performance of the standard that achieved twice the compression ratio of its predecessor (ITU, 1994), at the cost of a large increase in computational complexity of the CODEC. In the inter-frame prediction, the motion compensation (MC) module is responsible for the reconstruction of a pixel's block. In the decoder shown in (BONATTO 2012) an integrated hardware solution to the MC is included which can decode most of the H.264/AVC main profile tools. The motion compensation can be divided into motion vectors prediction and sample processing. In the sample processing part, samples interpolation and weighting are performed. The weighted samples prediction module uses scale factors to weight the samples for generating the output pixels. This is useful in video fading. Initially, this work presents a study of the motion compensation process, according to the H.264/AVC standard. It covers all of inter-frame prediction tools, including all possible coding modes for interlaced video. A hardware architecture for the weighted samples prediction of MC is shown next. It is in compliance with the main profile of H.264/AVC standard, therefore it can decode frame, field and MBAFF pictures. The architecture presented is based on the motion compensator used in the (BONATTO, 2012) decoder, which does not support the weighted prediction and interlaced video. The purposed architecture is composed by two modules: Scale Factor Prediction (SFP) and Weighted Samples Prediction (WSP). The hardware implementation was described using VHDL and the timing simulation has shown that it can decode MBAFF pictures in real time @60i. Therefore, this is an useful tool for hardware CODEC development. Similar hardware solution for H.264/AVC weighted prediction that supports MBAFF coding was not found is previous works.

Uma sequencia de vídeo pode ser adquirida de forma progressiva ou entrelaçada. No padrão de codificação de vídeo H.264/AVC os campos de uma imagem entrelaçada podem ser codificados em modo frame (campos top e bottom entrelaçados) ou em modo field (campos top e bottom agrupados separadamente). Quando a escolha é adaptativa para cada par de macro blocos a codificação é chamada de Macroblock Adaptive Frame- Field (MBAFF). Inovações na predição inter-quadro do H.264/AVC contribuíram significantemente para a performance do padrão alcançar o dobro da taxa de compressão do seu antecessor (ITU, 1994), ao custo de um grande aumento de complexidade computacional do CODEC. Dentro da predição inter-quadro, o bloco de compensação de movimento (MC) é responsável pela reconstrução de um bloco de pixels. No decodificador apresentado em (BONATTO, 2012) está integrada uma solução em hardware para o MC que suporta a maior parte do conjunto de ferramentas do perfil Main do H.264/AVC. A compensação de movimento pode ser dividida em predição de vetores e processamento de amostras. No processamento de amostras é realizada a interpolação e a ponderação de amostras. O módulo de ponderação de amostras, ou predição ponderada, utiliza fatores de escala para escalonar as amostras na saída do MC. Isso é muito útil quando há esvanecimento no vídeo. Inicialmente este trabalho apresenta um estudo do processo de compensação de movimento, segundo o padrão de codificação de vídeo H.264/AVC. São abordadas todas as ferramentas da predição inter-quadro, incluindo o tratamento de vídeo entrelaçado e todos os possíveis modos de codificação para o mesmo. A seguir é apresentada uma arquitetura em hardware para a predição ponderada do MC. Esta arquitetura atende o perfil main do H.264/AVC, que prevê a decodificação de imagens frame, field ou MBAFF. A arquitetura apresentada é baseada no compensador de movimento contido no decodificador apresentado em (BONATTO, 2012), que não tem suporte a predição ponderada e a vídeo entrelaçado. A arquitetura proposta é composta por dois módulos: Scale Factor Prediction (SFP) e Weighted Samples Prediction (WSP) . A arquitetura foi desenvolvida em linguagem VHDL e a simulação temporal mostrou que a mesma pode decodificar imagens MBAFF em tempo real @60i. Dessa forma, tornando-se uma ferramenta muito útil ao desenvolvimento de sistemas de codificação e decodificação em HW. Não foi encontrada, na literatura atual, uma solução em hardware para compensação de movimento do padrão H.264/AVC com suporte a codificação MBAFF. / A video sequence can be acquired in a progressive or interlaced mode. In the video coding H.264/AVC standard an interlaced picture can be encoded in frame mode (top and bottom fields interlaced) or field mode (top and bottom fields combined separately). When the choice for each pair of macro-blocks coding is adaptive, it is called Macroblock Adaptive Frame-Field (MBAFF). The innovations in the inter-frame prediction of H.264/AVC contributed significantly to the performance of the standard that achieved twice the compression ratio of its predecessor (ITU, 1994), at the cost of a large increase in computational complexity of the CODEC. In the inter-frame prediction, the motion compensation (MC) module is responsible for the reconstruction of a pixel's block. In the decoder shown in (BONATTO 2012) an integrated hardware solution to the MC is included which can decode most of the H.264/AVC main profile tools. The motion compensation can be divided into motion vectors prediction and sample processing. In the sample processing part, samples interpolation and weighting are performed. The weighted samples prediction module uses scale factors to weight the samples for generating the output pixels. This is useful in video fading. Initially, this work presents a study of the motion compensation process, according to the H.264/AVC standard. It covers all of inter-frame prediction tools, including all possible coding modes for interlaced video. A hardware architecture for the weighted samples prediction of MC is shown next. It is in compliance with the main profile of H.264/AVC standard, therefore it can decode frame, field and MBAFF pictures. The architecture presented is based on the motion compensator used in the (BONATTO, 2012) decoder, which does not support the weighted prediction and interlaced video. The purposed architecture is composed by two modules: Scale Factor Prediction (SFP) and Weighted Samples Prediction (WSP). The hardware implementation was described using VHDL and the timing simulation has shown that it can decode MBAFF pictures in real time @60i. Therefore, this is an useful tool for hardware CODEC development. Similar hardware solution for H.264/AVC weighted prediction that supports MBAFF coding was not found is previous works.