Through practice-led research this thesis will explore the phenomenology of interactions between the digital 'other', and the lived experience of the subject through real-time video performance practice. It challenges the assumption that the digital video image is merely or simply other to the subject and aims to re-position the 'other' as an integral part of self where we perform the other. It does this by drawing on Jacques Lacan's Mirror Stage and claims that through digital performance we can suspend divisions between the self and the digital other. By being immersed within the real-time video image the thesis argues we re-enter the Mirror Stage and become captivated within the digital counterpart. Through a disruption in the proprioception of the body there is a crossover of the actual self and digital other which are suspended in each other. Through the use of Head Mounted Display Systems in the work In[bodi]lmental it is claimed that the actual body can In[body] the other subject as part of self. The thesis argues that the digital other is a component of self mediated through new digital technologies to be understood as an augmented self. Therefore it is through an In[bodied] Mirror Stage we momentarily access the loss of the Lacanian real encountered through the uncanny experience. This investigation has been conducted in the form of four digital performance projects defined as Inter-Reactive Explorations I-REs (i-iv).The I-REs were subjected to critical analysis and reflection using a variety of disciplines including: psychoanalysis, philosophy, the study of perception, phenomenology, and ethnography. The methodological framework for this research has been coined 'auto-ethnophenomenology'; a mixed-method approach utilizing auto-ethnography and the phenomenological lived experiences of informants. This model has enabled both the 'I' of the researcher and the other to be equally represented from both first person and third person perspectives. The symbiotic relationship between the theory and the practice is exemplified through the phenomenology of interactions between the digital 'other', and the lived experience of the subjects supported by the writings of Sigmund Freud, Jacques Lacan, Maurice Merleau-Ponty, Drew Leder and Rane Willerslev.

A stereo vision system is a robust method to sense the distance information in a scene. This research explores the stereo vision system from the fundamentals of stereo vision and the computer stereo vision algorithm to the final implementation of the system on a FPGA chip. In a stereo vision system, images are captured by a pair of stereo image sensors. The distance information can be derived from the disparities between the stereo image pair, based on the theory of binocular geometry. With the increasing focus on 3D vision, stereo vision is becoming a hot topic in the areas of computer games, robot vision and medical applications. Particularly, most stereo vision systems are expected to be used in real-time applications. In this thesis, several stereo correspondence algorithms that determine the disparities between stereo image pair are examined. The algorithms can be categorized into global stereo algorithms and local stereo algorithms depending on the optimization techniques. The global algorithms examined are the Dynamic Time Warp (DTW) algorithm and the DTW with quantization algorithm, while the local algorithms examined are the window based Sum of Squared Differences (SSD), Sum of Absolute Differences (SAD) and Census transform correlation algorithms. With analysis among them, the window based SAD correlation algorithm is proposed for implementation on a FPGA platform. The proposed algorithm is implemented onto an Altera DE2 board featuring an Altera Cyclone II 2C35 FPGA. The implemented module of the algorithm is simulated using ModelSim-Altera to verify the correctness of its functionality. Along with a pair of stere image sensors and a LCD monitor, a stereo vision system is built. The entire system realizes a real-time video frame rate of 16.83 frames per second with an image resolution of 640 by 480 and produces disparity maps in which the objects are clearly distinguished by their relative distance information.

Real-time multimedia streaming is an extensively researched topic. The possibility of streaming video over the internet in real time requires smart solutions on many levels at the player and streamer side, as well as along the intermediate network. There are many different methods used to achieve this, but not all of them are suitable for the low latency real-time streaming needed for remote operations of vehicles. This thesis focuses on the bit-rate control at the streamer side to achieve low latency, meaning how the video quality is changed to adapt to the changes in the network. A literature study was conducted, in order to find what algorithms are currently being used for real-time streaming. It investigated both what control methods are used, as well as what feedback metrics are feed to these controllers. These approaches where then evaluated from a theoretical standpoint for real-time low latency streaming on 4G networks together with the rest of the assumed system. Using these discovered methods, two new algorithms were created. They were tested against an already existing benchmark controller, both in simulation and on a real network. As the benchmark algorithm proved to already be using all suitable feedback metrics, only small control alterations where done to the existing benchmark algorithm. The goal for the new algorithms was to increase the total throughput of the video stream, without decreasing the robustness and causing a higher latency.  Simulation and real network tests proved that the new algorithms are unable to provide a higher throughput without increasing the latency. The conclusion is that the benchmark controller is well designed and explicitly configured to work for the goal of low latency video streaming. This being the case with many controllers in the industry, as they are well designed and extensively trimmed for their specific task.

\prefacesection{Abstract} Multi-object tracking (MOT) is a key building block in video analytics and finds extensive use in surveillance, search and rescue, and autonomous driving applications. Object detection, a crucial stage in MOT, dominates in the overall tracking inference time due to its reliance on Deep Neural Networks (DNNs). Despite the superior performance of cutting-edge object detectors, their extensive computational demands limit their real-time application on embedded devices that possess constrained processing capabilities. Hence, we aim to reduce the computational burdens of object detection while maintaining tracking performance. As the first approach, we adapt frame resolutions to reduce computational complexity. During inference, frame resolutions can be tuned according to the complexity of visual scenes. We present DeepScale, a model-agnostic frame resolution selection approach that operates on top of existing fully convolutional network-based trackers. By analyzing the effect of frame resolution on detection performance, DeepScale strikes good trade-offs between detection accuracy and processing speed by adapting frame resolutions on-the-fly. Our second approach focuses on enhancing the efficiency of a tracker by model adaptation. We introduce AttTrack to expedite tracking by interleaving the execution of object detectors of different model sizes in inference. A sophisticated network (teacher) runs for keyframes only while, for non-keyframe, knowledge is transferred from the teacher to a smaller network (student) to improve the latter’s performance. Our third contribution involves exploiting temporal-spatial redundancies to enable real-time multi-camera tracking. We propose the MVSparse pipeline which consists of a central processing unit that aggregates information from multiple cameras (on an edge server or in the cloud) and distributed lightweight Reinforcement Learning (RL) agents running on individual cameras that predict the informative blocks in the current frame based on past frames on the same camera and detection results from other cameras. / Thesis / Doctor of Science (PhD)

Esta tesis doctoral aborda el problema de la transmisión de vídeo, en sistemas de transmisión de televisión y vídeo bajo demanda en tiempo real y de videoconferencia, a través de redes IP, haciendo un énfasis especial cuando la conexión se establece entre dispositivos móviles. En la primera parte del trabajo se introduce la transmisión de video en redes IP y se hace especial incapie en la videoconferencia, pasando posteriormente a presentar el estado del arte en ambos ámbitos. Tratamos y estudiamos cuestiones relacionadas con la compresión de vídeo, la calidad del vídeo recibido, la calidad de experiencia (QoE), sistemas para la mejora de la QoE para la videoconferencia y también tecnologías, como Software Defined Networking (SDN), que nos permiten mejorar dinámicamente la configuración de los dispositivos de red para mejorar la QoE percibida por los usuarios finales. Posteriormente, pasamos a describir el diseño de la arquitectura y el protocolo para la transmisión de vídeo en redes IP, con distintos objetivos. Proponemos una arquitectura y un protocolo para mejorar la QoE de los usuarios en la transmisión de vídeo en Internet Protocol Television (IPTV). También lo hacemos para la transmisión de vídeo en Redes Heterogéneas usando HTML5. Por último proponemos un algoritmo y protocolo para mejorar QoE de la transmisión de vídeo en redes empleadas para la monitorización ambiental, basándonos principalmente en un estudio fundamentado en la codificación en función del color predominante de las imágenes del vídeo. En el siguiente capítulo pasamos a proponer una nueva arquitectura y un protocolo para la mejora de la calidad de experiencia de extremo a extremo (E2E QoE) de los usuarios conectados a una videoconferencia. Definimos el proceso del sistema, la máquina de estados finita y el algoritmo necesario para su correcto funcionamiento. Por último, presentamos las distintas pruebas experimentales que se han realizado para estudiar y determinar los mejores sistemas de codificación, y su comportamiento al utilizar tecnologías como SDN en redes de dispositivos móviles, en Redes Heterogéneas usando HTML5 y en videoconferencia. / This doctoral thesis faces the problem of video transmission, in television transmission systems and video on demand in real time, and video conferencing, in IP networks, making a special emphasis when the connection is established between mobile devices. In the first part of the work, video transmission in IP networks is introduced, and a special emphasis is made on videoconference, afterwards presenting the state of the art in both areas. We address several issues related to video compression, the quality of the video received, the Quality of Experience (QoE), systems for improving the QoE for videoconferencing and also technologies, such as Software Defined Networking (SDN), which allow us to dynamically improve the configuration of the network devices to improve the QoE perceived by the end users. Later, we come to describe the design of the architecture and protocol for the transmission of video in IP networks, with different objectives. We propose an architecture and a protocol to improve the QoE of the end users in the transmission of video in Internet Protocol Television (IPTV). We also do it for the transmission of video in Heterogeneous Networks using HTML5. Finally, we propose an algorithm and protocol to improve QoE of video transmission in networks used for environmental monitoring, fundamentated mainly on a study substantiated on coding based on the predominant color of the images of the video. In the next chapter, we propose a new architecture and a protocol for improving the End-to-End Quality of Experience (E2E QoE) of the users connected to a video conference. We define the process of the system, the finite state machine and the algorithm necessary for its correct operation. Finally, we present the different experimental tests that have been carried out to study and determine the best coding systems, and their behavior when using technologies such as SDN in networks of mobile devices, in Heterogeneous Networks using HTML5 and in videoconference. / Aquesta tesi doctoral aborda el problema de la transmissió de vídeo, en sistemes de transmissió de televisió i vídeo sota demanda en temps real, i de videoconferència, a través de xarxes IP, fent un èmfasi especial quan la connexió s'estableix entre dispositius mòbils. En la primera part del treball s'introdueix la transmissió de vídeo en xarxes IP i es fa especial incapie en la videoconferència, passant posteriorment a presentar a l'estat de l'art en tots dos àmbits. Tractem i estudiem qüestions relacionades amb la compressió de vídeo, la qualitat del vídeo rebut, la qualitat d'experiència (QoE), sistemes per a la millora de la QoE per a la videoconferència i també tecnologies, com a Software Defined Networking (SDN), que ens permeten millorar dinàmicament la configuració dels dispositius de xarxa per a millorar la QoE percebuda pels usuaris finals. Posteriorment, passem a descriure el disseny de l'arquitectura i el protocol per a la transmissió de vídeo en xarxes IP, amb diferents objectius. Proposem una arquitectura i un protocol per a millorar la QoE dels usuaris en la transmissió de vídeo en Internet Protocol Television (IPTV). També ho fem per a la transmissió de vídeo en Xarxes Heterogènies usant HTML5. Finalment proposem un algoritme i un protocol per a millorar QoE de la transmissió de vídeo en xarxes emprades per al monitoratge ambiental, basant-nos principalment en un estudi fonamentat en la codificació en funció del color predominant del vídeo. En el següent capítol passem a proposar una nova arquitectura i un protocol per a la millora de la qualitat d'experiència d'extrem a extrem (I2I QoE) dels usuaris connectats a una videoconferència. Definim el procés del sistema, la màquina d'estats finita i l'algoritme necessari per al seu correcte funcionament. Finalment presentem les diferents proves experimentals que s'han realitzat per a estudiar i determinar els millors sistemes de codificació, i el seu comportament en utilitzar tecnologies com SDN en xarxes de dispositius mòbils, en Xarxes Heterogènies usant HTML5 i en videoconferència. / Jiménez Herranz, JM. (2018). Design of an architecture and communication protocol for video transmission and videoconference [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/107321

With the emergence of video-streaming applications extending beyond the entertainment section and becoming more accessible in everyday interactions. However, developing a video live-streaming application and choosing which services to use can be a hard task whether to go for a pre-built solution or build your own foundation. When deciding which services to include in your product a lot of aspects have to be taken into consideration, regarding performance and so on. The aim of this thesis is to emphasize the importance and relevancy of the development efficiency metric, including soft metrics such as maintainability and understandability. During this thesis three different video live-streaming services were chosen to be compared against each other regarding development efficiency: WebRTC, Agora and GetStream. Baseline applications were developed where each respective chosen service was implemented and further evaluated based on development efficiency. According to development efficiency, GetStream had the highest score compared to the other services and therefore should be the easiest to maintain and understand. On the other hand, WebRTC had the lowest development efficiency score and therefore should be the hardest to maintain and understand, however by choosing WebRTC you are not opting into a third-party service and therefore have more control over the implementation. Overall, it can be concluded that deciding which services to include a multitude of aspects has to be considered.

ITC/USA 2010 Conference Proceedings / The Forty-Sixth Annual International Telemetering Conference and Technical Exhibition / October 25-28, 2010 / Town and Country Resort & Convention Center, San Diego, California / Synchronized video is crucial for data acquisition and telecommunication applications. For real-time applications, out-of-sync video may cause jitter, choppiness and latency. For data analysis, it is important to synchronize multiple video channels and data that are acquired from PCM, MIL-STD-1553 and other sources. Nowadays, video codecs can be easily obtained to play most types of video. However, a great deal of effort is still required to develop the synchronization methods that are used in a data acquisition system. This paper will describe several methods that TTC has adopted in our system to improve the synchronization of multiple data sources.

Background. Instant replay allows an application to highlight events without exporting a video of the whole session. Hardware-accelerated video encoding allows replay footage to be encoded in real-time with less to no impact on the runtime performance of the actual simulation in the application. Hardware-accelerated video encoding, however, is not supported on all devices such as low-tier mobile devices, nor all platforms like web browsers. When hardware-acceleration is not supported, the replay has to be encoded using a software-implemented encoder instead. Objectives. To evaluate if recording rendering API commands is a suitable replacement of real-time encoding when hardware-accelerated video encoding is not supported. Method. An experimental research method is used to make quantitative measurements of the proposed approach, Reincore, and a real-time encoder. The measured metrics is frame time and memory consumption. The Godot game engine is modified with modules for real-time video encoding (H.264, H.265 and VP9 codecs) and rendering API command recording and replaying. The engine is also used to create test scenes to evaluate if object count, image motion, object loading/unloading, replay video resolution and replay video duration has any impact on the runtime overhead of frame time and memory consumption. Results. The implemented rendering API command replayer, Reincore, appears to have minimal to no impact on the frame time overhead in all scenarios, except for a spike in increased frame time when the replayer initializes synchronization. Reincore show to be overall inferior to real-time video encoding in terms of runtime memory overhead. Conclusions. Overall, real-time encoding using the H.264 or H.265 show a similar result in frame time as recording rendering commands. However, command recording implies a more significant overhead of memory usage than real-time encoding. The frame time of using the VP9 codec for real-time encoding is inferior to recording rendering API commands. / Bakgrund. Återspelning tillåter applikationer att visa upp händelser utan att exportera en video för hela sessionen. Hårdvaruaccelererad videokodning tillåter video av återspelning att kodas i realtid med minimal påverkan på applikationens prestanda för simulering. Hårdvaruaccelererad videokodning stöds dock inte alltid på alla enheter eller plattformar, så som lågt presterande mobila enheter eller webbläsare. När hårdvaruacceleration inte stöds, måste videokodning ske med en mjukvarubaserad implementering istället. Syfte. Att utvärdera om återspelning genom inspelade renderingskommandon som fördröjer arbetet för videokodning är ett lämpligt alternativ till videokodning i realtid, när hårdvaruacceleration inte stöds. Metod. En experimentel forskningsmetod används för att samla kvantitativ mätdata från den föreslagna tillvägagången, Reincore, and en realtidsvidekodare. Mätdatan består av bildtid och minnesanvändning. Genom att modifiera spelmotorn Godot skapas moduler för realtids-videokodning samt inspelning av renderingskommandon. Spelmotorn används också för att skapa testscener för att utvärdera om antal objekt, bildrörelse, skapande av objekt under körning, upplösning eller videolängd har någon inverkan på bildtid eller minnesanvändning. Resultat. Den implementerade renderingskommando-inspelaren, Reincore, visar minimal påverkan på bildtid, med undantag för en temporär ökning när återspelaren initierar synkronisering. Reincore visar sig vara underlägsen till realtids-videokodning när det gäller minnesanvändning. Slutsatser. Realtids-videokodning med H.264 eller H.265 som video-codec visar övergripande bättre resultat för återspelning än renderingskommandoinspelning, när det gäller både bildtid samt minnesanvändning. Bildtiden för VP9 video-codec för realtids-videokodning visar däremot sämre resultat än renderingskommandinspelning.

Le but de cette thèse est de proposer des mécanismes efficaces pour l'ordonnancement des chunks et l'allocation de la bande passante dans le contexte de la transmission vidéo sur les réseaux P2P,afin d'offrir une meilleure qualité de service pour l'utilisateur final. Dans un premier temps nousavons proposé un mécanisme d'ordonnancement des chunks pour la transmission de vidéomulticouche dans les réseaux P2P. Le mécanisme proposé est basé sur une nouvelle technique quipermet de sélectionner les chunks adéquats et les demander des pairs les plus appropriés. Ensuitenous avons proposé un mécanisme d'allocation de la bande passante, toujours dans le cadre detransmission de vidéo multicouche dans les réseaux P2P. Le pair émetteur organise une enchère pour«vendre » sa bande passante. L'allocation tient en considération la priorité des pairs et l'importancedes couches demandées. Finalement nous avons proposé un mécanisme d'adaptation lisse « smooth» d'une vidéo multicouche transportée sur un réseau P2P.Après une introduction, nous présentons dans le chapitre 2 les motivations du travail le but du travailet les problèmes recherche qui demeurent. Dans ce chapitre nous présentons les composants dessystèmes P2P et tout particulièrement la distribution et l'adaptation de contenus. Dans ce cadre,nous proposons une classification des applications de streaming vidéo P2P ainsi que des mécanismesd'allocation de bande passante et d'ordonnancement pour le streaming pair-à-pair. Nous nousintéressons également aux techniques d'adaptation de la qualité en se focalisant plusparticulièrement sur la norme SVC (Scalable Video Coding).Le chapitre 3 propose des mécanismes de priorisation pour la planification de streaming P2P multicouches.Nous proposons une heuristique pour résoudre un problème général d'affectationgénéralisé (Generalized Assignment Problem – GAP). La solution présentée est ensuite adaptée aucas du streaming non multicouches. Les résultats issus des simulations montrent que les solutionsproposées donnent de meilleurs résultats que les solutions traditionnelles.Le chapitre 4 décrit un mécanisme d'allocation dynamique de la bande passante pour les réseaux destreaming P2P multicouches qui se base sur l'allocation d'une bande passante aux pairs tout enassurant un minimum de qualité de service à l'ensemble des pairs. Les bonnes performances desmécanismes proposés, qui sont détaillées à travers l'étude du ratio concernant l'utilisation de labande passante ainsi que du niveau de satisfaction des pairs, montrent que ces derniers permettentd'obtenir une utilisation optimale de la bande passante.Le chapitre 5 porte sur le lissage du streaming multicouches dans les réseaux P2P en se basant sur lesmétriques liées à la variation de la fréquence et de l'amplitude. Les mécanismes proposés ont étéimplémentés dans un banc d'essai réel et l'évaluation des performances montrent l'efficacité desmécanismes pour le lissage du streaming.Dans le chapitre 6 (conclusion and perspectives), nous résumons les contributions proposées danscette thèse ainsi qu’une ouverture sur les travaux futures / Recently we witnessed an increasing demand for scalable deployment of real-time multimediastreaming applications over Internet. In this context, Peer-to-Peer (P2P) networks are playing asignificant role for supporting large-scale and robust distribution of multimedia content to end-users.However, due to peers’ dynamicity, heterogeneity of terminals and access networks, the deploymentof real-time video streaming applications over P2P networks arises lot of challenges. Indeed, animportant issue in P2P overlays is the capacity to self-organize in the face of the dynamic behavior ofpeers in order to ensure content availability and continuity. In addition, the heterogeneity in networks,terminals, and P2P characteristics make the situation more challenging. In this context, layered videostreaming in P2P networks has drawn great interest to overcome these challenges, since it can notonly accommodate large numbers of users, but also handle heterogeneity of peers. However, there isstill a lack of comprehensive studies on video data blocks (chunks) scheduling and bandwidthallocation for the smooth playout in layered streaming over P2P networks.The aim of this thesis is to analyze these concerns and to propose an efficient real-time chunksscheduling and bandwidth allocation mechanisms for QoS provisioning of layered streamingapplications over P2P networks. Our contributions in this thesis are threefold. First, we propose ascheduling mechanism for layered P2P streaming. The proposed mechanism relies on a novelscheduling algorithm that enables each peer to select appropriate stream layers, along withappropriate peers to provide them. The presented mechanism makes efficient use of networkresources and provides high system throughput. Second, we propose a bandwidth allocation modelfor P2 layered streaming systems based on auction mechanisms to optimize the allocation of senderpeers’ uploads bandwidth. The upstream peers organize auctions to “sell” theirs items (links’bandwidth) according to bids submitted by the downstream peers taking into consideration the peerspriorities and the requested layers importance. The ultimate goal is to satisfy the quality levelrequirement for each peer, while reducing the overall streaming cost. Finally, we present a smoothingmechanism for layered streaming in P2P networks. The mechanism aims to reduce the number oflayer changes under varying network conditions, and ensure a smooth playout for the end-user.

As técnicas de renderização baseadas em imagens permitem que novas visualizações de uma cena sejam geradas a partir de um conjunto de imagens, obtidas a partir de pontos de vista distintos. Pela extensão dessas técnicas para o tratamento de vídeos, podemos permitir a navegação no tempo e no espaço de uma cena obtida a partir de múltiplas câmeras. Nesse trabalho, abordamos o problema de gerar novas visualizações fotorealistas de cenas dinâmicas, com objetos móveis independentes, a partir de vídeos obtidos de múltiplas câmeras com pontos de vista distintos. Os desafios para a solução do problema envolvem a fusão das imagens das múltiplas câmeras minimizando as diferenças de brilho e cor entre elas, a detecção e extração dos objetos móveis da cena e a renderização de novas visualizações combinando um modelo estático da cena com os modelos aproximados dos objetos móveis. Além disso, é importante que novas visualizações possam ser geradas em taxas de quadro interativas de maneira a permitir que um usuário navegue com naturalidade pela cena renderizada. As aplicações dessas técnicas são diversas e incluem aplicações na área de entretenimento, como nas televisões digitais interativas que permitem que o usuário escolha o ponto de vista de filmes ou eventos esportivos, e em simulações para treinamento usando realidade virtual, onde é importante que se haja cenas realistas e reconstruídas a partir de cenas reais. Apresentamos um algoritmo para a calibração das cores capaz de minimizar a diferença de cor e brilho entre as imagens obtidas a partir de câmeras que não tiveram as cores calibradas. Além disso, descrevemos um método para a renderização interativa de novas visualizações de cenas dinâmicas capaz de gerar visualizações com qualidade semelhante à dos vídeos da cena. / Image-based rendering techniques allow the synthesis of novel scene views from a set of images of the scene, acquired from different viewpoints. By extending these techniques to make use of videos, we can allow the navigation in time and space of a scene acquired by multiple cameras. In this work, we tackle the problem of generating novel photorealistic views of dynamic scenes, containing independent moving objects, from videos acquired by multiple cameras with different viewpoints. The challenges presented by the problem include the fusion of images from multiple cameras while minimizing the brightness and color differences between them, the detection and extraction of the moving objects and the rendering of novel views combining a static scene model with approximate models for the moving objects. It is also important to be able to generate novel views in interactive frame rates allowing a user to navigate and interact with the rendered scene. The applications of these techniques are diverse and include applications in the entertainment field, with interactive digital televisions that allow the user to choose the viewpoint while watching movies or sports events, and in virtual-reality training simulations, where it is important to have realistic scenes reconstructed from real scenes. We present a color calibration algorithm for minimizing the color and brightness differences between images acquired from cameras that didn\'t have their colors calibrated. We also describe a method for interactive novel view rendering of dynamic scenes that provides novel views with similar quality to the scene videos.