Telesurgery relies on fast and reliable data transmission between the surgeon and tele-operator side over lossy and delay constrained networks. Medical data involves audio, video, ECG and Force Feedback data. When these media streams are transmitted through best effort networks, the temporal information gets affected due to network constraints. Major network degradation is due to the Force Feedback device with rendering rate of 1 KHz, hence data is generated every millisecond. In our proposal we concentrate on improving the synchronization of force feedback device on varying networking conditions. Force feedback data is generated by operating a source (surgical) device which controls the movement of remote device. It has a great potential in improving telemedicine facilities, when included with the support of different multimedia services. The channel imposes delay and packet loss constraints for such devices which require unique solutions, unlike audio or video media, due to its high rendering rate. Current research supports Force Feedback in fiber optic communication, packet switched networks. However, such schemes are not feasible in supporting surgical telepresence system. While efforts are made to support force feedback media in wireless medium, few works have addressed delay synchronization and loss of data. There exists no previous work which has attempted to provide an efficient integrated solution where video and force feedback information have been supported by the same network. This thesis focuses in providing an integrated architecture that supports the force feedback data over a collaborative network and improves the data synchronization and packet loss prediction in the remote side over a varying network link. The goal will be to evaluate the support of such data types. We have implemented a Linear Packet Predictor Algorithm which predicts the missing packet value. Data generated from the source device are sent as UDP packets. UDP transmission is unreliable and hence we use an RTP over UDP to make it reliable. Each packet will have the current position of the device and force applied. We use a Microsoft Sidewinder Force Feedback joystick. The handle of the joystick is located at the center of the base. So we record the position of the device on both positive and negative axis moving in a two dimensional space. This device provides rotational movement and hence drastic change in position occurs within milliseconds. Once the packet arrives at the receiver side, the control unit checks for the sequence number of the packet. If continuity is missing then, the control unit passes the packet to the predictor algorithm which predicts the packet else it directly updates the packet to the Virtual Time Rendering Algorithm Another major issue is the delay jitter. On the source (server) side the intra time difference between two packets will be 1msec. But due to varying delay in the network the data packets arrive at the receiver with fluctuating intra time difference. In order to counter the delay jitter effect, we implement the Virtual Time Rendering algorithm which reads the time stamp value at which the packet was generated at the source and modifies the update time at the receiver side. In our work we do not control another device on the remote side, rather an applet which was developed using a Virtual Reality Markup Language in Matlab. Another challenge which is imposed when other multimedia is introduced with force feedback is the intra media synchronization. Real time video is captured from the applet side and given as feedback to the server side to improve the interactivity of the application. At every instant in time, different multimedia data produce data to be updated at the remote end. Since all the information are inter dependent with other media in time, efficient intra media synchronization is required. This thesis also focuses in providing an architecture which not only supports force feedback data but have a multiplexed model which allows an efficient transmission of all surgical information in real time. Each data occupies significant part of bandwidth in the network and the effect of multiplexing might affect the synchronization scheme of the force feedback device. Our architecture supports the efficient transmission of all types of multimedia information and also maintain the synchronization of the scheme. This method is unique with its methodical approach to support different multimedia information.

[pt] Em apresentações hipermídia, umas das principais tarefas coordenadas pelo orquestrador da apresentação é a sincronização entre os diversos objetos componentes, que pode ser obtida através do ajuste elástico do tempo de exibição dos objetos. Esta técnica pode ser aplicada em tempo de compilação, de forma a manter os relacionamentos de sincronização especificados pelo autor, ou em tempo de apresentação, para prevenir qualquer descasamento temporal causado pelos ambientes de transmissão e de execução. Este trabalho descreve um conjunto de mecanismos para executar o ajuste elástico em fluxos MPEG-2 de Vídeo e de Sistemas, propondo algoritmos para a realização da compressão e expansão do tempo de exibição, do controle da ocupação do buffer do decodificador, da sincronização intermídia e da reconstrução do relógio de referência. Visando seu emprego em tempo de execução, todo o processo de ajuste é realizado diretamente no fluxo MPEG, sem qualquer transcodificação. / [en] In hypermedia presentations, one of the main tasks provided by the orchestrator is the synchronization of all presentation objects, which may be achieved by elastic time adjustment of period of exhibition of the objects, or simply timescale adaptation. This technique can be applied at compilation time, in order to keep track of synchronism relationships specified by authors, or at presentation time, to prevent any temporal mismatch caused by transmission or execution environments. This work presents a set of mechanisms to carry out timescale adaptation in MPEG-2 Systems and Video streams, proposing algorithms to perform compression and expansion of exhibition period, also called playback dilation, rate control, inter-media synchronization and clock reconstruction. In order to be performed at execution time, timescale operations are realized directly in compressed MPEG-2 streams, requiring no transcodification.

[pt] Ajuste elástico é a técnica de modificar a duração de exibição de um objeto de mídia. Este trabalho propõe algoritmos de ajuste elástico que funcionam em tempo de exibição para fluxos de áudio comprimidos, com aplicabilidade principal em cenários onde não existe interferência do usuário como, por exemplo, a manutenção da consistência temporal de um documento hipermídia. Os algoritmos de ajuste de áudio são propostos, desenvolvidos e integrados a algoritmos de ajuste de vídeo. Além disso, um algoritmo de resincronização é proposto, e implementado, para preservar o sincronismo intermídia durante a realização de ajuste em fluxos de sistemas MPEG-2. A união dos algoritmos implementados com o desenvolvimento da integração a exibidores de conteúdo deu origem à ferramenta de ajuste, capaz de ser facilmente utilizada por aplicações que precisam solicitar ajuste elástico em fluxos de mídia comprimidos. Como exemplo, a ferramenta de ajuste foi integrada a ferramentas de exibição de um formatador hipermídia. Os algoritmos propostos podem ajustar a duração de áudio em até 10% mantendo a qualidade da percepção do áudio dentro de limites aceitáveis, conforme demonstrado através de medidas de qualidade que também são apresentadas neste trabalho. / [en] Timescale is a technique used to modify media-object playing duration. This work proposes timescale algorithms, to be performed on-the- fly, for compressed audio streams. The focus is situations where there is no user interference as, for example, the temporal consistency maintenance of a hypermedia document. The algorithms are proposed, developed and integrated to video timescale algorithms. Furthermore, a resynchronization algorithm is also proposed, and implemented, in order to maintain inter-media synchronization in MPEG-2 system streams. The timescale tool is composed by the aforementioned algorithms and their integration to content rendering tools. The proposed solution can be easily used in applications that need timescale compressed media streams. As an example, the adjustment tool was integrated to a hypermedia formatter. The proposed algorithms can adjust audio durations up to 10%, maintaining the audio perceptual quality within acceptable limits, as observed through quality measurements also described in this work.

[ES] Actualmente, existe una gran variedad de tecnologías de distribución de contenidos multimedia, de dispositivos de consumo, así como de tipos de contenidos. En cuanto a tecnologías de distribución, dichos contenidos se pueden enviar bien a través de redes broadcast, como las redes DVB, o bien a través de redes broadband, como las redes IP (por ejemplo, a través de Internet). La coordinación y convergencia entre los dos tipos de tecnologías para distribuir contenidos relacionados ofrece un gran potencial a los proveedores de contenidos, así como grandes beneficios a los consumidores (una prueba de ello es el estándar Hybrid Broadcast Broadband TV o HbbTV). Sin embargo, la recepción de contenidos relacionados procedentes de diferentes redes híbridas (broadcast/broadband) conlleva una variabilidad de retardos significativa, por lo que se deben superar una serie de retos importantes, como, por ejemplo, la necesidad de mecanismos que garanticen un consumo sincronizado de dichos contenidos, bien en un mismo dispositivo o bien en distintos dispositivos (locales o remotos). Se han especificado, implementado y evaluado soluciones de sincronización de los procesos de reproducción de contenidos híbridos (prevenientes de redes broadcast y/o broadband) tanto en entornos multidispositivo, entre dispositivos locales (Inter-Device Synchronization o IDES), como en entornos multiusuario, entre destinatarios distribuidos en red (Inter-Destination Media Synchronization o IDMS), incluyendo la sincronización de efectos multisensoriales. Se ha diseñado e implementado una plataforma extremo-a-extremo para la generación, distribución (adaptativa) y consumo de contenidos híbridos relacionados, tanto en un mismo dispositivo como en múltiples dispositivos locales (por ejemplo, escenario multipantalla) o remotos (por ejemplo, TV Social). Se han generado y preparado contenidos multimedia tradicionales (2D) y omnidireccionales (360), enriquecidos con efectos multisensoriales que mejoran el grado de realismo y la inmersividad de los usuarios. La plataforma es compatible con el estándar HbbTV y, además, incluye soluciones eficientes para aspectos clave que no están especificados en dicho estándar, pero que son necesarios para la implementación satisfactoria de servicios de TV híbridos y multidispositivo en la misma, como, por ejemplo: la necesidad de un buffer para el contenido broadcast (existente y opcional hasta la especificación HbbTV 2.0.2); mecanismos de señalización, asociación y descripción de contenido disponible y relacionado con el contenido principal; mecanismos de coordinación e interacción entre los dispositivos de consumo; o soluciones de sincronización para alinear en el tiempo el consumo de contenidos de forma satisfactoria. Los resultados obtenidos en la evaluación realizada muestran que el nivel de sincronización adquirido entre los procesos de reproducción de los diversos tipos de contenido disponibles a través de distintas tecnologías de transmisión alcanza unos valores suficientes para proporcionar al usuario un consumo con un nivel de asincronías imperceptible. Dicha precisión alcanzada ha resultado en una QoE para el consumidor muy satisfactoria, ofreciéndole al mismo un completo ecosistema de consumo de contenidos que incluye tanto contenido multimedia tradicional, como contenido omnidireccional y mulsemedia. / [CA] Actualment, hi ha una gran varietat de tecnologies de distribució de contingut multimèdia, dispositius de consum, així com tipus de contingut. Pel que fa a les tecnologies de distribució, aquest contingut es pot lliurar a través de xarxes de difusió, com ara xarxes DVB, o a través de xarxes de banda ampla, com les xarxes IP (per exemple, a través d'Internet). La coordinació i convergència entre els dos tipus de tecnologies per distribuir continguts relacionats ofereix un gran potencial als proveïdors de continguts, així com grans beneficis per als consumidors (una prova d'això és l'estàndard Hybrid Broadcast Broadband TV o HbbTV). Tanmateix, la recepció de continguts relacionats des de diferents xarxes híbrides (emissió/banda ampla) comporta una important variabilitat de retard, per la qual cosa s'han de superar una sèrie de reptes importants, com la necessitat de mecanismes que garanteixin el consum sincronitzat d'aquests continguts, ja siga en el mateix dispositiu. o en diferents dispositius (locals o remots). S'han especificat, implementat i evaluat solucions de sincronització per a processos de reproducció de continguts híbrids (xarxes de broadcast i/o de banda ampla) tant en entorns multidispositiu, entre dispositius locals (Inter-Device Synchronization o IDES), com en entorns IDMS (Inter-Destination Media Synchronization o IDMS), incloent la sincronització d'efectes multisensorials. S'ha dissenyat i implementat una plataforma d'extrem a extrem per a la generació, distribució (adaptativa) i consum de contingut híbrid relacionat, tant en un sol dispositiu com en múltiples locals (p. ex., escenari multipantalla) o remots (p. ex., Social TV). dispositius. S'han generat i preparat contingut multimèdia tradicional (2D) i omnidireccional (360), enriquit amb efectes multisensorials que milloren el grau de realisme i immersivitat dels usuaris. La plataforma és compatible amb l'estàndard HbbTV i, a més, inclou solucions eficients per a aspectes clau que no s'especifiquen en aquest estàndard, però que són necessaris per a la implantació amb èxit de serveis de televisió híbrids i multidispositiu en el mateix, com ara: la necessitat per a un buffer per a continguts de difusió (existent i opcional fins a HbbTV 2. 0.2); mecanismes de senyalització, associació i descripció dels continguts disponibles i relacionats amb el contingut principal; mecanismes de coordinació i interacció entre dispositius de consum; o solucions de sincronització per alinear en el temps el consum de continguts de manera satisfactòria. Els resultats obtinguts en l'avaluació realitzada mostren que el nivell de sincronització adquirit entre els processos de reproducció dels diferents tipus de continguts disponibles a través de diferents tecnologies de transmissió assoleix valors suficients per proporcionar a l'usuari un consum amb un nivell d'asincronies imperceptibles. La precisió aconseguida ha donat com a resultat una QoE molt satisfactòria per al consumidor, oferint-li un ecosistema complet de consum de continguts que inclou tant contingut multimèdia tradicional com contingut omnidireccional i mulsemedia. / [EN] Currently, there is a wide variety of multimedia content distribution technologies, consumer devices, as well as types of content. In terms of distribution technologies, such content can be delivered either over broadcast networks, such as DVB networks, or over broadband networks, such as IP networks (e.g. over the Internet). The coordination and convergence between the two types of technologies to distribute related content offers great potential to content providers, as well as great benefits to consumers (a proof of this is the Hybrid Broadcast Broadband TV or HbbTV standard). However, the reception of related content from different hybrid networks (broadcast/broadband) entails significant delay variability, so a number of important challenges must be overcome, such as the need for mechanisms to ensure synchronized consumption of such content, either on the same device or on different devices (local or remote). We have specified, implemented and evaluated synchronization solutions for hybrid content playback processes (broadcast and/or broadband networks) both in multi-device environments, between local devices (Inter-Device Synchronization or IDES), and in multi-user environments, between distributed network recipients (Inter-Destination Media Synchronization or IDMS), including the synchronization of multi-sensory effects. An end-to-end platform has been designed and implemented for the generation, distribution (adaptive) and consumption of related hybrid content, both on a single device and on multiple local (e.g., multiscreen scenario) or remote (e.g., Social TV) devices. Traditional (2D) and omnidirectional (360) multimedia content has been generated and prepared, enriched with multisensory effects that enhance the degree of realism and immersiveness for users. The platform is compatible with the HbbTV standard and, in addition, includes efficient solutions for key aspects that are not specified in that standard, but are necessary for the successful implementation of hybrid and multi-device TV services on it, such as: the need for a buffer for broadcast content (existing and optional up to the HbbTV 2. 0.2); mechanisms for signaling, association and description of content available and related to the main content; coordination and interaction mechanisms between consumption devices; or synchronization solutions to align in time the consumption of content in a satisfactory way. The results obtained in the evaluation carried out show that the level of synchronization acquired between the reproduction processes of the different types of content available through different transmission technologies reaches sufficient values to provide the user with a consumption with an imperceptible level of asynchronies. The accuracy achieved has resulted in a very satisfactory QoE for the consumer, offering him a complete content consumption ecosystem that includes traditional multimedia content as well as omnidirectional and mulsemedia content. / Marfil Reguero, D. (2022). Especificación, implementación y evaluación de soluciones de sincronización multimedia híbrida (broadcast/broadband) e inter-destinatario para posibilitar nuevas experiencias televisivas enriquecidas, personalizadas, inmersivas y compartidas [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/185746 / TESIS

Made available in DSpace on 2016-06-02T19:06:06Z (GMT). No. of bitstreams: 1 5369.pdf: 9612866 bytes, checksum: 5042244a0ce446b39888368498a1bff3 (MD5) Previous issue date: 2013-05-08 / Financiadora de Estudos e Projetos / The evolution of ubiquitous computing, techniques for handling continuous media and composition, representation and presentation of multimedia objects, among other areas of information technology and communication, enable the exploration and study of interesting results that can be achieved with the incorporation of these technologies in different areas of knowledge and application. The areas of Electronic Learning (EE) and Massive Online Open Courses (MOOC) were chosen to perform this kind of study and exploration. Besides providing a rich environment for research in education, the work required applied research in ubiquitous computing, human-computer interaction and multimedia systems. The work also included the development and improvement of prototypes, particularly presentation machines and capture & access systems for human activities. As a byproduct, it presents an intrumentalized classroom which can perform the ubiquitous capture of multimodal educational presentation for further generation of an equivalent multimedia learning object. These learning objects can be accessed via Learning Management System (LMS) on the Web in an asynchronous way or in synchronous sessions with professors/tutors/students. The multimedia learning objects may be used in MOOCs and as support for classroom learning. Moreover, the log of users interactions with the objects can provide clues for improvement in the educational presentation. / A evolução da computação ubíqua, das técnicas de manipulação de mídia contínua e da composição, representação e apresentação de objetos multimídia, entre outras áreas das tecnologias da informação e comunicação, viabilizam a exploração e o estudo de resultados interessantes que podem ser alcançados com a incorporação dessas tecnologias em diferentes áreas de conhecimento e de aplicação. Escolheu-se as áreas de Electronic Learning (EE) e Massive Online Open Courses (MOOC) para se realizar esse tipo de estudo e exploração. Além de propiciar um rico ambiente para pesquisas em educação, o trabalho requereu a elaboração de pesquisa aplicada em computação ubíqua, interação humano-computador e sistemas multimídia. O trabalho também envolveu o desenvolvimento e aperfeiçoamento de protótipos, particularmente de máquinas de apresentação e sistemas para captura & acesso de atividades humanas. Como subproduto, é apresentada uma sala de aula instrumentada capaz de realizar a captura ubíqua de apresentações educacionais multimodais para posteriormente transformá-las em objetos de aprendizagem multimídia. Esses objetos de aprendizagem podem ser acessados pela Web, via Learning Management System (LMS), tanto de forma assíncrona como em sessões síncronas entre professores/tutores/alunos. Os objetos de aprendizagem multimídia podem ser utilizados em qualquer contexto educacional (como MOOCs e no apóio ao ensino presencial). Além disso, pode-se utilizar o log das interações dos usuários com os objetos para encontrar possíveis alvos de melhoria nas apresentações educacionais.