Automatic translation tools have facilitated navigating multilingual contexts, by providing accessible shortcuts for gathering, processing, and spreading information. As language technologies become more widely used and deployed on a large scale, however, their societal impact has sparked concern both within and outside the research community. This thesis adresses gender bias affecting Machine Translation (MT) and Speech Translation (ST) models. It contributes to this pressing area of research with an interdisciplinary perspective, to raise awareness of bias, improve the understanding of the phenomenon, and investigate best practices and methods to unveil and mitigate it in translation systems.

Audiovisual Translation (AVT) is a field where Machine Translation (MT) has long found limited success mainly due to the multimodal nature of the source and the formal requirements of the target text. Subtitling is the predominant AVT type, quickly and easily providing access to the vast amounts of audiovisual content becoming available daily. Automation in subtitling has so far focused on MT systems which translate source language subtitles, already transcribed and timed by humans. With recent developments in speech translation (ST), the time is ripe for extended automation in subtitling, with end-to-end solutions for obtaining target language subtitles directly from the source speech. In this thesis, we address the key steps for accomplishing the new paradigm of automatic subtitling: data, models and evaluation. First, we address the lack of representative data by compiling MuST-Cinema, a speech-to-subtitles corpus. Segmenter models trained on MuST-Cinema accurately split sentences into subtitles, and enable automatic data augmentation techniques. Having representative data at hand, we move to developing direct ST models for three scenarios: offline subtitling, dual subtitling, live subtitling. Lastly, we propose methods for evaluating subtitle-specific aspects, such as metrics for subtitle segmentation, a product- and process-based exploration of the effect of spotting changes in the subtitle post-editing process, and finally, a comprehensive survey on subtitlers' user experience and views on automatic subtitling. Our findings show the potential of speech technologies for extending automation in subtitling to provide multilingual access to information and communication.

No description available.

京都大学 / 新制・課程博士 / 博士(情報学) / 甲第23541号 / 情博第771号 / 新制||情||132(附属図書館) / 京都大学大学院情報学研究科知能情報学専攻 / (主査)教授 河原 達也, 教授 黒橋 禎夫, 教授 森 信介 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM

The objective of this thesis was to study the coupling of automatic speech recognition (ASR) systems with rule-based machine translation (MT) systems. In this thesis, a unique approach to integrating ASR with MT for speech translation (ST) tasks was proposed. The proposed approach is unique, essentially because it includes the rst rule-based MT system that can process speech data in a word graph format. Compared to other rule-based MT systems, our system processes both a word graph and a stream of words. Thus, the suggested integration method of the ASR and the rule-based MT system is more detailed than a simple software engineering practice. The second reason why it is unique is because this coupling approach performed better than the rst-best and N-best list techniques, which are the only other methods used to integrate an ASR with a rule-based MT system. The enhanced performance of the coupling approach was verified with experiments. The utilization of rule-based MT systems for ST tasks is important / however, there are some unresolved issues. Most of the literature concerning coupling systems has focused on how to integrate ASR with statistical MT rather than rule-based MT. This is because statistical MT systems can process word graphs as input, and therefore, the resolution of ambiguities can be moved to the MT component. With the new approach proposed in this thesis, this same advantage exists in rule-based MT systems. The success of such an approach could facilitate the efficient usage of rule-based systems for ST tasks.

We refactor the traditional two-step approach of automatic speech recognition for spoken language translation. Instead of conventional graphemes, we use phonemes as an intermediate speech representation. Starting with the acoustic model, we revise the cross-lingual transfer and propose a coarse-to-fine method providing further speed-up and performance boost. Further, we review the translation model. We experiment with source and target encoding, boosting the robustness by utilizing the fine-tuning and transfer across ASR and SLT. We empirically document that this conventional setup with an alternative representation not only performs well on standard test sets but also provides robust transcripts and translations on challenging (e.g., non-native) test sets. Notably, our ASR system outperforms commercial ASR systems. 1

The goal of the thesis is to examine methods for presenting the outputs of simul- taneous speech translation systems to users. Compared to manually edited subtitles in movies, current translation systems achieve lower quality in automatic subtitling due to limitations of reading speed and space constraints. Therefore, this thesis examines possibilities of relaxing these constraints, achieving an improvement in overall quality of translated speech presentation. The work relies on existing translation and speech recognition systems, proposing methods solely for user evaluation of the presentation. It provides a thorough analysis of selecting the best layout from a number of presenta- tion options, recommending its usage given available resources. The experimenting is performed given our proposed interface for subtitling and testing. Additionally, the re- sults bring new insights to the relation between the evaluation methods, suggesting less time-consuming evaluation for specific usecases in future experiments. 1

Sequence-to-sequence learning led to significant improvements to machine translation (MT) and automatic speech recognition (ASR) systems. These advancements were first reflected in spoken language translation (SLT) when using a cascade of (at least) ASR and MT with the new "neural" models, then by using sequence-to-sequence learning to directly translate the input audio speech into text in the target language. In this thesis we cover both approaches to the SLT task. First, we show the limits of NMT in terms of robustness to input errors when compared to the previous phrase-based state of the art. We then focus on the NMT component to achieve better translation quality with higher computational efficiency by using a network based on weakly-recurrent units. Our last work involving a cascade explores the effects on the NMT robustness when adding automatic transcripts to the training data. In order to move to the direct speech-to-text approach, we introduce MuST-C, the largest multilingual SLT corpus for training direct translation systems. MuST-C increases significantly the size of publicly available data for this task as well as their language coverage. With such availability of data, we adapted the Transformer architecture to the SLT task for its computational efficiency . Our adaptation, which we call S-Transformer, is meant to better model the audio input, and with it we set a new state of the art for MuST-C. Building on these positive results, we finally use S-Transformer with different data applications: i) one-to-many multilingual translation by training it on MuST-C; ii participation to the IWSLT 19 shared task with data augmentation; and iii) instance-based adaptation for using the training data at test time. The results in this thesis show a steady quality improvement in direct SLT. Our hope is that the presented resources and technological solutions will increase its adoption in the near future, so to make multilingual information access easier in a globalized world.

When this PhD started, the translation of speech into text in a different language was mainly tackled with a cascade of automatic speech recognition (ASR) and machine translation (MT) models, as the emerging direct speech translation (ST) models were not yet competitive. To close this gap, part of the PhD has been devoted to improving the quality of direct models, both in the simplified condition of test sets where the audio is split into well-formed sentences, and in the realistic condition in which the audio is automatically segmented. First, we investigated how to transfer knowledge from MT models trained on large corpora. Then, we defined encoder architectures that give different weights to the vectors in the input sequence, reflecting the variability of the amount of information over time in speech. Finally, we reduced the adverse effects caused by the suboptimal automatic audio segmentation in two ways: on one side, we created models robust to this condition; on the other, we enhanced the audio segmentation itself. The good results achieved in terms of overall translation quality allowed us to investigate specific behaviors of direct ST systems, which are crucial to satisfy real users’ needs. On one side, driven by the ethical goal of inclusive systems, we disclosed that established technical choices geared toward high general performance (statistical word segmentation of the target text, knowledge distillation from MT) cause an exacerbation of the gender representational disparities in the training data. Along this line of work, we proposed mitigation techniques that reduce the gender bias of ST models, and showed how gender-specific systems can be used to control the translation of gendered words related to the speakers, regardless of their vocal traits. On the other side, motivated by the practical needs of interpreters and translators, we evaluated the potential of direct ST systems in the “augmented translation” scenario, focusing on the translation and recognition of named entities (NEs). Along this line of work, we proposed solutions to cope with the major weakness of ST models (handling person names), and introduced direct models that jointly perform ST and NE recognition showing their superiority over a pipeline of dedicated tools for the two tasks. Overall, we believe that this thesis moves a step forward toward adopting direct ST systems in real applications, increasing the awareness of their strengths and weaknesses compared to the traditional cascade paradigm.

Tesis por compendio / [ES] Gracias a avances significativos en aprendizaje profundo, la traducción del habla (ST) se ha convertido en un campo consolidado, lo que permite la utilización de la tecnología ST en soluciones para entornos de producción. Como consecuencia del aumento constante del número de horas de contenido audiovisual generado cada año, así como una mayor sensibilización sobre la importancia de la accesibilidad, la ST está preparada para convertirse en un elemento clave para la producción de contenidos audiovisuales, tanto de ocio como educativos. A pesar de que se ha progresado significativamente en ST, la mayor parte de la investigación se ha centrado en el escenario en diferido (offline), en el cual todo el audio de entrada está disponible. En cambio, la ST en directo (online) es una temática en la que falta mucho por investigar. En concreto, existe un caso de traducción en directo, la traducción continua (streaming), que traduce un flujo continuo de palabras en tiempo real y bajo unas estrictas condiciones de latencia. Este es un problema mucho más realista, que es necesario resolver para que sea posible aplicar la ST a una variedad de tareas de la vida real. Esta tesis está centrada en investigar y desarrollar las técnicas claves que son necesarias para una solución de ST continua. En primer lugar, de cara a permitir el desarrollo y la evaluación de sistemas de ST, se ha recopilado un nuevo conjunto de datos para ST multilingüe, que expande significativamente el número de horas disponibles para ST. A continuación se ha desarrollado un segmentador preparado para la condición continua, que se utiliza para segmentar las transcripciones intermedias de nuestra solución por etapas, que consiste en un sistema de reconocimiento automático del habla (ASR), seguido de un sistema de traducción automática (MT) encargado de traducir las transcripciones intermedias al idioma de destino elegido. Diversas investigaciones han concluido que la calidad de la segmentación es un factor muy influyente es la calidad del sistema MT, por lo que el desarrollo de un segmentador efectivo es un paso fundamental en el proceso de ST continua. Este segmentador se ha integrado en la solución por etapas, y estas se optimizan de manera conjunta para alcanzar el equilibrio óptimo entre calidad y latencia. La ST continua tiene unas restricciones de latencia mucho más estrictas que la ST en directo, ya que el nivel deseado de latencia tiene que mantenerse durante todo el proceso de traducción. Por tanto, es crucial ser capaz de medir de manera precisa esta latencia, pero las métricas estándar de ST en directo no se adaptan bien a esta tarea. Como consecuencia de esto, se proponen nuevos métodos para la evaluación de ST continua, que garantizan unos resultados precisos a la vez que interpretables. Por último, se presenta un nuevo método para mejorar la calidad de la traducción continua mediante el uso de información contextual. Mientras que los sistemas tradicionales de ST en directo traducen audios de manera aislada, existe abundante información contextual que está disponible para mejorar los sistemas de ST continua. Nuestra propuesta introduce el concepto de historia continua, que consiste en el almacenamiento de la información más reciente del proceso de traducción, que se utiliza más adelante por el modelo para mejorar la calidad de la traducción. / [CA] Gràcies a avanços significatius en aprenentatge profund, la traducció de la parla (ST) s'ha convertit en un camp consolidat, la qual cosa permet la utilització de la tecnologia ST en solucions per a entorns de producció. A conseqüència de l'augment constant del nombre d'hores de contingut audiovisual generat cada any, així com una major sensibilització sobre la importància de l'accessibilitat, la ST està preparada per a convertir-se en un element clau per a la producció de continguts audiovisuals, tant d'oci com educatius. A pesar que s'ha progressat significativament en ST, la major part de la recerca s'ha centrat en l'escenari en diferit, en el qual tot l'àudio d'entrada està disponible. En canvi, la ST en directe és una temàtica en la qual falta molt per investigar. En concret, existeix un cas de traducció en directe, la traducció contínua, que tradueix un flux continu de paraules en temps real i sota unes estrictes condicions de latència. Aquest és un problema molt més realista, que és necessari resoldre perquè sigui possible aplicar la ST a una varietat de tasques de la vida real. Aquesta tesi està centrada en investigar i desenvolupar les tècniques claus que són necessàries per a una solució de ST contínua. En primer lloc, de cara a permetre el desenvolupament i l'avaluació de sistemes de ST, s'ha recopilat un nou conjunt de dades per a ST multilingüe, que expandeix significativament la quantitat de dades disponibles per a ST. A continuació s'ha desenvolupat un segmentador preparat per a la condició contínua, que s'utilitza per a segmentar les transcripcions intermèdies de la nostra solució per etapes, que consisteix en un sistema de reconeixement automàtic de la parla (ASR), seguit d'un sistema de traducció automàtica (MT) encarregat de traduir les transcripcions intermèdies a l'idioma de destí triat. Diveros treballs de recerca han conclòs que la qualitat de la segmentació és un factor molt important en la qualitat del sistema MT, per la qual cosa el desenvolupament d'un segmentador efectiu és un pas fonamental en el procés de ST contínua. Aquest segmentador s'ha integrat en la solució per etapes, i aquestes s'optimitzen de manera conjunta per a aconseguir l'equilibri òptim entre qualitat i latència. La ST contínua té unes restriccions de latència molt més estrictes que la ST en directe, ja que el nivell desitjat de latència ha de mantindre's durant tot el procés de traducció. Per tant, és crucial ser capaç de mesurar de manera precisa aquesta latència, però les mètriques estàndard de ST en directe no s'adapten bé a aquesta tasca. A conseqüència d'això, es proposen nous mètodes per a l'avaluació de ST contínua, que garanteixen uns resultats precisos alhora que interpretables. Finalment, es presenta un nou mètode per a millorar la qualitat de la traducció contínua mitjançant l'ús d'informació contextual. Mentre que els sistemes tradicionals de ST en directe tradueixen àudios de manera aïllada, existeix abundant informació contextual que està disponible per a millorar els sistemes de ST contínua. La nostra proposta introdueix el concepte d'història contínua, que consisteix en l'emmagatzematge de la informació més recent del procés de traducció, que s'utilitza més endavant pel model per a millorar la qualitat de la traducció. / [EN] Thanks to significant advances in Deep Learning, Speech Translation (ST) has become a mature field that enables the use of ST technology in production-ready solutions. Due to the ever-increasing hours of audio-visual content produced each year, as well as higher awareness of the importance of media accessibility, ST is poised to become a key element for the production of entertainment and educational media. Although significant advances have been made in ST, most research has focused on the offline scenario, where the entire input audio is available. In contrast, online ST remains an under-researched topic. A special case of online ST, streaming ST, translates an unbounded input stream in a real-time fashion under strict latency constraints. This is a much more realistic problem that needs to be solved in order to apply ST to a variety of real-life tasks. The focus of this thesis is on researching and developing key techniques necessary for a successful streaming ST solution. First, in order to enable ST system development and evaluation, a new multilingual ST dataset is collected, which significantly expands the amount of hours available for ST. Then, a streaming-ready segmenter component is developed to segment the intermediate transcriptions of our proposed cascade solution, which consists in an Automatic Speech Recognition (ASR) system that transcribes the audio, followed by a Machine Translation (MT) system that translates the intermediate transcriptions into the desired language. Research has shown that segmentation quality plays a significant role in downstream MT performance, so the development of an effective streaming segmenter is a critical step in the streaming ST process. This segmenter is then integrated and the components of the cascade are jointly optimized to achieve an appropriate quality-latency trade-off. Streaming ST has much more strict latency constraints than standard online ST, as the desired latency level must be maintained during the whole translation process. Therefore, it is crucial to be able to accurately measure this latency, but the standard online ST metrics are not well suited for this task. As a consequence, new evaluation methods are proposed for streaming ST evaluation, which ensure realistic, yet interpretable results. Lastly, a novel method is presented for improving translation quality through the use of contextual information. Whereas standard online ST systems translate audios in isolation, there is a wealth of contextual information available for improving streaming ST systems. Our approach introduces the concept of streaming history by storing the most recent information of the translation process, which is then used by the model in order to improve translation quality. / The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements no. 761758 (X5Gon) and 952215 (TAILOR), and Erasmus+ Educa- tion programme under grant agreement no. 20-226-093604-SCH (EXPERT); the Government of Spain’s grant RTI2018-094879-B-I00 (Multisub) funded by MCIN/AEI/10.13039/501100011033 & “ERDF A way of making Europe”, and FPU scholarships FPU18/04135; and the Generalitat Valenciana’s research project Classroom Activity Recognition (ref. PROMETEO/2019/111) and predoctoral research scholarship ACIF/2017/055. / Iranzo Sánchez, J. (2023). Streaming Neural Speech Translation [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/199170 / Compendio