Thesis (Ph.D.)--The University of Texas at Arlington, 2007. / Adviser: Diane J. Cook. Includes bibliographical references.

Closed captioning has not improved since early 1970s, while film and television technology has changed dramatically. Closed captioning only conveys verbatim dialogue to the audience while ignoring music, sound effects and speech prosody. Thus, caption viewers receive limited and often erroneous information. My thesis research attempts to add some of the missing sounds and emotions back into captioning using animated text. The study involved two animated caption styles and one conventional style: enhanced, extreme and closed. All styles were applied to two clips with animations for happiness, sadness, anger, fear and disgust emotions. Twenty-five hard of hearing and hearing participants viewed and commented on the three caption styles and also identified the character’s emotions. The study revealed that participants preferred enhanced, animated captions. Enhanced captions appeared to improve access to the emotive information in the content. Also, the animation for fear appeared to be most easily understood by the participants.

Closed captioning has not improved since early 1970s, while film and television technology has changed dramatically. Closed captioning only conveys verbatim dialogue to the audience while ignoring music, sound effects and speech prosody. Thus, caption viewers receive limited and often erroneous information. My thesis research attempts to add some of the missing sounds and emotions back into captioning using animated text. The study involved two animated caption styles and one conventional style: enhanced, extreme and closed. All styles were applied to two clips with animations for happiness, sadness, anger, fear and disgust emotions. Twenty-five hard of hearing and hearing participants viewed and commented on the three caption styles and also identified the character’s emotions. The study revealed that participants preferred enhanced, animated captions. Enhanced captions appeared to improve access to the emotive information in the content. Also, the animation for fear appeared to be most easily understood by the participants.

Thesis (Ph. D.)--University of Texas at Austin, 1998. / Vita. Includes bibliographical references (leaves 145-157). Available also in a digital version from Dissertation Abstracts.

Thesis--PlanB (M.S.)--University of Wisconsin--Stout, 2002. / Includes bibliographical references.

Learning in a multimodal environment entails the presentation of information in a combination of more than one mode (i.e. written words, illustrations, and sound). Past research regarding the benefits of multimodal presentation of information includes both school age children and adult learners (e.g. Koolstra, van der Voort & d'Ydewalle, 1999; Neumen & Koskinen, 1992), as well as both native and non-native language learners (e.g. d'Ydewalle & Gielen, 1992; Kothari et al, 2002). This dissertation focuses how the combination of various modalities are used by learners of differing proficiencies in English to gain better comprehension (cf. Mayer, 1997, 2005; Graber, 1990; Slykhuis et al, 2005). The addition of the written mode (closed captioning) to the already multimodal environment that exists in film and video presentations is analyzed. A Multimodal Multimedia Communicative Event is used to situate the language learner. Research questions focus on the eye movements of the participants as they read moving text both with and without the audio and video modes of information. Small case studies also give a context to four participants by bringing their individual backgrounds and observations to bear on the use of multimodal texts as language learning tools in a second or foreign language learning environment. It was found that Non Native English Speakers (NNS) (L1 Arabic) show longer eye movement patterns in reading dynamic text (closed captioning), echoing past research with static texts while Native Speakers of English (NS) tend to have quicker eye movements. In a multimodal environment the two groups also differed: NNS looked longer at the closed captioning and NS were able to navigate the text presentation quickly. While associative activation (Paivio, 2007) between the audio and print modalities was not found to alter the eye movement patterns of the NNS, participants did alternate between the modalities in search of supplementary information. Other research using additional closed captioning and subtitling have shown that viewing a video program with written text added turns the activity into a reading activity (Jensema, 2000; d'Ydewalle, 1987). The current study found this to be the case, but the results differed in regard to proficiency and strategy.

Thesis (Ph.D.)--Tampereen yliopisto, 2004. / Includes bibliographical references (p. 253-263) and discography (p. 251-252). Also available online.

Netflix is currently the world's largest subscription-based streaming platform, with 221.8 million subscribers worldwide (Maglio, 2022). Part of Netflix's enormous global appeal is its Netflix Original brand of films and TV shows - content it produces specifically for broadcast on its streaming platform. To make its content accessible to deaf and hard-of-hearing audiences, Netflix subcontracts the creation of closed captioning to vendors, instructing them to follow the Timed Text Style Guide (TTSG), which it makes freely available online. My study examines how closed captions for Netflix Original content endeavour to make audiovisual content accessible to deaf audiences, and I demonstrate how the platonic ideal of "equal access" is out of reach due to the limitations of timed text. The objective of my study is to highlight and critique the transformations of meaning that occur when captions translate sound and spoken dialogue into timed text. Drawing on D'Acci's circuit model of media studies (2004) my thesis links the sociohistorical conditions from which captioning techniques and technologies were developed, the conditions of caption production, and the way in which the needs of deaf audiences are articulated in the TTSG. I explore how these three forces affect the content of closed captions. To this end, I engage in a close reading of the TTSG and a selection of closed captions for Netflix Original series and films, borrowing from Berman's (2000) theories regarding the deforming tendencies of translation to describe the changes that result from the intralingual and intersemiotic translation involved in captioning (Jakobson, 2004). My study is informed and inspired by my personal experience as a professional captioner.

To solve real-world problems, machines are required to perceive multiple modalities and fuse the information from them. This thesis studies learning to understand and fuse multimodal information. Existing approaches follow a three-stage learning paradigm. The first stage is to train models for each modality. This process for video understanding models is usually based on supervised training, which is not scalable. Moreover, these modality-specific models are updated rather frequently nowadays with improving single-modality perception abilities. The second stage is crossmodal pretraining, which trains a model to align and fuse multiple modalities based on paired multimodal data, such as video-caption pairs. This process is resource-consuming and expensive. The third stage is to further fine-tune or prompt the resulting model from the second stage towards certain downstream tasks. The key bottleneck of conventional methods lies in the continuous feature representation used for non-textual modalities, which is usually costly to align and fuse with text. In this thesis, we investigate the representation and the fusion based on textual concepts. We propose to map non-textual modalities to textual concepts and then fuse these textual concepts using text models. We systematically study various specific methods of mapping and different architectures for fusion. The proposed methods include an end-to-end video-based text generation model with differentiable tokenization for video and audio concepts, a contrastive-model-based architecture with zero-shot concept extractor, a deep concept injection algorithm enabling language models to solve multimodal tasks without any training, and a distant supervision framework learning concepts in a long temporal span. With our concept representation, we empirically demonstrate that without several orders of magnitude more cost for the crossmodal pretraining stage, our models are able to achieve competitive or even superior performance on downstream tasks such as video question answering, video captioning, text-video retrieval, and audio-video dialogue. We also examine the possible limitations of concept representations such as when the text quality of a dataset is poor. We believe we show a potential path towards upgradable multimodal intelligence, whose components can be easily updated towards new models or new modalities of data.

邁入數位電視紀元乃是全球之趨，而其對於增進身障者獲取各類資訊的「媒介近用權」具有莫大助益，其中針對聽障社群接取內容最重要的近用需求──「字幕」和「手語」服務，在數位科技匯流發展下，皆可以「隱藏式」之方式供應，同時造福聽障和非聽障之傳播權益，以及減輕廣電業者相關技術的支付成本。因此，近用服務的提供從過去的消極被動轉向現今的積極樂觀。而外國先進國家大多皆以公共廣電媒體之設立價值與目標，作為該國近用服務推動的核心主體，希望藉由數位電視的技術研發，達成更多聽障輔助應用之需求和供應滿足，協助其順利進入數位包容社會。故本研究以探詢國外落實近用服務情形，以做為我國公共廣電服務借力使力之參考，期許對我國聽障社群在傳播權益上產生影響。 　　 　　研究發現，英國、歐盟針對聽障社群的媒體近用落實，無論在法規的制定、實務的推行以及技術的研發等各層面皆有所重視，認為數位電視平台的時代，應協助聽障融入數位包容社會，並設法增進其傳播權益，以彰顯聽障與一般大眾之平權的公民地位；而在我國公視部份，其營運目標始終視英國BBC為效法對象，希冀在內、外資源充份下能達至同BBC供應近用服務之標準水平。然而在多種因素交織下，現階段公視對於聽障媒體近用服務的提供，則依舊保持類比電視時代之作為，不過，經本研究與其互動後了解，公視未來可能朝向增加其他近用服務項目發展，期望數位電視真正來臨時，其能化過往被動態度轉向積極進取：公視目前在電視平台持續兩個「手語專門」節目的製播，並預計規劃將手語服務擴大至「運動」類型節目，以符合聽障收視的期待；至於字幕服務，在已完備的基礎上，試圖朝向「表情字幕」與「即時字幕」發展；另外，於2011年HiHD數位頻道將推出「隱藏式字幕」功能。在網路平台方面，公視服務仍然延伸至電視頻道的節目宣傳與相關資訊供給為主，對於加強聽障的網路近用權益，例如「無障礙網頁空間」以及「近用小組」，認為必然有公共義務介入加以落實，但礙於目前並無相關資源規劃與投入，因此要實際推行仍有很大的進步空間。 / The main purpose of this study is to discover the practice of the right of media access in foreign countries, in order to provide reference to Taiwan’s Public Service Broadcasting (PSB) and to make progress on communication interests for hearing impaired community. “Caption” and “Sign Language” are the most important tools for hearing impaired people to gather all kinds of information and fulfill the necessity of access service. Under the digital convergence, these tools can be provided in special ways, which makes the hearing impaired people and the hearing people share the benefits simultaneously and the cost-down effect of broadcasting industry. We know that most developed countries positioned their access service project by referring to nation’s PSB. They believed the new era of digital TV is a solution to attend the balance between demand and supply of hearing impaired aid applications. While the provision of access services is getting more active and optimistic, the digital inclusion is much close to us. The study shows, British and Europe Union think they should assist hearing impaired people to be involved in e-Inclusion society and highlight equally citizen status by enhancing the rights of hearing impaired people. All the aspects such as regulation enactments, practical implementations and technique developments has been considering all the time on the stage of digital TV platform. Just like the BBC in British area, Public Television Service (PTS) in Taiwan is taking BBC as a benchmark to achieve the access services standard in condition of sufficient resources. However, changing the status quo is not so easy for inextricably interwoven reasons. PTS still works in an analog status. In spite of the circumstances haven't changed much till now, there are much more possibilities in the future. The study discovered some new progressive plans are possible for PTS’s access services in digital journey: PTS will continue to provide two programs which are sign-presented, and moreover, sign language service is going to show up in sports genre; As to caption services, PTS is working on facial expression caption and real-time caption provision; HiHD would have closed caption function in 2011. In the case of Internet platform, PTS is focused on propaganda and related information of TV programs. Barrier-free web space and access group are considered necessary for strengthening hearing impaired people’s Internet access rights and interests, but with insufficient resources planning and investment to put into realization. We can see there is still so much to do if we believe we have the affirmative obligations.