Los sistemas de reconocimiento biométrico utilizan ciertas características humanas como la voz, los rasgos faciales, la huella dactilar, el iris o la geometría de la mano para identificar a un individuo o verificar su identidad. Dichos sistemas se han desarrollado de forma individual para cada una de estas modalidades biométricas hasta llegar a obtener unos niveles notables de rendimiento. Los sistemas biométricos multimodales combinan diversas modalidades en un sistema de reconocimiento único. La fusión multimodal permite mejorar los resultados obtenidos por una sola característica biométrica y hacen el sistema más robusto a ruidos e interferencias y más resistente a posibles ataques. La fusión se puede realizar a nivel de las señales adquiridas por los distintos sensores, de los parámetros obtenidos para cada modalidad, de las puntuaciones proporcionadas por expertos unimodales o de la decisión tomada por dichos expertos. En la fusión a nivel de parámetros o puntuaciones es necesario homogeneizar las características provenientes de las diferentes modalidades biométricas de manera previa al proceso de fusión. A este proceso de homogeneización se le denomina normalización y se ha demostrado determinante en la obtención de buenos resultados de reconocimiento en los sistemas multimodales. En esta tesis, se presentan diversos métodos de normalización que modifican la estadística de parámetros o puntuaciones. En primer lugar, se propone la normalización de la media y la varianza de las puntuaciones unimodales por medio de transformaciones afines que tienen en cuenta las estadísticas separadas de las puntuaciones de clientes e impostores. En este ámbito se presenta la normalización conjunta de medias, que iguala las medias de las puntuaciones de clientes e impostores para todas las modalidades biométricas. También se han propuesto técnicas que minimizan la suma de las varianzas de las puntuaciones multimodales de clientes e impostores. Estas técnicas han obtenido buenos resultados en un sistema bimodal de fusión de puntuaciones de espectro de voz e imágenes faciales y se ha demostrado que una reducción de las varianzas multimodales puede comportar un mejor resultado de reconocimiento. Por otro lado, se ha utilizado la ecualización de histograma, un método ampliamente utilizado en el tratamiento de imágenes, como técnica de normalización. Para ello, se han ecualizado los histogramas de las características unimodales sobre diversas funciones de referencia. En primer lugar, se ha utilizado el histograma de las puntuaciones de una de las modalidades biométricas como referencia en el proceso de ecualización. Esta técnica se ha mostrado especialmente efectiva al combinarla con métodos de fusión basados en la ponderación de las puntuaciones unimodales. En una segunda aproximación, se han ecualizado las características biométricas a funciones previamente establecidas, en concreto, a una gaussiana y a una doble gaussiana. La ecualización a gaussiana ha obtenido buenos resultados como normalización en sistemas de fusión de parámetros. La ecualización de doble gaussiana se ha diseñado específicamente para la normalización de puntuaciones. Las dos gaussianas representan los lóbulos de las puntuaciones de clientes e impostores que se pueden observar en los histogramas unimodales. Se han probado diferentes variantes para determinar las varianzas de dichas gaussianas. Las técnicas de normalización estadística presentadas en esta tesis se han probado utilizando diferentes estrategias y técnicas para la fusión, tanto para bases de datos quiméricas como para una base de datos multimodal. Además, la fusión se ha realizado a diferentes niveles, en concreto, a nivel de puntuaciones para diferentes escenarios multimodales incluyendo características de espectro voz, prosodia y caras, y a los niveles de parámetros, puntuaciones y decisión en el entorno del proyecto Agatha.

A key requirement for developing any innovative system in a computing environment is to integrate a sufficiently friendly interface with the average end user. Accurate design of such a user-centered interface, however, means more than just the ergonomics of the panels and displays. It also requires that designers precisely define what information to use and how, where, and when to use it. Recent advances in user-centered design of computing systems have suggested that multimodal integration can provide different types and levels of intelligence to the user interface. The work of this thesis aims at improving speech recognition-based interfaces by making use of the visual modality conveyed by the movements of the lips. Designing a good visual front end is a major part of this framework. For this purpose, this work derives the optical flow fields for consecutive frames of people speaking. Independent Component Analysis (ICA) is then used to derive basis flow fields. The coefficients of these basis fields comprise the visual features of interest. It is shown that using ICA on optical flow fields yields better classification results than the traditional approaches based on Principal Component Analysis (PCA). In fact, ICA can capture higher order statistics that are needed to understand the motion of the mouth. This is due to the fact that lips movement is complex in its nature, as it involves large image velocities, self occlusion (due to the appearance and disappearance of the teeth) and a lot of non-rigidity. Another issue that is of great interest to audio-visual speech recognition systems designers is the integration (fusion) of the audio and visual information into an automatic speech recognizer. For this purpose, a reliability-driven sensor fusion scheme is developed. A statistical approach is developed to account for the dynamic changes in reliability. This is done in two steps. The first step derives suitable statistical reliability measures for the individual information streams. These measures are based on the dispersion of the N-best hypotheses of the individual stream classifiers. The second step finds an optimal mapping between the reliability measures and the stream weights that maximizes the conditional likelihood. For this purpose, genetic algorithms are used. The addressed issues are challenging problems and are substantial for developing an audio-visual speech recognition framework that can maximize the information gather about the words uttered and minimize the impact of noise.

A key requirement for developing any innovative system in a computing environment is to integrate a sufficiently friendly interface with the average end user. Accurate design of such a user-centered interface, however, means more than just the ergonomics of the panels and displays. It also requires that designers precisely define what information to use and how, where, and when to use it. Recent advances in user-centered design of computing systems have suggested that multimodal integration can provide different types and levels of intelligence to the user interface. The work of this thesis aims at improving speech recognition-based interfaces by making use of the visual modality conveyed by the movements of the lips. Designing a good visual front end is a major part of this framework. For this purpose, this work derives the optical flow fields for consecutive frames of people speaking. Independent Component Analysis (ICA) is then used to derive basis flow fields. The coefficients of these basis fields comprise the visual features of interest. It is shown that using ICA on optical flow fields yields better classification results than the traditional approaches based on Principal Component Analysis (PCA). In fact, ICA can capture higher order statistics that are needed to understand the motion of the mouth. This is due to the fact that lips movement is complex in its nature, as it involves large image velocities, self occlusion (due to the appearance and disappearance of the teeth) and a lot of non-rigidity. Another issue that is of great interest to audio-visual speech recognition systems designers is the integration (fusion) of the audio and visual information into an automatic speech recognizer. For this purpose, a reliability-driven sensor fusion scheme is developed. A statistical approach is developed to account for the dynamic changes in reliability. This is done in two steps. The first step derives suitable statistical reliability measures for the individual information streams. These measures are based on the dispersion of the N-best hypotheses of the individual stream classifiers. The second step finds an optimal mapping between the reliability measures and the stream weights that maximizes the conditional likelihood. For this purpose, genetic algorithms are used. The addressed issues are challenging problems and are substantial for developing an audio-visual speech recognition framework that can maximize the information gather about the words uttered and minimize the impact of noise.

Kuso videos as an emergent Internet genre have gained its popularity among a great number of online participants, especially via the most popular video-sharing website, Youtube. The present study explores the meaning-making mechanism that operates in the production of kuso videos, which are worthy of note inasmuch as they have been more and more widely used in response to issues arising in Taiwanese society. Different from previous research on monomodal representation of kuso (e.g. written, graphic kuso), the present research takes the previous studies as a point of departure and will analyze a multimodal and multimedial kuso video uploaded onto Youtube. The analysis will center around its performance, power, and meaning potentials and seek to answer the following research questions. (1) From a multimodal social-semiotic perspective, how are the meaning potentials of kuso videos being performed in the process of meaning-making? (2) In multimodal kuso videos, what accounts as a mode and what are the affordances of a specific mode? Or, is it necessary to define mode as a separate unit which creates meaning potentials in the whole multimodal production? (3) In social-semiotic production of kuso videos, who is in control in the process of meaning-making? In other words, in the meaning-making process, how is the identity of kuso video being contextualized and being negotiated with other participants? (4) How are decontextualization, recontextualization, and identity contextualization interconnected to each other in the production and the distribution of kuso videos? To analyze the meaning-making process and meaning potentials of the target kuso video, multisemiotic models (Halliday, 1978; Lim, 2002) as well as Bakhtin¡¦s (1981; 1986) genre theory are applied as the basis of the analytical framework while ¡§projective act of identity¡¨ (Coupland, 2007) is used to account for the how identity is contextualized in the video. In addition, the present research proposes the Functional Multisemiotic Model to illustrate with examples how layers of remix operate in the processes of decontextualization, recontextualization, and identity contextualization and to elucidate what semiotic functions they serve at three levels (i.e. ideational, textual, and interpersonal levels). On the whole, this research may shed light on the intricate meaning-making process in multimodal kuso analysis.

The aim of the study is to investigate how a teacher uses multimodal resources with an interactive whiteboard while teaching reading and writing. The aim is also to investigate what view the teacher has on using an interactive whiteboard while teaching reading and writing. Video-recorded observations and audio-recorded interviews are the methods used to collect data. The analysis was performed from a multimodal perspective and is also inspired by interaction analysis. The result of the analysis showed that the teacher uses several multimodal sign systems in combination with those found on the interactive whiteboard. Another finding is that the attitude of the principal and the school to interactive whiteboards is not always crucial for the extent to which the whiteboard is used. It is instead the time available and the teacher’s attitude to using the interactive whiteboard that enables its use.

This qualitative case study focuses on the efforts of the education staff at the Meadows Museum of Art as they planned and implemented INsights and OUTlooks – an inclusive educational program, meaning that it was designed for sighted and non-sighted visitors. Although this is an inclusive program, the study concentrates mostly on how it was designed to make art accessible to visitors with visual impairments. The researcher interviewed the staff in charge of leading the program, observed two program sessions, and attended staff meetings related to INsights and OUTlooks to gain a better understanding of how it functions. Current literature (Andrus, 1999; McGinnis, 2007) within the field of art education suggests that staff at art museums should adopt inclusive practices as they design programs and exhibitions. This study explores the process of creating such a program, the tools and approaches utilized to make art accessible to visitors with visual impairments, and the benefits of being inclusive. Research studies such as this one add to the existing but limited literature regarding inclusive programming in art museums. / text

The purpose of this qualitative research was to better understand the multilingual and multimodal composition resources appropriated by students during a study of Latino children’s picturebooks within a predominantly Latino, third grade classroom. A conceptual framework guided by socio-cultural perspectives, a social semiotic theory of communication, and Composition 2.0 studies was employed to investigate the ways in which students remixed multilingual and multimodal composition resources and manifested identities in texts. This research was guided by both design-based and case study methods and drew upon constant-comparative, discourse, and visual discourse analytic methods to examine the data. Analysis was also located in the literature on identity and texts so as to better understand the socio-cultural histories and identities attached to the children's picturebooks. Data collection was focused on both the multilingual and multimodal resources students appropriated to compose and the ways students orchestrated those resources during the classroom picturebook study. Analysis was structured by two interrelated strands. The first strand explores more broadly the composition resources in use during the classroom picturebook study, and the second analyzes explicitly the ways two focal students remixed composition resources within their picturebook productions and sedimented identities in texts. Three findings generated from the two related strands of analysis provided insights into the potential of a picturebook study as a viable multilingual and multimodal composition curriculum. First, in the context of the teacher and researcher co-designed curriculum and instruction, students appropriated literary, illustrated, material, and picturebook form resources from Latino children’s picturebooks in diverse ways. Second, in the act of picturebook making, students invoked other socio-cultural texts as mentors and remixed composition resources from diverse sources to craft their own picturebooks. Finally, students manifested aspects of their identities within the material worlds and languages reflected within their picturebooks. Together, these findings situate picturebook study and picturebook making as creative and intellectual acts for students. Moreover, this study features Latino children’s picturebooks as culturally responsive mentor texts. Several pedagogical implications related to composition instruction for young writers and diverse population are also discussed. / text

abstract / Electrical and Electronic Engineering / Doctoral / Doctor of Philosophy

A number of software that supports multiple interactions such as synergistic use of speech and gesture is increasing. With power and versatility of multimodal interfaces also comes and new issues like increased complexity of software design. These issues should be considered in early design and testing stages of software development process. In this work we are researching a complex use of users multiple interactions with software. We will represent a method for the specification of multimodal user interfaces offered by other authors. This method is based on CARE properties. The CARE properties provide a formal framework for reasoning design of multimodal systems. Multimodal systems are using fusion mechanisms to combine different modalities. Later in this work we will take a closer look at testing process of multimodal applications. A new approach for testing modalities fusion mechanism in multimodal applications is offered. This approach will be experimentally demonstrated, evaluated and compared to alternative method.

No description available.