Thesis (Ph. D.)--University of Tampere, 1999. / Description based on contents viewed Apr. 19, 2007; title from title screen. Includes bibliographical references (p. 58-67).

abstract / Electrical and Electronic Engineering / Doctoral / Doctor of Philosophy

Multimodal systems can represent and manipulate semantics from different human communication modalities at different levels of abstraction, in which multimodal integration is required to integrate the semantics from two or more modalities and generate an interpretable output for further processing. In this work, we develop a framework pertaining to automatic cross-modality semantic integration of multimodal user interactions using speech and pen gestures. It begins by generating partial interpretations for each input event as a ranked list of hypothesized semantics. We devise a cross-modality semantic integration procedure to align the pair of hypothesis lists between every speech input event and every pen input event in a multimodal expression. This is achieved by the Viterbi alignment that enforces the temporal ordering and semantic compatibility constraints of aligned events. The alignment enables generation of a unimodal paraphrase that is semantically equivalent to the original multimodal expression. Our experiments are based on a multimodal corpus in the navigation domain. Application of the integration procedure to manual transcripts shows that correct unimodal paraphrases are generated for around 96% of the multimodal inquiries in the test set. However, if we replace this with automatic speech and pen recognition transcripts, the performance drops to around 53% of the test set. In order to address this issue, we devised the hypothesis rescoring procedure that evaluates all candidates of cross-modality integration derived from multiple recognition hypotheses from each modality. The rescoring function incorporates the integration score, N-best purity of recognized spoken locative references (SLRs), as well as distances between coordinates of recognized pen gestures and their interpreted icons on the map. Application of cross-modality hypothesis rescoring improved the performance to generate correct unimodal paraphrases for over 72% of the multimodal inquiries of the test set. / We have also performed a latent semantic modeling (LSM) for interpreting multimodal user input consisting of speech and pen gestures. Each modality of a multimodal input carries semantics related to a domain-specific task goal (TG). Each input is annotated manually with a TG based on the semantics. Multimodal input usually has a simpler syntactic structure and different order of semantic constituents from unimodal input. Therefore, we proposed to use LSM to derive the latent semantics from the multimodal inputs. In order to achieve this, we characterized the cross-modal integration pattern as 3-tuple multimodal terms taking into account SLR, pen gesture type and their temporal relation. The correlation term matrix is then decomposed using singular value decomposition (SVD) to derive the latent semantics automatically. TG inference on disjoint test set based on the latent semantics achieves accurate performance for 99% of the multimodal inquiries. / Hui, Pui Yu. / Adviser: Helen Meng. / Source: Dissertation Abstracts International, Volume: 73-02, Section: B, page: . / Thesis (Ph.D.)--Chinese University of Hong Kong, 2010. / Includes bibliographical references (leaves 294-306). / Electronic reproduction. Hong Kong : Chinese University of Hong Kong, [2012] System requirements: Adobe Acrobat Reader. Available via World Wide Web. / Electronic reproduction. [Ann Arbor, MI] : ProQuest Information and Learning, [201-] System requirements: Adobe Acrobat Reader. Available via World Wide Web. / Abstract also in Chinese.

Thesis--Auburn University, 2009. / Abstract. Includes bibliographic references (p.22-23).

Thesis (Ph. D.)--Auburn University. / Abstract. Includes bibliographical references (p. 86-89).

L'haptique joue un rôle majeur dans l'interaction avec des environnements virtuels, avec de nombreuses applications en entraînement virtuel, en prototypage et en assistance de téléopérations. En particulier, les objets déformables représentent un défi pour la simulation à cause de leur comportement intrinsèquement complexe. À cause des besoins particuliers en terme de puissance liés à l'interaction haptique, il est en général nécessaire de faire un compromis entre efficacité et précision, et tirer le meilleur parti de ce compromis reste un défi majeur. Les objectifs de ce doctorat sont premièrement d'améliorer l'interaction haptique avec des objets virtuels déformables au comportement complexe, et enfin d'étudier en quoi la perception peut nous aider dans cette tâche.Dans cette thèse, nous proposons dans un premier temps un modèle pour la simulation physique d'objets hétérogènes déformables. Plus précisément, nous nous intéressons au problème de la multirésolution géométrique pour les objets hétérogènes, en nous concentrant sur la représentation de l'hétérogénéité à basse résolution des objets simulés. La contribution consiste en une méthode d'attribution de l'élasticité pour la basse résolution de l'objet, et une évaluation de ce changement de géométrie sur la perception haptique.Nous nous intéressons ensuite à une autre classe de comportements complexes, les changements topologiques, en proposant un pipeline de simulation pour la déchirure haptique bimanuelle d'objets déformables fins. Cette contribution se concentre sur deux aspects essentiels à la simulation efficace de déchirure, à savoir la détection de collision pour les objets surfaciques, et la simulation physique efficace de déchirure. La simulation est particulièrement optimisée pour la propagation de déchirure.Le dernier aspect couvert dans cette thèse est l'influence de l'environnement sur la perception haptique de raideur, et plus particulièrement les environnements de Réalité Augmentée (RA). Comment perçoit-on les objets en RA par rapport à la Réalité Virtuelle (RV)? Est-ce que nous interagissons de la même manière dans ces deux environnements? Pour répondre à ces questions, nous avons mené une expérience pour comparer la perception haptique de raideur d'un piston virtuel entouré dans un premier cas d'objets de la vie quotidienne en RA, et du même piston entouré par une reproduction virtuelle de cet environnement réel en RV.Ces contributions ouvrent de nouvelles perspectives pour l'interaction haptique avec des environnements virtuels, depuis la simulation efficace et fidèle d'objets déformables au comportement complexe à une meilleure compréhension de la perception haptique et des stratégies d'interaction. / Haptics is a key part for the interaction with physically-based environments, with many applications in virtual training, prototyping and teleoperations assistance. In particular, deformable objects are challenging, due to the complexity oftheir behavior. Due to the specific need in performance associated to haptic interaction, a trade-off is usually necessarybetween accuracy and efficiency, and taking the best of this trade-off is a major challenge. The objectives of this PhD are to improve haptic rendering with physically-based deformable objects that exhibit complex behavior, and study how perception can be used to achieve this goal.In this PhD, we first propose a model for the physically-based simulation of complex heterogeneous deformable objects. More specifically, we address the issue of geometric multiresolution for deformable heterogeneous objects, with a major focus on the heterogeneity representation at the coarse resolution of the simulated objects. The contribution consists in a method for elasticity attribution at coarser resolution of the object, and an evaluation of the geometric coarsening on the haptic perception.We then focus on another class of complex objects behavior, topology changes, by proposing a simulation pipeline forbimanual haptic tearing of thin deformable surfaces. This contribution mainly focuses on two main aspects for an efficientsimulation of tearing, namely collision detection for thin objects and efficient physically-based simulation of tearing phenomena. The simulation is especially optimized for tear propagation.The last aspect that is covered by this PhD is the influence of the environment over haptic perception of stiffness, and more specifically of Augmented Reality (AR) environments. How are objects perceived in AR compared to Virtual Reality (VR)? Do we interact the same way on these two environments? In order to assess these questions, we conducted an experiment aiming at comparing the haptic stiffness perception of a piston surrounded by everyday life objects in AR and of the same piston surrounded by a virtual replica of the real environment in VR.These contributions open new perspectives for haptic interaction with virtual environments, from the efficient yet faithful simulation of complex deformable objects behavior to a better understanding of haptic perception and interaction strategies.

La communication multimodale, qui est primordiale dans les relations interpersonnelles, reste encore très limitée dans les interfaces homme-machine actuelles. Parmi les différentes modalités qui ont été adoptées par les recherches en interaction homme-machine, la modalité posturale a été moins explorée que d’autres modalités comme la parole ou les expressions faciales. Les postures corporelles sont pourtant indispensables pour interpréter et situer l’interaction entre deux personnes, que ce soit en termes de contexte spatial ou de contexte social. Les briques de base que sont les dispositifs et les modalités d’interaction (par exemple la kinect ou les avatars) sont pourtant disponibles. Il manque cependant des modèles informatiques reliant ces médias et modalités aux fonctions de communication pertinentes dans les interactions interpersonnelles comme celles liées à l’espace ou aux émotions.L’objectif de cette thèse est de concevoir un premier modèle informatique permettant d’exploiter les postures dans les interactions homme-machine. Cela soulève plusieurs questions de recherche : comment représenter symboliquement des postures exprimées durant des interactions interpersonnelles ? Comment spécifier les comportements posturaux de personnages virtuels ? Quelles doivent être les caractéristiques d’un modèle d’interaction corporel permettant des interactions entre un personnage virtuel et un utilisateur dans différents contextes physiques ou sociaux ?L’approche proposée consiste dans un premier temps à prendre comme point de départ des corpus vidéo filmés dans différentes situations. Nous avons défini un schéma de codage pour annoter manuellement les informations posturales à différents niveaux d’abstraction et pour les différentes parties du corps. Ces représentations symboliques ont été exploitées pour effectuer des analyses des relations spatiales et temporelles entre les postures exprimées par deux interlocuteurs.Ces représentations symboliques de postures ont été utilisées dans un deuxième temps pour simuler des expressions corporelles de personnages virtuels. Nous nous sommes intéressés à un composant des émotions particulièrement pertinent pour les études sur les postures : les tendances à l’action. Des animations impliquant deux personnages virtuels ont été ainsi conçues puis évaluées perceptivement.Enfin, dans un troisième temps, les expressions corporelles d’un personnage virtuel ont été conçues dans une application mixte faisant intervenir un personnage virtuel et un utilisateur dans un cadre d’interaction ambiante. Les postures et gestes du personnage virtuel ont été utilisées pour aider l’utilisateur à localiser des objets du monde réel.Cette thèse ouvre des perspectives sur des études plus spécifiques de l’interaction corporelle nécessitant par exemple des annotations automatiques via des dispositifs de capture de mouvement ou la prise en compte des différences individuelles dans l’expression posturale. / Multimodal communication is key to smooth interactions between people. However, multimodality remains limited in current human-computer interfaces. For example, posture is less explored than other modalities, such as speech and facial expressions. The postural expressions of others have a huge impact on how we situate and interpret an interaction. Devices and interfaces for representing full-body interaction are available (e.g., Kinect and full-body avatars), but systems still lack computational models relating these modalities to spatial and emotional communicative functions.The goal of this thesis is to lay the foundation for computational models that enable better use of posture in human-computer interaction. This necessitates addressing several research questions: How can we symbolically represent postures used in interpersonal communication? How can these representations inform the design of virtual characters' postural expressions? What are the requirements of a model of postural interaction for application to interactive virtual characters? How can this model be applied in different spatial and social contexts?In our approach, we start with the manual annotation of video corpora featuring postural expressions. We define a coding scheme for the manual annotation of posture at several levels of abstraction and for different body parts. These representations were used for analyzing the spatial and temporal relations between postures displayed by two human interlocutors during spontaneous conversations.Next, representations were used to inform the design of postural expressions displayed by virtual characters. For studying postural expressions, we selected one promising, relevant component of emotions: the action tendency. Animations were designed featuring action tendencies in a female character. These animations were used as a social context in perception tests.Finally, postural expressions were designed for a virtual character used in an ambient interaction system. These postural and spatial behaviors were used to help users locate real objects in an intelligent room (iRoom). The impact of these bodily expressions on the user¡¯s performance, subjective perception and behavior was evaluated in a user studyFurther studies of bodily interaction are called for involving, for example, motion-capture techniques, integration with other spatial modalities such as gaze, and consideration of individual differences in bodily interaction.

Scene understanding attempts to produce a textual description of visible and latent concepts in an image to describe the real meaning of the scene. Concepts are either objects, events or relations depicted in an image. To recognize concepts, the decision of object detection algorithm must be further enhanced from visual similarity to semantical compatibility. Semantically relevant concepts convey the most consistent meaning of the scene. Object detectors analyze visual properties (e.g., pixel intensities, texture, color gradient) of sub-regions of an image to identify objects. The initially assigned objects names must be further examined to ensure they are compatible with each other and the scene. By enforcing inter-object dependencies (e.g., co-occurrence, spatial and semantical priors) and object to scene constraints as background information, a concept classifier predicts the most semantically consistent set of names for discovered objects. The additional background information that describes concepts is called context. In this dissertation, a framework for building context-based concept detection is presented that uses a combination of multiple contextual relationships to refine the result of underlying feature-based object detectors to produce most semantically compatible concepts. In addition to the lack of ability to capture semantical dependencies, object detectors suffer from high dimensionality of feature space that impairs them. Variances in the image (i.e., quality, pose, articulation, illumination, and occlusion) can also result in low-quality visual features that impact the accuracy of detected concepts. The object detectors used to build context-based framework experiments in this study are based on the state-of-the-art generative and discriminative graphical models. The relationships between model variables can be easily described using graphical models and the dependencies and precisely characterized using these representations. The generative context-based implementations are extensions of Latent Dirichlet Allocation, a leading topic modeling approach that is very effective in reduction of the dimensionality of the data. The discriminative contextbased approach extends Conditional Random Fields which allows efficient and precise construction of model by specifying and including only cases that are related and influence it. The dataset used for training and evaluation is MIT SUN397. The result of the experiments shows overall 15% increase in accuracy in annotation and 31% improvement in semantical saliency of the annotated concepts. / Includes bibliography. / Dissertation (Ph.D.)--Florida Atlantic University, 2016. / FAU Electronic Theses and Dissertations Collection

Multimodal communication is key to smooth interactions between people. However, multimodality remains limited in current human-computer interfaces. For example, posture is less explored than other modalities, such as speech and facial expressions. The postural expressions of others have a huge impact on how we situate and interpret an interaction. Devices and interfaces for representing full-body interaction are available (e.g., Kinect and full-body avatars), but systems still lack computational models relating these modalities to spatial and emotional communicative functions.The goal of this thesis is to lay the foundation for computational models that enable better use of posture in human-computer interaction. This necessitates addressing several research questions: How can we symbolically represent postures used in interpersonal communication? How can these representations inform the design of virtual characters' postural expressions? What are the requirements of a model of postural interaction for application to interactive virtual characters? How can this model be applied in different spatial and social contexts?In our approach, we start with the manual annotation of video corpora featuring postural expressions. We define a coding scheme for the manual annotation of posture at several levels of abstraction and for different body parts. These representations were used for analyzing the spatial and temporal relations between postures displayed by two human interlocutors during spontaneous conversations.Next, representations were used to inform the design of postural expressions displayed by virtual characters. For studying postural expressions, we selected one promising, relevant component of emotions: the action tendency. Animations were designed featuring action tendencies in a female character. These animations were used as a social context in perception tests.Finally, postural expressions were designed for a virtual character used in an ambient interaction system. These postural and spatial behaviors were used to help users locate real objects in an intelligent room (iRoom). The impact of these bodily expressions on the user¡¯s performance, subjective perception and behavior was evaluated in a user studyFurther studies of bodily interaction are called for involving, for example, motion-capture techniques, integration with other spatial modalities such as gaze, and consideration of individual differences in bodily interaction.

In a large number of electronic devices, users interact with the system by navigating through various menus. Auditory menus can complement or even replace visual menus, so research on auditory menus has recently increased with mobile devices as well as desktop computers. Despite the potential importance of auditory displays on touch screen devices, little research has been attempted to enhance the effectiveness of auditory menus for those devices. In the present study, I investigated how advanced auditory cues enhance auditory menu navigation on a touch screen smartphone, especially for new input gestures such as tapping, wheeling, and flicking methods for navigating a one-dimensional menu. Moreover, I examined if advanced auditory cues improve user experience, not only for visuals-off situations, but also for visuals-on contexts. To this end, I used a novel auditory menu enhancement called a "spindex" (i.e., speech index), in which brief audio cues inform the users of where they are in a long menu. In this study, each item in a menu was preceded by a sound based on the item's initial letter. One hundred and twenty two undergraduates navigated through an alphabetized list of 150 song titles. The study was a split-plot design with manipulated auditory cue type (text-to-speech (TTS) alone vs. TTS plus spindex), visual mode (on vs. off), and input gesture style (tapping, wheeling, and flicking). Target search time and subjective workload for the TTS + spindex were lower than those of the TTS alone in all input gesture types regardless of visual type. Also, on subjective ratings scales, participants rated the TTS + spindex condition higher than the plain TTS on being 'effective' and 'functionally helpful'. The interaction between input methods and output modes (i.e., auditory cue types) and its effects on navigation behaviors was also analyzed based on the two-stage navigation strategy model used in auditory menus. Results were discussed in analogy with visual search theory and in terms of practical applications of spindex cues.