Turn-taking is a fundamental process that governs social interaction. When humans interact, they naturally take initiative and relinquish control to each other using verbal and nonverbal behavior in a coordinated manner. In contrast, existing approaches for controlling a robot's social behavior do not explicitly model turn-taking, resulting in interaction breakdowns that confuse or frustrate the human and detract from the dyad's cooperative goals. They also lack generality, relying on scripted behavior control that must be designed for each new domain. This thesis seeks to enable robots to cooperate fluently with humans by automatically controlling the timing of multimodal turn-taking. Based on our empirical studies of interaction phenomena, we develop a computational turn-taking model that accounts for multimodal information flow and resource usage in interaction. This model is implemented within a novel behavior generation architecture called CADENCE, the Control Architecture for the Dynamics of Embodied Natural Coordination and Engagement, that controls a robot's speech, gesture, gaze, and manipulation. CADENCE controls turn-taking using a timed Petri net (TPN) representation that integrates resource exchange, interruptible modality execution, and modeling of the human user. We demonstrate progressive developments of CADENCE through multiple domains of autonomous interaction encompassing situated dialogue and collaborative manipulation. We also iteratively evaluate improvements in the system using quantitative metrics of task success, fluency, and balance of control.

Todays technology provides the ability for people to interact with devices at all time, but it also makes them gaze down and loose touch with their surrounding while doing so.  This project explored alternatives to the current way of providing guidance while walking by designing in the context, rather than design for the context. This led to several iterations of prototypes that were tested with people in the context. The findings are showcased in a form of a multi modal guidance system called UP, that provides reassurance in all steps of the way without having to look down at the screen.

Syftet med studien är att undersöka fiollärares syn på och användning av stråkteknik i sin undervisning. Detta görs med följande forskningsfrågor: Hur representerar fiollärare sin syn på stråkteknik? och Hur designar fiollärare sin undervisning i stråkteknik? Studien är kvalitativ och data har samlats in genom semistrukturerade intervjuer med fyra olika fiollärare på musik- och kulturskolor. Studiens teoretiska utgångspunkt utgörs av socialsemiotiskt perspektiv. Intervjuerna ljudinspelades och även videodokumenterades för att sedan transkriberas och analyseras med hjälp av analysverktyg från det valda teoretiska ramverket. I studiens resultat beskrivs fiollärarnas syn på stråkteknik samt hur de designar sin undervisning i stråkteknik. De teman som presenteras är stråkteknik som tonbildning, stråkteknik som hållning, stråkteknik som instrumentteknik, materiella resurser i undervisningen samtkroppsliga och klingande resurser i undervisningen.I resultatet framkommer många skillnader men även likheter i de intervjuade fiollärarnas utsagor, men alla belyser stråkteknikens betydelse i undervisningen och dess funktion. I studiens avslutande kapitel diskuteras resultatet i förhållande till tidigare litteratur och forskning samt till studiens socialsemiotiska perspektiv. De teman som presenteras är undervisning ur ett socialsemiotiskt perspektiv samt stråkteknik kopplat till stråkhållning och tonbildning. Det tas även upp om metoddiskussion, arbetets betydelse samt fortsatt forskning. / The purpose of the study is to investigate violin teachers’ view of and use of bowing technique in their teaching. The study´s research questions concern how do violin teachers represent their view of bow technique? And how do violin teachers design their teaching in bow technique? The study is qualitative, and data has been collected through semi structured interviews with four different violin teachers at music and cultural schools. The theoretical starting point of the study is social semiotic perspective. The interviews were recorded by audio and video and were then transcribe and analyzed. The analysis was conducted with tools from the chosen theoretical framework. In the study's results, the violin teachers' view of bow technique is described and how they design their teaching in bow technique. The themes presented are string techniques such as tone formation, bow technique such as posture, bow technique such as instrument technology, material resources in teachingandbodily and sounding resources in teaching. The result shows many differences but also similarities in the interviewed violin teachers' statements, but all highlight the significance of the bow technique in teaching and its function. In the final chapter of the study, the result is discussed in relation to previous literature and research, and to the social semiotic perspective of the study. The themes presented are teaching from a social semiotic perspectiveand bow technique linked to posture and tone formation. The method discussion, the importance of work and continued research is also mentioned.

The preliminary design phase occurs near the launch of an engineering project, normally after an initial requirements gathering phase. Through a series of meetingswhich gathers the key actors of a project, effective preliminary design involves discussion and decision-making punctuated by group creativity techniques. These activities are designed to explore the potential solutions of the problem, such asbrainstorming or causal analysis, or to address the project itself, such as collaborative project planning. Such activities are usually conducted in traditional meeting rooms with pen and paper media, which requires significant time and effort to prepare, perform, and later render into a digitally exploitable format. These processes have resisted previous attempts of computer-supported solutions, because any additional instruments risk obstructing the natural collaboration and workflow that make these activities so beneficial. Over the past decade, technologies such as interactive table tops, interactive wall displays, speech recognition software, 3D motion sensing cameras, and handheld tablets and smartphones have experienced significant advances in maturity. Theirform factors resemble the physical configuration of traditional pen-and-paper environments,while their "natural" input devices (based on multi-touch, gestures, voice, tangibles, etc.) allow them to leverage a user's pre-existing verbal, spatial,social, motor and cognitive skills. Researchers hypothesize that having these devices working in concert inside interactive spaces could augment collaboration forco-located (i.e. physically present) groups of users.There currently exist several interactive spaces in the literature, illustrating awide range of potential hardware configurations and interaction techniques. The goal of this thesis is first to explore what qualities these interactive spaces should exhibit in their interaction design, particularly with regard to preliminary designactivities, and second, to investigate how their heterogeneous and distributed computing devices can be unified into a flexible and extensible distributed computing architecture. The first main contribution of this thesis is an extensive presentation of an interactive space, which at its core uses a configuration not yet fully explored inprevious literature : a large multitouch table top and a large multitouch interactive Abstract board display. The design of this interactive space is driven by observations o fgroups engaged in preliminary design activities in traditional environments and a literature review aimed at extracting user-centered design guide lines. Special consideration is given to the user interface as it extends across multiple shared displays, and maintains a separation of concerns regarding personal and group work. Finally, evaluations using groups of five and six users show that using such an interactive space, coupled with our proposed multi-display interaction techniques, leads to a more effective construction of the digital artifacts used in preliminary design.The second main contribution of this thesis is a multi-agent infrastructure forthe distributed computing environment which effectively accommodates a widerange of platforms and devices in concerted interaction. By using agent-oriented programming and by establishing a common content language for messaging, the infrastructure is especially tolerant of network faults and suitable for rapid prototyping of heterogeneous devices in the interactive space.

Made available in DSpace on 2016-06-02T19:07:09Z (GMT). No. of bitstreams: 1 LEME_Ricardo_2015.pdf: 7202991 bytes, checksum: e01ea6e52b9be5ca7ceca1056abb2bcc (MD5) Previous issue date: 2015-01-23 / The use of small mobile devices such as smartphones and tablets has been growing in various public in Brazil. Even though many devices have features such as touch, gestures, voice, many applications do not adopt such resources to support different ways of interaction. Among the various audiences that have been using the mobile devices in Brazil, it can highlight the elderly person. To understand the needs of elderly audience is important not only observe their actions as well as brings them to inside of the process of development. User Centered Design (UCD) is a methodology that uses models, methods and processes for software design aiming to fulfill the user needs, composed by the steps of research, development and evaluation. Based on the UCD principals, this work presents an approach to support the design of interaction which besides being to the user, is also developed with the elderly user adding to the UCD cycle the techniques of participatory design and of personas. The application developed by the approach allows the elderly users to use the multimodal interaction according to their preferences and to the context of use. The methodology of this work was composed by three steps. First, a systematic review gave the support to the fundamentals and to the delimitation of the work scope. After that, the approach was designed based on UCD method including the participatory aspects. The proposal verification was conducted following the approach phases with the participation of a total of 279 elderly users. The results show that the elderly user, as a active member of the development process, aiding on the identification of real interaction aspects of the application. / O uso de dispositivos móveis, como smartphones e tablets vem crescendo junto a vários públicos no Brasil. Mesmo que muitos dispositivos possuam características tais como toque, gestos, voz, muitas aplicações não adotam tais recursos para apoiar diferentes formas de interação. Entre os diversos públicos que utilizam os dispositivos móveis no Brasil, pode-se destacar o usuário da terceira idade. Para entender as necessidades deste público, é importante não somente observar as suas ações, mas trazê-los para o processo de desenvolvimento. O Design Centrado no Usuário (UCD) é uma metodologia que utiliza modelos, métodos e processos de design de software com o objetivo para atender as necessidades do usuário, composto pelas etapas de pesquisa, desenvolvimento e avaliação. Com base nos princípios da UCD, este trabalho apresenta uma abordagem para apoiar o projeto de interação que além de ser para o usuário, também é desenvolvido com o usuário da terceira idade somando-se ao ciclo da UCD as técnicas de design participativo e de personas. A aplicação desenvolvida pela abordagem permite que os usuários idosos utilizem a interação multimodal de acordo com as suas preferências e para o contexto de utilização. A metodologia deste trabalho foi composta por três etapas. Em primeiro lugar, uma revisão sistemática deu o apoio para os fundamentos e para a delimitação do âmbito de trabalho. Depois disso, a abordagem foi concebida com base no método da UCD incluindo os aspectos de participação. A abordagem proposta foi realizada seguindo as fases de aproximação com a participação de um total de 279 usuários idosos. Os resultados mostram que o usuário idoso, como um membro ativo do processo de desenvolvimento, pode auxiliar na identificação de aspectos reais de interação da aplicação.

The thesis project concerns the development and evaluation of a real-time music visualization system aimed at creating a multi-modal perceptual experience of music emotions. The purpose of the project is to provide singers with real-time visual feedback on their singing, to enhance their expression of emotions in the music. Built upon results from previous studies on emotional expression in music, crossmodal correspondences, and associations among sound, shape, color, and emotions, a singing voice visualization system is proposed that generates real-time graphics to reflect the emotional expression in the input singing in an intuitive fashion. A mapping between musical and visual features was established and tested within a user study regarding the setting of its polarities. The singing voice visualization system was developed as as a software system that runs on personal computers, utilizing Pure Data and Unity. This implementation allows for instantaneous feedback to the user during their singing. The mapping was evaluated in a user study, where participants engage in expressive singing to test the system,in order to assess the meaningfulness of the visual feedback and the effectiveness of the mapping, as well as the impact of the polarity. The results show that color as a strong visual cue of emotional expression provided meaningful information on some participants’ expression of typical happiness and sadness. Other cues of the visual feedback possibly enhanced some participants’ emotional expression of singing in an indirect way. The polarity had a noticeable impact on the perception of the visual feedback. The current study is limited by the reliability of the techniques used for the extracting acoustic features from real-time singing, particularly in the detection of attack speed. The evaluation were limited by the broad definition of one of the research questions. The findings of this study suggest potential applications for the singing voice visualization system in fields of music education, art, and entertainment. Additionally, the research highlights the need for further exploration and refinement in the design of the mapping and the evaluation methodology. / Avhandlingsprojektet handlar om utveckling och utvärdering av ett realtidssystem för musikvisualisering som syftar till att skapa en multimodal perceptuell upplevelse av musikkänslor. Syftet med projektet är att ge sångare visuell feedback i realtid på deras sång, för att förstärka deras uttryck av känslor i musiken. Med utgångspunkt i resultat från tidigare studier om känslouttryck i musik, korsmodala korrespondenser och associationer mellan ljud, form, färg och känslor, föreslås ett visualiseringssystem för sångstämmor som genererar realtidsgrafik för att på ett intuitivt sätt återspegla känslouttrycket i den inmatade sången. En mappning mellan musikaliska och visuella funktioner etablerades och testades i en användarstudie avseende inställningen av dess polariteter. Visualiseringssystemet för sångröst utvecklades som ett mjukvarusystem som körs på persondatorer med Pure Data och Unity. Denna implementering möjliggör omedelbar feedback till användaren under dennes sång. Kartläggningen utvärderades i en användarstudie, där deltagarna fick sjunga uttrycksfullt för att testa systemet, för att bedöma meningsfullheten i den visuella feedbacken och kartläggningens effektivitet, samt polaritetens inverkan. Resultaten visar att färg som en stark visuell signal för känslomässiga uttryck gav meningsfull information om vissa deltagares uttryck av typisk lycka och sorg. Andra ledtrådar i den visuella återkopplingen förstärkte möjligen vissa deltagares känslomässiga uttryck av sång på ett indirekt sätt. Polariteten hade en märkbar inverkan på uppfattningen av den visuella feedbacken. Den aktuella studien begränsas av tillförlitligheten hos de tekniker som används för att extrahera akustiska egenskaper från sång i realtid, särskilt när det gäller att upptäcka attackhastighet. Utvärderingen begränsades av den breda definitionen av en av forskningsfrågorna. Resultaten av denna studie visar på potentiella tillämpningar för visualiseringssystemet för sångröster inom musikutbildning, konst och underhållning. Dessutom belyser forskningen behovet av ytterligare utforskning och förfining av kartläggningens utformning och utvärderingsmetodiken.

[pt] A presente tese investiga a multimodalidade em interações sinalizadas, em aulas via Google Meet. Os objetivos principais são descrever e analisar os aspectos verbais e não verbais da Libras. Este estudo está alinhado com as bases teóricas da Análise da Conversa (AC), da Sociolinguística Interacional (SI) e da Linguística Aplicada (LA). Esta pesquisa é de natureza qualitativa e utiliza a Análise da Conversa Multimodal como teoria e método. Utiliza a sequencialidade como teoria e, também, como categoria analítica. Os princípios da sequencialidade auxiliaram na compreensão das ações conjuntas construídas pelos surdos, em aulas virtuais. Os dados apresentados foram gerados de setembro a dezembro de 2020, nos encontros virtuais entre professor surdo e estudantes surdos que participam de um projeto de extensão situado na Zona da Mata Mineira. Para fins de transcrição e de análise de dados, adotaram-se critérios capazes de abarcar a interação multimodal em línguas sinalizadas. Então, utilizaram-se princípios da Tradução Audiovisual para se poder descrever as ações em Libras, aliando-se aos critérios de transcrição multimodal. Através da análise sequencial de turnos de fala, compreendem-se os diferentes enquadres presentes nas aulas virtuais sinalizadas. Nota-se que os aspectos multimodais presentes na interação dão a clave da interação, mostrando contextualmente qual ação os participantes estão performando momento a momento. Assim, as diferentes ações performadas em sala de aula, como a brincadeira, a narração, a explicação, mostram os papéis institucionais presentes nas aulas. Em alguns momentos, as relações tornam-se mais assimétricas, em outros, mais simétricas, dependendo das ações que os participantes estão performando. A compreensão desses significados interacionais só é possível quando conseguimos analisar a situação social como um todo, considerando os aspectos linguísticos, gramaticais, prosódicos, materiais e gestuais presentes nos turnos de fala, ou turnos sinalizados. Ao se olhar para os gestos presentes na Libras, é possível contribuiem mais r uma maneira de m se compreender o para o campo da Linguística Aplicada, pensando b aplicação no campo educacional, pensando abarque a experiência linguística que o surdo possu escolar institucional. / [en] This thesis investigates multimodality in signaled interactions, in classes given through Google Meet. The main objectives are to describe and analyze verbal and non-verbal aspects of Libras. This study is aligned with theoretical bases of Conversation Analysis (CA), Interactional Sociolinguistics (IS) and Applied Linguistics (AL). This research is qualitative in nature and uses Multimodal Conversation Analysis as theory and method. Uses sequentiality as a theory and also as an analytical category. The principles of sequentiality helped to understand the joint actions built by the deaf, in virtual classes. The data presented was generated from September to December 2020, in virtual meetings between deaf professor and deaf students who participate in an extension project located in Zona da Mata Mineira. For transcription and data analysis, criteria capable of encompassing multimodal interaction in signed languages were adopted. Then, principles of Audiovisual Translation were used to be able to describe the actions in Libras, allied to the criteria of multimodal transcription. Through sequential analysis of speech turns, different framings present in the signed virtual classes are understood. It is noted that the multimodal aspects present in the interaction give the key to the interaction, contextually showing which action the participants are performing moment by moment. Thus, different actions performed in the classroom, such as game, narration, explanation, show the institutional roles present in classes. In some moments, relationships become more asymmetrical, in others, more symmetrical, depending on the actions that participants are performing. Understanding these interactional meanings is only possible when we are able to analyze the social situation as a whole, considering linguistic, grammatical, prosodic, material and gestural aspects present in speech turns, or signed turns. When looking at the gestures present in Libras, it is possible to contribute to the field of Applied Linguistics, thinking of another way of understanding deaf bilingualism. There are also possibilities of application in educational field, thinking about educational policies that cover linguistic experience that the deaf has before going to the institutional school space.

Eye tracking and gestures are relatively new input methods, changing the way humans interact with computers. Gestures can be used for games or controlling a computer through an interface. Eye tracking is another way of interacting with computers, often by combining with other inputs such as a mouse or touch pad. Gestures and eye tracking have been used in commercially available products, but seldom combined to create a multimodal interaction. This thesis presents a prototype which combines eye tracking with gestures to interact with a computer. To accomplish this, the report investigates different methods of recognizing hand gestures. The aim is to combine the technologies in such a way that the gestures can be simple, and the location of a user’s gaze will decide what the gesture does. The report concludes by presenting a final prototype where the gestures are combined with eye tracking to interact with a computer. The final prototype uses an IR camera together with an eye tracker. The final prototype is evaluated with regards to learnability, usefulness, and intuitiveness. The evaluation of the prototype shows that usefulness is low, but learnability and intuitiveness are quite high. / Eye tracking och gester är relativt nya inmatningsmetoder, som förändra sättet människor interagerar med datorer. Gester kan användas för till exempel spel eller för att styra en dator via ett gränssnitt. Eye tracking är ett annat sätt att interagera med datorer, ofta med hjälp av genom att kombinera med andra styrenheter såsom en mus eller styrplatta. Gester och eye tracking har använts i kommersiellt tillgängliga produkter, men sällan kombinerats för att skapa en multimodal interaktion. Denna avhandling presenterar en prototyp som kombinerar eye tracking med gester för att interagera med en dator. För att åstadkomma detta undersöker rapporten olika metoder för att känna igen gester. Målet är att kombinera teknologierna på ett sådant sätt att gestern kan vara enkla, och platsen för användarens blick kommer bestämma vad gesten gör. Rapporten avslutas genom att presentera en slutlig prototyp där gester kombineras med eye tracking för att interagera med en dator. Den slutliga prototypen använder en IR kamera och en eye tracker. Den slutliga prototypen utvärderas med avseende på lärbarhet, användbarhet, och intuition. Utvärderingen av prototypen visar att användbarheten är låg, men både lärbarhet och intuition är ganska höga.

No description available.

This doctoral dissertation is an empirical qualitative research study on the emergence of learnables in classrooms of Swedish as a second language. It adopts a dialogical and praxeological approach, and analysis is based on video recorded teacher-student interactivities in classrooms. Learnables are taken to be linguistic items or constructs that are displayed as unknown by students, or problematized by students or teachers, and therefore oriented to as explainable, remediable, or improvable. Learnables are introduced in planned or less planned classroom activities, either in passing, while continuing the current main activity, or in sidesequences. In these activities, teachers and students not only talk, but also use other embodied resources (e.g. pointing) or available artifacts (e.g. worksheets) to highlight linguistic learnables. Teachers and students use these resources for achieving and maintaining intersubjectivity as well as contributing learnables to the interactivities. Through manifest embodied practices, abstract linguistic learnables become objectified, and knowledge about them gets organized in and through joint co-operative activities. / Denna avhandling är en empirisk, kvalitativ studie om uppkomsten av s.k. ”learnables” i svenska som andraspråksutbildning. Studien antar ett dialogiskt och praxeologiskt perspektiv, och analysen baseras på video-inspelade lärare-elevinteraktiviteter i klassrummet. ”Learnables” utgörs av språkliga objekt eller konstruktioner, som hanteras som obekanta av elever, eller som problematiseras av elever eller lärare, och därför orienteras emot som objekt som kan förklaras, korrigeras eller förbättras. ”Learnables” kan uppstå i planerade eller mindre planerade klarssrumsaktiviteter, antingen i förbigående, samtidigt som huvudaktiviteten fortsätter utan avbrott, eller i sidosekvenser. I dessa aktiviteter använder lärare och elever inte bara talspråk, utan även andra kroppsliga resurser (t.ex. pekningar) eller tillgängliga artefakter (t.ex. papper) för att fokusera på språkliga ”learnables”. Lärare och elever använder dessa medel för att uppnå och bibehålla intersubjektivitet, samt för att bidra med ”learnables” till interaktiviteterna. Genom konkreta kroppsliga metoder blir abstrakta, språkliga ”learnables” objektifierade och kunskapen om dem organiseras i och genom deltagarnas koordinerade handlingar.