Voice User Interfaces (VUI) are rapidly gaining popularity, revolutionizing user interaction with technology through the widespread adoption in devices such as desktop computers, smartphones, and smart home assistants, thanks to significant advancements in voice recognition and processing technologies. Over a hundred million users now utilize these devices daily, and smart home assistants have been sold in massive numbers, owing to their ease and convenience in controlling a diverse range of smart devices within the home IoT environment through the power of voice, such as controlling lights, heating systems, and setting timers and alarms. VUI enables users to interact with IoT technology and issue a wide range of commands across various services using their voice, bypassing traditional input methods like keyboards or touchscreens. With ease, users can inquire in natural language about the weather, stock market, and online shopping and access various other types of general information.However, as VUI becomes more integrated into our daily lives, it brings to the forefront issues related to security, privacy, and usability. Concerns such as the unauthorized collection of user data, the potential for recording private conversations, and challenges in accurately recognizing and executing commands across diverse accents, leading to misinterpretations and unintended actions, underscore the need for more robust methods to test and evaluate VUI services. In this dissertation, we delve into voice interface testing, evaluation for privacy and security associated with VUI applications, assessment of the proficiency of VUI in handling diverse accents, and investigation into access control in multi-user environments. We first study the privacy violations of the VUI ecosystem. We introduced the definition of the VUI ecosystem, where users must connect the voice apps to corresponding services and mobile apps to function properly. The ecosystem can also involve multiple voice apps developed by the same third-party developers. We explore the prevalence of voice apps with corresponding services in the VUI ecosystem, assessing the landscape of privacy compliance among Alexa voice apps and their companion services. We developed a testing framework for this ecosystem. We present the first study conducted on the Alexa ecosystem, specifically focusing on voice apps with account linking. Our designed framework analyzes both the privacy policies of these voice apps and their companion services or the privacy policies of multiple voice apps published by the same developers. Using machine learning techniques, the framework automatically extracts data types related to data collection and sharing from these privacy policies, allowing for a comprehensive comparison. Next, researchers studied the voice apps' behavior to conduct privacy violation assessments. An interaction approach with voice apps is needed to extract the behavior where pre-defined utterances are input into the simulator to simulate user interaction. The set of pre-defined utterances is extracted from the skill's web page on the skill store. However, the accuracy of the testing analysis depends on the quality of the extracted utterances. An utterance or interaction that was not captured by the extraction process will not be detected, leading to inaccurate privacy assessment. Therefore, we revisited the utterance extraction techniques used by prior works to study the skill's behavior for privacy violations. We focused on analyzing the effectiveness and limitations of existing utterance extraction techniques. We proposed a new technique that improved prior work extraction techniques by utilizing the union of these techniques and human interaction. Our proposed technique makes use of a small set of human interactions to record all missing utterances, then expands that to test a more extensive set of voice apps. We also conducted testing on VUI with various accents to study by designing a testing framework that can evaluate VUI on different accents to assess how well VUI implemented in smart speakers caters to a diverse population. Recruiting individuals with different accents and instructing them to interact with the smart speaker while adhering to specific scripts is difficult. Thus, we proposed a framework known as AudioAcc, which facilitates evaluating VUI performance across diverse accents using YouTube videos. Our framework uses a filtering algorithm to ensure that the extracted spoken words used in constructing these composite commands closely resemble natural speech patterns. Our framework is scalable; we conducted an extensive examination of the VUI performance across a wide range of accents, encompassing both professional and amateur speakers. Additionally, we introduced a new metric called Consistency of Results (COR) to complement the standard Word Error Rate (WER) metric employed for assessing ASR systems. This metric enables developers to investigate and rewrite skill code based on the consistency of results, enhancing overall WER performance. Moreover, we looked into a special case related to the access control of VUI in multi-user environments. We proposed a framework for automated testing to explore the access control weaknesses to determine whether the accessible data is of consequence. We used the framework to assess the effectiveness of voice access control mechanisms within multi-user environments. Thus, we show that the convenience of using voice systems poses privacy risks as the user's sensitive data becomes accessible. We identify two significant flaws within the access control mechanisms proposed by the voice system, which can exploit the user's private data. These findings underscore the need for enhanced privacy safeguards and improved access control systems within online shopping. We also offer recommendations to mitigate risks associated with unauthorized access, shedding light on securing the user's private data within the voice systems. / Computer and Information Science

<p>This diploma work has been done as a part of the EC funded projects, MUSIC VK1- CT-2000-00058 and SmartDoc IST-2000-28137. The objective was to create an intuitive and easy to use visualization of flood forecasting data provided in the MUSIC project. This visualization is focused on the Visual User Interface and is built on small, reusable components. The visualization, FloodViewer, is small enough to ensure the possibility of distribution via the Internet, yet capable of enabling collaboration possibilities and embedment in electronic documents of the entire visualization. Thus, FloodViewer has been developed in three versions for different purposes. </p><p>Analysis and report generation (FloodViewer ) Collaborative analysis (FloodViewerNet ) Presentation and documentation (FloodViewerX).</p>

This diploma work has been done as a part of the EC funded projects, MUSIC VK1- CT-2000-00058 and SmartDoc IST-2000-28137. The objective was to create an intuitive and easy to use visualization of flood forecasting data provided in the MUSIC project. This visualization is focused on the Visual User Interface and is built on small, reusable components. The visualization, FloodViewer, is small enough to ensure the possibility of distribution via the Internet, yet capable of enabling collaboration possibilities and embedment in electronic documents of the entire visualization. Thus, FloodViewer has been developed in three versions for different purposes. Analysis and report generation (FloodViewer ) Collaborative analysis (FloodViewerNet ) Presentation and documentation (FloodViewerX).

Voice is the primary tool of communication to a majority of people on earth. Humans are wired to process speech, meaning voice interaction require little cognitive effort. Advancements in voice technology over the last 20 years have seen an increased prevalence of voice-controlled applications. However, false expectations can potentially cause severe interaction deficiencies to many of these voice user interfaces. One of the many application areas being connected to voice is access solutions such as smart locks. With a fundamental value proposition of keeping people and their belongings safe and secure in convenient ways, access solutions require that many of the design decisions are delicate. The present thesis aimed to evaluate how expectations of access solutions in home environments can be affected by different voice attributes. A literature study was conducted to explore the rich body of research on the topic of voice technology and the psychological effects of synthesized speech. Based on the literature study, a design process with recognized methods for developing voice user interfaces was conducted. The design process led up to a Wizard of Oz test that was used to evaluate how different conversational strategies and voices affected expectations and perception of a voice-controlled smart lock. The results showed that choosing an appropriate conversational style is fundamental to provide users with a sense of control. Furthermore, the study provided insight on how previous experience of interacting with voice-controlled devices have an impact on the feeling of personalization in gendered synthesized voices. Finally, the study discusses some ethical considerations that have to be made when designing voice user interfaces that ultimately should provide value to users, not confine their privacy. / Röst är den primära kommunikationskällan för de flesta människorna på jorden. Förmågan att bearbeta tal är något människor föds med, vilket gör att interaktion genom röst kräver liten kognitiv ansträngning. Framsteg inom röstteknologi under de senaste 20 åren har lett till ett ökat utbud av röststyrda applikationer. För många av dessa röststyrda applikationer existerar en risk att falska förväntningar leder till avsevärt försämrad interaktion. Ett av många applikationsområden där röst börjar framträda är accesslösningar såsom smarta lås. Med sitt fundamentala värde att hålla människor och deras tillgångar trygga och säkra utan att kompromissa enkelheten, kräver utformningen av accesslösningar flera delikata beslut. Därmed har studien ämnat att utvärdera hur användares förväntningar på röststyrda accesslösningar påverkas av röstattribut. För att undersöka och öka förståelsen av röstteknologi och de psykologiska effekterna av tal, genomfördes en litteraturstudie av existerande forskning initialt. Baserat på fynden i litteraturstudien startades sedan en designprocess för utveckling av ett röstgränssnitt. Erkända metoder användes för framtagandet av ett användarvänligt gränssnitt. Designprocessen låg till grund för ett Wizard of Oz test där olika konversationsstrategier och röster påverkade förväntningar av och uppfattningen av ett röststyrt smart lås. Resultaten visar att det är fundamentalt att välja en passande konversationsstil för att ge användare en känsla av kontroll. Studien påvisade även hur tidigare erfarenheter av röststyrda applikationer påverkar förväntningar av andra röststyrda applikationer och gör dem mer eller mindre personliga. Vidare diskuteras etiska avvägningar som måste göras när man designar röstgränssnitt som ska medföra användarvärde och inte inskränka användarnas integritet.

<p>This study has been carried out in cooperation with Unilever and partly with the EC founded project, Smartdoc IST-2000-28137. </p><p>In areas of statistics and image processing, both the amount of data and the dimensions are increasing rapidly and an interactive visualization tool that lets the user perform real-time analysis can save valuable time. Real-time cropping and drill-down considerably facilitate the analysis process and yield more accurate decisions. </p><p>In the Smartdoc project, there has been a request for a component used for smart filtering in multidimensional data sets. As the Smartdoc project aims to develop smart, interactive components to be used on low-end systems, the implementation of the self-organizing map algorithm proposes which dimensions to visualize. </p><p>Together with Dr. Robert Treloar at Unilever, the SOM Visualizer - an application for interactive visualization and analysis of multidimensional data - has been developed. The analytical part of the application is based on Kohonen’s self-organizing map algorithm. In cooperation with the Smartdoc project, a component has been developed that is used for smart filtering in multidimensional data sets. Microsoft Visual Basic and components from the graphics library AVS OpenViz are used as development tools.</p>

The purpose of this qualitative case study is to evaluate an early stage survey skill developed for the social robot Furhat, and look into how the user experience (UX) and voice user interface (VUI) of that skill can be improved. Several qualitative methods have been used: expert evaluations using heuristics for human-robot interaction (HRI), user evaluations including observations and interviews, as well as a quantitative questionnaire (RoSAS – Robot Social Attribution Scale). The empirical findings have been classified into the USUS Evaluation Framework for Human-Robot Interaction. The user evaluations were performed in two modes, one group of informants talked and interacted with Furhat with the support of a graphical user interface (GUI), and the other group without the GUI. A positive user experience was identified in both modes, showing that the informants found interacting with Furhat a fun, engaging and interesting experience. The mode with the supportive GUI could be suitable in noisy environments, and for longer surveys with many response alternatives to choose from, whereas the other mode could work better for less noisy environments and for shorter surveys. General improvements that can contribute to a better user experience in both modes were found; such as having the robot adopt a more human-like character when it comes to the dialogue and the facial expressions and movements, along with addressing a number of technical and usability issues. / Syftet med den här kvalitativa fallstudien är att utvärdera en enkätskill för den sociala roboten Furhat. Förutom utvärderingen av denna skill, som är i ett tidigt skede av utvecklingen, är syftet även att undersöka hur användarupplevelsen (UX) och röstgränssnittet (VUI) kan förbättras. Olika kvalitativa metoder har använts: expertutvärderingar med heuristik för MRI (människa-robot-interaktion), användarutvärderingar bestående av observationer och intervjuer, samt ett kvantitativt frågeformulär (RoSAS – Robot Social Attribution Scale). Resultaten från dessa har placerats in i ramverket USUS Evaluation Framework for Human- Robot Interaction. Användarutvärderingarna utfördes i två olika grupper: en grupp pratade och interagerade med Furhat med stöd av ett grafiskt användargränssnitt (GUI), den andra hade inget GUI. En positiv användarupplevelse konstaterades i båda grupperna: informanterna tyckte att det var roligt, engagerande och intressant att interagera med Furhat. Att ha ett GUI som stöd kan passa bättre för bullriga miljöer och för längre enkäter med många svarsalternativ att välja bland, medan ett GUI inte behövs för lugnare miljöer och kortare enkäter. Generella förbättringar som kan bidra till att höja användarupplevelsen hittades i båda grupperna; till exempel att roboten bör agera mer människolikt när det kommer till dialogen och ansiktsuttryck och rörelser, samt att åtgärda ett antal tekniska problem och användbarhetsproblem.

This study has been carried out in cooperation with Unilever and partly with the EC founded project, Smartdoc IST-2000-28137. In areas of statistics and image processing, both the amount of data and the dimensions are increasing rapidly and an interactive visualization tool that lets the user perform real-time analysis can save valuable time. Real-time cropping and drill-down considerably facilitate the analysis process and yield more accurate decisions. In the Smartdoc project, there has been a request for a component used for smart filtering in multidimensional data sets. As the Smartdoc project aims to develop smart, interactive components to be used on low-end systems, the implementation of the self-organizing map algorithm proposes which dimensions to visualize. Together with Dr. Robert Treloar at Unilever, the SOM Visualizer - an application for interactive visualization and analysis of multidimensional data - has been developed. The analytical part of the application is based on Kohonen’s self-organizing map algorithm. In cooperation with the Smartdoc project, a component has been developed that is used for smart filtering in multidimensional data sets. Microsoft Visual Basic and components from the graphics library AVS OpenViz are used as development tools.

In this thesis, a context-aware, personalized virtual assistant for use in automobiles is presented. With the increasing use of technology in automobiles, there is a growing need for safer and more practical ways for drivers to access information and perform tasks while driving. Voice-based interfaces, such as natural language processing, provide a solution to this problem as they do not require visual or manual input. In this thesis, a fine-tuned model of GPT-3 is used to understand user intentions and identify the user’s needs. The voice assistant is trained to understand the environment and the actions it can perform. The use of triggers such as drowsiness detection is also implemented to make the virtual assistant proactive in ensuring the user’s safety. User testing and evaluation was conducted to demonstrate the effectiveness of the context-aware, personalized virtual assistant in improving the driving experience and promoting safe driving practices.

The largest software companies in the world are racing to develop their Intelligent Personal Assistants. Each with more and more capabilities and ways for users to access them. The different assistants have different capabilities and one of the ways the companies are differentiating is in the interaction techniques their assistants that are controlled with, most uses voice, and some uses text. This thesis walks through these interaction techniques, give a brief overview of intelligent personal assistants today and explores how learnability differentiate when using these interaction techniques. The thesis also investigates when users have access to multiple interaction techniques to use in combination when interacting with such an assistant. To evaluate the field, a literature study was conducted covering Intelligent Personal Assistants, interaction techniques, conversational user interfaces, modalities, multimodal interaction, voice and chat interfaces. Interviews with early adopters of Intelligent Personal Assistants was performed to better understand the current state of the technology and how users were using it. To evaluate the learnability differences in different interaction techniques, a series of user tests in a wizard-of-oz setup was conducted. In the user tests the different groups was using different interaction techniques to interact with their assistants. The main findings of the user tests was that there are indications to that the type of language used differs depending on the interaction techniques. Users who interact through voice is more prone to use a more natural language, while users using a chat interface quickly started to treat the interactions with the Intelligent Personal Assistant as commands. The result observed was that users who interacted with the assistant using chat and then switched to interacting through voice used a shorter, more command-like language when they interacted through voice as well.

There is a current rising trend of using voice assistants (VAs) to perform tasks in new ways, and various companies are considering introducing them as part of their service propositions. One such company is the railway service provider SJ AB that is interested in understanding how a VA may benefit their passengers. To better understand the utility of a VA in the railway service context, this thesis aims to explore the value creation opportunities that the introduction of such a technology presents. This exploration is done through identifying the potential functionality of such a VA, what value that functionality can create for passengers, and what barriers exist for creating that value. This identification is done through a user-centred research process, during which design probes and experience prototyping were carried out with railway passengers and analysed through qualitative content analysis. The results show that a VA presents fast, convenient, and intuitive access to a wide set of functions but is hindered by its invisible affordances and the user’s preconceptions. It shares several functions with existing channels, presented in a new way, that span the entire course of the train journey. The functionality and the identified functions primarily create utilitarian values, which connects to the passenger’s pragmatic view of the service. Furthermore, the passengers perceived the VA as an agent able to take over responsibilities and tasks from the passenger, resulting in a peace of mind but also a diminished sense of control. Finally, the railway service context affected what functionality is suitable, what values can be created and what barriers need to be considered. Some of the value creation possibilities and barriers are also contemporary and might change with shifting social norms and further technological development.