Interactive system developers spend most of their time and resources on user interface evaluation in traditional usability laboratories. Since the network itself and the remote work setting have become parts of usage patterns, evaluators do not have unlimited access to representative users for user interface evaluation. Reproducing the user's work context in a laboratory setting is also difficult. These problems have led to the concept of Remote usability evaluation that takes interface evaluation of any application beyond the laboratory setting. The main aim of this thesis work is to develop a tool that can record problems faced by remote users in the form of text and video. The text report and video, which is a sequence of the user's actions while encountering the problem, would help evaluators in preparing usability problem descriptions. This thesis reports the development of the remote usability evaluation method and the process of usability evaluation performed in enhancing features offered by the tool. / Master of Science

Traditional in lab usability evaluation has been used as the 'standard' evaluation method for evaluating and improving usability of software user interfaces (Andre, Williges, & Hartson, 2000). However, traditional in lab evaluation has its drawbacks such as availability of representative end users, high cost of testing and lack of true representation of a user's actual work environment. To counteract these issues various alternative and less expensive usability evaluation methods (UEMs) have been developed over the past decade. One such UEM is the Remote Usability Evaluation method. Remote evaluation is a relatively new area and lacks empirical data to support the approach. The need for empirical support was addressed in this study. The overall purpose of this study was to determine the differences in the effectiveness of the two evaluation types, the remote evaluation approach (SREM) and the traditional evaluation approach, in collecting usability data. This study also compared the effectiveness between the two methods based on user type, usability novice users and usability experienced users. Finally, the hypothesis that users, in general, will prefer the remote evaluation approach of reporting to the traditional in-lab evaluation approach was also tested. Results indicated that, in general, the synchronous remote approach is at least as effective as the traditional in lab usability evaluation approach in collecting usability data across all user types. However, when user type was taken into consideration, it was found that there was a significant difference in the high severity negative critical incident data collected between the two approaches for the novice user group. The traditional approach collected significantly more high severity negative critical incident data than the remote approach. Additionally, results indicate that users tend to be more willing to participate in the same approach as the one they participated previously. Recommendations for usability evaluators for conducting the SREM approach and areas for future research are identified in the study. / Master of Science

Remote usability evaluation is a usability evaluation method (UEM) where the experimenter, performing observation and analysis, is separated in space and/or time from the user. There are several approaches by which to implement remote evaluation, limited only by the availability of supporting technology. One such implementation method is RECITE (the REmote Critical Incident TEchnique), an adaptation of the user-reported critical incident technique developed by Castillo (1997). This technique requires that trained users, working in their normal work environment, identify and report critical incidents. Critical incidents are interactions with a system feature that prove to be particularly easy or difficult, leading to extremely good or extremely poor performance. Critical incident reports are submitted to the experimenter using an on-line reporting tool, who is responsible for their compilation into a list of usability problems. Support for this approach to remote evaluation has been reported (Hartson, H.R., Castillo, J.C., Kelso, J., and Neale, W.C., 1996; Castillo, 1997). The purpose of this study was to quantitatively assess the effectiveness of RECITE with respect to traditional, laboratory-based applications of the critical incident technique. A 3x2x 5 mixed-factor experimental design was used to compare the frequency and severity ratings of critical incidents reported by remote versus laboratory-based users. Frequency was measured according to the number of critical incident reports submitted and severity was rated along four dimensions: task frequency, impact on task performance, impact on satisfaction, and error severity. This study also compared critical incident data reported by trained users versus by usability experts observing end-users. Finally, changes in critical incident data reported over time were evaluated. In total, 365 critical incident reports were submitted, containing 117 unique usability problems and 50 usability success descriptions. Critical incidents were classified using the Usability Problem Inspector (UPI). A higher number of web-based critical incidents occurred during Planning than expected. The distribution of voice-based critical incidents differed among participant groups: users reported a greater than expected number of Planning incidents while experts reported fewer than expected Assessment incidents. Usability expert performance was not correlated, requiring that separate analyses be conducted for each expert data set. Support for the effectiveness in applying critical incidents to remote usability was demonstrated, with all research hypotheses at least partially supported. Usability experts gave significantly different ratings of impact on task performance than did user reporters. Remote user performance versus laboratory-based users failed to reveal differences in all but one measure: laboratory-based users reported more positive critical incidents for the voice interface than did remote users. In general, the number of negative critical incidents decreased over time; a similar result did not apply to the number of positive critical incidents. It was concluded that RECITE is an effective means of capturing problem-oriented data over time. Recommendations for its use as a formative evaluation method applied during the latter stages of product development (i.e. when a high fidelity prototype is available) are made. Opportunities for future research are identified. / Master of Science

Much traditional user interface evaluation is conducted in usability laboratories, where a small number of selected users is directly observed by trained evaluators. However, as the network itself and the remote work setting have become intrinsic parts of usage patterns, evaluators often have limited access to representative users for usability evaluation in the laboratory and the users' work context is difficult or impossible to reproduce in a laboratory setting. These barriers to usability evaluation led to extending the concept of usability evaluation beyond the laboratory, typically using the network itself as a bridge to take interface evaluation to a broad range of users in their natural work settings. The over-arching goal of this work is to develop and evaluate a cost-effective remote usability evaluation method for real-world applications used by real users doing real tasks in real work environments. This thesis reports the development of such a method, and the results of a study to: • investigate feasibility and effectiveness of involving users with to identify and report critical incidents in usage • investigate feasibility and effectiveness of transforming remotely-gathered critical incidents into usability problem descriptions • gain insight into various parameters associated with the method. / Master of Science

The aim with this thesis is to investigate two different methods used to test usability on web-sites to determine the advantages and disadvantages of two methods and compare the results. The two selected methods are the think aloud method and asynchronous remote method. Re-mote usability testing has existed for more than ten years but the possibilities with this meth-od needs to be explored more. To achieve the purpose of this thesis a case study was conducted on an e-commerce site and for the online remote usability testing software was used. The results of this study show that think aloud usability testing encounter more issues with the website than the remote usability testing.

This thesis aimed to investigate how moderated remote usability testing (RUT) tools can support the moderator's role and skills to create design principles for RUT tools. The study gathered knowledge of how to perform a RUT and the skills of a good moderator. This study resulted in a contextual inquiry with findings of what could support or obstruct the moderator according to the moderator's role and skills, opportunities and difficulties with RUT tools. The findings led to design principles of designing RUT tools to support the moderator's role and skills. The study resulted in usability, user and functional requirements with a prototype of a RUT tool to visualize how to use the design principles. / Denna studie syftade till att undersöka hur verktyg för modererade användbarhetstests på distans kan designas för att stödja moderatorns roll och färdigheter. Studien innefattar kunskap om hur man utför ett modererat användbarhetstest på distans och vilka färdigheter som krävs för att vara en bra moderator i sådana test. Studien innefattade en fältstudie som resulterade i funktioner i ett verktyg för användbarhetstester på distans som kan stödja eller hindra en moderators utförande med avseende på dess roll och färdigheter samt möjligheter och svårigheter i sådana verktyg. Designprinciper skapades i syfte att vägleda designen av verktyg för användbarhetstest på distans som stödjer moderatorns roll och färdigheter. Studien resulterade i användbarhets, användar- och funktionskrav som med designprinciperna visualiserades i en prototyp av ett verktyg för modererad användbarhetstest på distans.

This thesis deals with the topic of website optimization and user testing, with focus on the user experience. The main objective of this thesis is to design a software tool for on-line testing of the user experience of web applications, which currently does not exist on the Czech Internet. The first part deals with the user experience in theoretical aspect. It contains a comparison of experts' opinions on the issue and deals with the definition of basic concepts. The theoretical part contains an analysis of research techniques used in designing the user experience, which is an important output for the second part. Practical part of thesis includes at the beginning a research of the availability of tools to support analyzed research techniques on Czech Internet and assess their suitability for implementation into the designed tool. Afterwards thesis follow up the main goal - the development of a software tool for usability testing of web applications and websites (incl. the proposed business model and potential areas for future development). This tool should among other things provide benefits in terms of speed up and simplification of usability testing, including reducing the costs of testing.

No description available.