Much traditional user interface evaluation is conducted in usability laboratories, where a small number of selected users is directly observed by trained evaluators. However, as the network itself and the remote work setting have become intrinsic parts of usage patterns, evaluators often have limited access to representative users for usability evaluation in the laboratory and the users' work context is difficult or impossible to reproduce in a laboratory setting. These barriers to usability evaluation led to extending the concept of usability evaluation beyond the laboratory, typically using the network itself as a bridge to take interface evaluation to a broad range of users in their natural work settings. The over-arching goal of this work is to develop and evaluate a cost-effective remote usability evaluation method for real-world applications used by real users doing real tasks in real work environments. This thesis reports the development of such a method, and the results of a study to: • investigate feasibility and effectiveness of involving users with to identify and report critical incidents in usage • investigate feasibility and effectiveness of transforming remotely-gathered critical incidents into usability problem descriptions • gain insight into various parameters associated with the method. / Master of Science

Traditional in lab usability evaluation has been used as the 'standard' evaluation method for evaluating and improving usability of software user interfaces (Andre, Williges, & Hartson, 2000). However, traditional in lab evaluation has its drawbacks such as availability of representative end users, high cost of testing and lack of true representation of a user's actual work environment. To counteract these issues various alternative and less expensive usability evaluation methods (UEMs) have been developed over the past decade. One such UEM is the Remote Usability Evaluation method. Remote evaluation is a relatively new area and lacks empirical data to support the approach. The need for empirical support was addressed in this study. The overall purpose of this study was to determine the differences in the effectiveness of the two evaluation types, the remote evaluation approach (SREM) and the traditional evaluation approach, in collecting usability data. This study also compared the effectiveness between the two methods based on user type, usability novice users and usability experienced users. Finally, the hypothesis that users, in general, will prefer the remote evaluation approach of reporting to the traditional in-lab evaluation approach was also tested. Results indicated that, in general, the synchronous remote approach is at least as effective as the traditional in lab usability evaluation approach in collecting usability data across all user types. However, when user type was taken into consideration, it was found that there was a significant difference in the high severity negative critical incident data collected between the two approaches for the novice user group. The traditional approach collected significantly more high severity negative critical incident data than the remote approach. Additionally, results indicate that users tend to be more willing to participate in the same approach as the one they participated previously. Recommendations for usability evaluators for conducting the SREM approach and areas for future research are identified in the study. / Master of Science

Στόχος της παρούσας διπλωματικής εργασίας είναι η ανάπτυξη ενός διαδικτυακού συστήματος το οποίο παρέχει την δυνατότητα εκτέλεσης πειραμάτων Ταξινόμησης Καρτών καθώς και ανάλυσης των αποτελεσμάτων που προκύπτουν. Αφότου έγινε συστηματική μελέτη της θεωρίας προσδιορίστηκαν οι κύριες αρχές, πάνω στις οποίες βασίστηκε η σχεδίαση και ανάπτυξη του συστήματος. Δόθηκε έμφαση στην ευχρηστία και στον διαδικτυακό χαρακτήρα του συστήματος καθώς και στην δημιουργία περιβάλλοντος υψηλής διαδραστικότητας. Για την επίτευξη του στόχου αξιοποιήθηκαν τεχνολογίες βάσεων δεδομένων και διαδικτύου. Η ευχρηστία του συστήματος είναι απόρροια διαδοχικών αξιολογήσεων από ειδικούς ευχρηστίας αλλά και δοκιμής του σε πραγματικές συνθήκες χρήσης. Το σύστημα αποτελεί ένα χρήσιμο εργαλείο στα χέρια ενός ερευνητή, αποκαλύπτοντας τις εννοιολογικές δομές ή κατηγοριοποιήσεις των ατόμων που εκτελούν το πείραμα, αλλά και ενός σχεδιαστή πληροφοριακών συστημάτων, ώστε να κατανοήσει καλύτερα τους χρήστες στους οποίους απευθύνεται και να βελτιώσει με αυτό τον τρόπο την ευχρηστία και την εμπειρία χρήστη. Τέλος, η λογική της μεταφοράς μιας μεθόδου όπως η Ταξινόμηση Καρτών σε ένα διαδικτυακό σύστημα μπορεί να επεκταθεί και σε άλλες ερευνητικές μεθόδους, απολαμβάνοντας αντίστοιχα οφέλη. / The aim of this thesis is to develop an online system that provides the ability to perform Card Sorting experiments and analysis of Card Sorting results. After a systematic study of theory, the main principles were identified upon which the design and development of the system was based on. Emphasis was placed on achieving a high level of usability and to create an online and highly interactive environment using RIA (Rich Internet Applications). To achieve this goal, database technologies and internet technologies as well as RIA were used. The usability of the system is the result of successive evaluations by experts and usability test in real conditions. The system is a useful tool for a researcher to reveal the conceptual structure or classifications(user mental model) of people participating in a Card Sorting experiment, but also for a designer of information systems to better understand the users to whom the design refers to and thereby to improve the usability and user experience. Finally, the logic of transferring a method such as Card Sorting from the "real world" in an online system can be extended to other research methods, enjoying the benefits of modern computer and internet technologies .