Organizations have recognized usability engineering as a needed step in the development process to ensure the success of any product. As is the case in all competitive settings areas for improvement are scouted and always welcomed. In the case of usability engineering a lot of time, money, equipment, and other resources are spent to gather usability data to identify and resolve usability problems in order to improve their product. The usability data gained from the expenditure of resources is often only applied to the development effort at hand and not reused across projects and across different development groups within the organization. More over, the usability data are often used at a level that forces the organization to only apply the data to that specific development effort. However, if usability data can be abstracted from the specific development effort and analyzed in relation to the process that created and identified the data; the data can then be used and applied over multiple development efforts. The User Action Framework (UAF) is a hierarchical framework of usability concepts that ensures consistency through completeness and precision. The UAF by its nature classifies usability problems at a high level. This high level classification affords usability engineers to not only apply the knowledge gained to the current development effort but to apply the knowledge across multiple development efforts. This author presents a mechanism and a process to allow usability engineers to find insights in their usability data to identify both strengths and weaknesses in their process. In return usability practitioners and companies can increase their return on investment by extending the usefulness of usability data over multiple development efforts. / Master of Science

Various methods exist for performing usability evaluations, but there is no systematic framework for guiding and structuring assessment and reporting activities (Andre et al., 2000). Researchers at Virginia Tech have developed a theoretical foundation called the User Action Framework (UAF), which is an adaptation and extension of Norman's action model (1986). The main objective of developing the User Action Framework was to provide usability practitioners with a reliable and structured tool set for usability engineering support activities like classifying and reporting usability problems. In practice, the tool set has a web-based interface, with the User Action Framework serving as an underlying foundation. To be an effective classification and reporting tool, the UAF should be usable and reliable. This work addressed two important research activities to help determine the usability and reliability of the User Action Framework. First, we conducted a formative evaluation of the UAF Explorer, a component of the UAF, and its content. This led a re-design effort to fix these problems and to provide an interface that resulted in a more efficient and satisfying user experience. Another purpose of this research was to conduct a reliability study to determine if the User Action Framework showed significantly better than chance agreement when usability practitioners classified a given set of usability problem descriptions according to the structure of the UAF. The User Action Framework showed higher agreement scores compared to previous work using the tool. / Master of Science

explosion in the use of computing and interactive systems of all kinds. This growth has brought with it an awareness among developers of interactive systems about the importance of user centered design and usability. In the Virginia Tech Usability tools lab, efforts are underway to develop tools and processes that assist in usability evaluation. This has led to the development of the User Action Framework (UAF) (Andre et al., 1999), a framework that forms the basis of several usability inspection tools like the Explorer, UPI (Usability Problem Inspector), UPC (Usability Problem Classifier) and the usability problem database. The UAF explorer is the tool in the toolkit, which allows users to explore and learn the structure of the UAF. The framework, in its final stages of development, is based on an interaction cycle derived from Norman's action model (1986). For its acceptance, the user action framework, which is based on human factors and cognitive concepts, needed a training program to make it accessible and understandable to the usability practioners in industry and academia. This thesis addressed the following research activities: (1) Developing a web-based training tool for the User Action Framework explorer and (2) Evaluating a web- based training tool using various formative evaluation techniques and a final summative evaluation to measure effectiveness of the training, transfer of training, knowledge/skill acquisition and reaction to the training. The summative evaluation used a pretest-posttest between subjects experimental design to determine the effectiveness of the training tool with the use of lecture-based training as a control group. The data collection included objective measures of performance and subjective measures through questionnaires and rating scales. The summative evaluation found no significant differences between gain scores on lecture-based training and web-based training under controlled conditions. Opportunities for future research were identified and the training tool is expected to contribute to the efforts of the VT usability tools lab towards educating usability professionals and researchers alike on the usefulness of the User Action Framework and its associated tools. This work also seeks to proliferate the use of web based training methods as a valuable way to train remote learners on such developing frameworks and toolkits. / Master of Science

Laboratory usability tests can be a rich source of usability information for software design, but are expensive to run and involve time-consuming data analysis. Expert review of software is cheaper, but highly dependent on the experience of the expert. Techniques are needed that maintain user involvement while reducing both the cost of user involvement and the time required to analyze data. The User Action Framework (UAF) is a classification scheme for usability problems that facilitates data analysis and reusability of information learned from one project to another, but is also reliant on expert interpretation of usability data, and classification can be difficult when user-supplied problem descriptions are incomplete. This study explored end-user classification of self-reported critical incidents (usability issues) using the UAF, a technique that was hoped to reduce expert interpretation of usability problems. It also explored end-user critical incident reporting from a usability session recording, rather than reporting incidents as soon as they occur, a technique that could be used in future studies to compare effectiveness of usability methods. Results indicate that users are not good at diagnosing their own critical incidents due to the level of detail required for proper classification, although observations suggest that users were able to provide usability information that would not have been captured by an expert observer. The recording technique was successful, and is recommended for future studies to further explore differences in the kind of information that can be gathered from end-users and from experts during usability studies. / Master of Science