Doctor of Philosophy / Department of Psychology / John J. Uhlarik / Usability practitioners need effective usability assessment techniques in order to facilitate development of usable consumer products. Many usability evaluation methods have been promoted as the ideal. Few, however, fulfill expectations concerning effectiveness. Additionally, lack of empirical data forces usability practitioners to rely on personal judgments and/or anecdotal statements when deciding upon which usability method best suits their needs. Therefore the present study had two principal objectives: (1) to validate a hybrid usability technique that identifies important and ignores inconsequential usability problems, and (2) to provide empirical performance data for several usability protocols on a variety of contemporary comparative metrics. The User-Centered Hybrid Assessment Tool (User-CHAT) was developed to maximize efficient diagnosis of usability issues from a behaviorally-based perspective while minimizing time and resource limitations typically associated with usability assessment environments. Several characteristics of user-testing, the heuristic evaluation, and the cognitive walkthrough were combined to create the User-CHAT. Prior research has demonstrated that the User-CHAT supports an evaluation within 3-4 hrs, can be used by individuals with limited human factors / usability background, and requires little training to be used competently, even for complex systems. A state-of-the-art suite of avionics displays and a series of benchmark tasks provided the context where the User-CHAT’s performance was measured relative to its parent usability methods. Two techniques generated comparison lists of usability problems – user-testing data and various inclusion criteria for usability problems identified by the User-CHAT, heuristic evaluation, and cognitive walkthrough. Overall the results demonstrated that the User-CHAT attained higher effectiveness scores than the heuristic evaluation and cognitive walkthrough, suggesting that it helped evaluators identify many usability problems that actually impact users, i.e., higher thoroughness, while attenuating time and effort on issues that were not important, i.e., higher validity. Furthermore, the User-CHAT had the greatest proportion of usability problems that were rated as serious, i.e., usability issues that hinder performance and compromise safety. The User-CHAT’s performance suggests that it is an appropriate usability technique to implement into the product development lifecycle. Limitations and future research directions are discussed.

The Cognitive Dimensions of Notations (CDs) is an evaluation framework consisting of a set of heuristics meant to be used as discussion tools for usability evaluation of notational systems. Programming languages and frameworks are examples of notational systems, and the development of new such languages and frameworks could improve by analysing the properties with CDs. However, the CDs framework has been criticized for not truly being a scientific method as well as being unfit to be used in certain specific research fields, even though the general nature of the framework allows for it. In this paper, the CDs are applied to evaluate SwiftUI, a declarative user interface framework for iOS. The possibilities and limitations of CD is discussed in relation to this CD example, as well as previous criticism and other evaluation frameworks. The paper concludes with visions for the future of the CDs framework. / Cognitive Dimensions of Notations (CDs) är ett utvärderingsramverk bestående av ett flertal heuristiker vars syfte är att användas som diskussionsverktyg vid användarbarhetsutvärderingar av notationssystem. Utvecklingen av nya notationssystem, exemplevis nya programmeringsspråk och ramverk, kan förbättras genom att använda CDs för att analysera dess egenskaper. Kritik har dock riktats mot CDs i att metoden inte är vetenskaplig nog samt att den inte är applicerbar på vissa specifika forskningsområden trots utvärderingsramverkets breda användningsområde. I denna masteruppsats används CDs för att utvärdera SwiftUI, ett deklarativt ramverk för att skapa användargränssnitt på iOS. Möjligheter och begränsningar av att använda CDs i detta syfte diskuteras tillsammans med föregående kritik och relaterade utvärderingsramverk. Avslutningsvis diskuteras framtida möjligheter för CDs.

京都大学 / 0048 / 新制・課程博士 / 博士(工学) / 甲第18249号 / 工博第3841号 / 新制||工||1589(附属図書館) / 31107 / 京都大学大学院工学研究科都市社会工学専攻 / (主査)教授 河野 広隆, 教授 宮川 豊章, 准教授 服部 篤史 / 学位規則第4条第1項該当 / Doctor of Philosophy (Engineering) / Kyoto University / DFAM

This paper can be seen as a first step towards developing genre tailored heuristics for visual novels. The focus being visual novels come from the fact that it is a genre of games that is growing in popularity globally. Besides players and developers, it is also popular among researchers because the genre is very accessible (Camingue, Carstensdottir & Melcer 2021). But let’s say the developers and researchers want to evaluate a visual novel they are creating, what methods would be fitting. There isn’t a lot of research on this, so the method suggested in this paper is heuristic evaluation, due to it being a very accessible method (Nielsen 1994). To tailor heuristics for this genre, a compiled list of heuristics was used to evaluate a number of visual novels and the data from the evaluation was analyzed and used to identify problems that weren’t addressed by the heuristics.

Evaluating User Experiences early in a development process can save both time and money by pro-actively mapping out user needs and behavior patterns. However, since most well-known UX-evaluation methods applies during or after user interaction, due to the “second-order” design problem of experiences being a byproduct of interaction, there is a desire within HCI for more early-stage UX-evaluation methods that could be applied to concept stages as well. This paper investigates the experiential evaluation of a storyboard and narrative through the Anticipated eXperience Method (AxE) and discusses how it compares to a re-iterated high-fidelity prototype created in Figma. The process of this study is described out of the context it has been executed in, which is together with the company Scania AB at their cabin production facility in Oskarshamn, Sweden. The study explores what insights can be found from evaluating anticipated user experiences in early concept development and how these insights can apply towards further development of a touchpad user Interface. The underlying foundation of this study has followed the approach of a design-inclusive UX-research project, which heavily incorporates design activities into the process of conducting research. Thus, the storyboard, narrative and interactive Figma prototype have been created along the process and takes center stage in the investigation of experiential evaluation at early stages of interactive product development.

As states continue to consider taking on more responsibility in transportation, a major issue State Departments of Transportation (DOTs) face relates to financing future transportation investments. At present, many state transportation policymakers and State DOT administrators are considering alternative financing approaches to generate future revenue sources for transportation investments. This dissertation focuses on several user fee based approaches currently being considered by state transportation policymakers and administrators in the U.S. Examples of such approaches include: increasing the current fuel tax and indexing the fuel tax to inflation; implementing an odometer based vehicle miles traveled (VMT) fee approach through vehicle inspection programs in selected states; establishing a global positioning system (GPS) based VMT fee approach for heavy vehicles where privacy and implementation costs are less of a concern; and increasing existing tolls and charging tolls on existing roads that do not have tolls, preferably with open-road tolling (ORT) and all-electronic toll (AET) payment systems. Meanwhile, major questions of interest relate to the potential impacts or consequences of such financing approaches. Central to this dissertation is the development of a conceptual framework and analytical methods to aid state transportation policymakers and administrators in the planning and formulation of alternative financing approaches suitable for consideration in their state. The application of the framework and methods is illustrated in a case study. This case study includes an evaluation of alternative toll scenarios on a section of Interstate 93 in the Boston Metropolitan area where at present tolls are not charged. A major conclusion of the case study is that placing tolls along interstate highways where tolls are not currently collected has the potential to provide a significant source of revenue for State DOTs but that other impacts including route diversion, privacy, and equity need to be considered and addressed in the decision-making process. It is expected that the results of the dissertation will be of interest to state transportation policy makers as well as State DOT administrators currently involved in the development of a comprehensive transportation finance policy.

Additive manufacturing (AM) is a novel fabrication technique capable of producing highly complex parts. Nevertheless, a major challenge is improving the quality of the fabricated parts. While there are several ways of approaching this problem, developing data-driven methods that use AM process signatures to identify these part anomalies can be rapidly applied to improve the overall part quality during the build. The objective of this dissertation is to model multiple processes within the AM to quantify the quality of the parts and reduced the uncertainty due to variation in input process parameters. The objective of first study is to build a new layer-wise process signature model to characterize the thermal-defect relationship. Based on melt pool images, we propose novel layer-wise key process signatures, which are calculated using multilinear principal component analysis (MPCA) and are directly correlated with layer-wise quality of the part. Second study broadens the spectrum of the dissertation to include mechanical properties, where a novel two-phase modeling methodology is proposed for fatigue life prediction based on in-situ monitoring of thermal history. In final study, our objective is to pave the way toward a better understanding of the uncertainty in the process-defect-structures relationship using an inverse robust design exploration method. The method involves two steps. In the first step, mathematical models are developed to characterize and model the forward flow of information in the intended additive manufacturing process. In the second step, inverse robust design exploration is carried out to investigate satisfying design solutions that meet multiple AM goals.

Previous usability evaluation method (UEM) comparison studies have noted an evaluator effect on problem detection in heuristic evaluation, with evaluators differing in problems found and problem severity judgments. There have been few studies of the evaluator effect in usability testing (UT), task-based testing with end-users. UEM comparison studies focus on counting usability problems detected, but we also need to assess the content of usability problem descriptions (UPDs) to more fully measure evaluation effectiveness. The goals of this research were to develop UPD guidelines, explore the evaluator effect in UT, and evaluate the usefulness of the guidelines for grading UPD content. Ten guidelines for writing UPDs were developed by consulting usability practitioners through two questionnaires and a card sort. These guidelines are (briefly): be clear and avoid jargon, describe problem severity, provide backing data, describe problem causes, describe user actions, provide a solution, consider politics and diplomacy, be professional and scientific, describe your methodology, and help the reader sympathize with the user. A fourth study compared usability reports collected from 44 evaluators, both practitioners and graduate students, watching the same 10-minute UT session recording. Three judges measured problem detection for each evaluator and graded the reports for following 6 of the UPD guidelines. There was support for existence of an evaluator effect, even when watching pre-recorded sessions, with low to moderate individual thoroughness of problem detection across all/severe problems (22%/34%), reliability of problem detection (37%/50%) and reliability of severity judgments (57% for severe ratings). Practitioners received higher grades averaged across the 6 guidelines than students did, suggesting that the guidelines may be useful for grading reports. The grades for the guidelines were not correlated with thoroughness, suggesting that the guideline grades complement measures of problem detection. A simulation of evaluators working in groups found a 34% increase in severe problems found by adding a second evaluator. The simulation also found that thoroughness of individual evaluators would have been overestimated if the study had included a small number of evaluators. The final recommendations are to use multiple evaluators in UT, and to assess both problem detection and description when measuring evaluation effectiveness. / Ph. D.

No description available.

Evaluation is a vital--yet challenging--part of IS/IT management and governance. The benefits (or lack therefore) associated with IS/IT investments have been widely debated within academic and industrial communities alike. Investments in information technology may or may not result in desirable outcomes. Yet, organizations must rely on information systems to remain competitive. Effective evaluation serves as one pathway to ensuring success. However, despite a growing multitude of measures and methods, practitioners continue to struggle with this intractable problem. Responding to the limited success of existing methods, scholars have argued that academicians should first develop a better understanding of the process of IS/IT evaluation. In addition, scholars have also posited that IS/IT evaluation practice should be tailored to fit a given organization's particular context. Of course, one cannot simply tell practitioners to "be contextual" when conducting evaluations and then hope for improved outcomes. Instead, having developed an improved understanding of the IS/IT evaluation process, researchers should articulate unambiguous guidelines to practitioners. The researcher addressed this need using a multi-phase research methodology. To start, the researcher conducted a literature review to identify and describe the relevant contextual elements operating in the IS/IT evaluation process: the purpose of conducting the evaluation (why); the subject of the evaluation (what); the specific aspects to be evaluated (which); the particular evaluation methods and techniques used (how); the timing of the evaluation (when); the individuals involved in, or affected by, the evaluation (who); and the environmental conditions under which the organization operates (where). Based upon these findings, the researcher followed a modeling-as-theorizing approach to develop a conceptual model of IS/IT evaluation. Next, the conceptual model was validated by applying it to multiple case studies selected from the extant literature. Once validated, the researcher utilized the model to develop a series of methodological guidelines to aid organizations in conducting evaluations. The researcher summarized these guidelines in the form of a checklist for professional practitioners. The researcher believes this holistic, conceptual model of IS/IT evaluation serves as an important step in advancing theory. In addition, the researcher's guidelines for conducting IS/IT evaluation based on organizational goals and conditions represents a significant contribution to industrial practice. Thus, the implications of this study come full circle: an improved understanding of evaluation should result in improved evaluation practices.