University of Macau / Faculty of Science and Technology / Department of Computer and Information Science

Human-machine interfaces influence both operator effectiveness and machine efficiency. Further immersion of the operator into the machine’s working environment gives the operator a better feel for the status of the machine and its working conditions. With this knowledge, operators can more efficiently control machines. The use of multi-modal HMIs involving haptics, sound, and visual feedback can immerse the operator into the machine’s environment and provide assistive clues about the state of the machine. This thesis develops a realistic excavator model that mimics a mini-excavator’s dynamics and soil interaction during digging tasks. A realistic graphical interface is written that exceeds the quality of current academic simulators. The graphical interface and new HMI are placed together with a model of the excavator’s mechanical and hydraulic dynamics into an operator workstation. Two coordinated control schemes are developed on an haptic display for a mini-excavator and preliminary tests are run to measure increases in operator effectiveness and machine efficiency.

In a large number of electronic devices, users interact with the system by navigating through various menus. Auditory menus can complement or even replace visual menus, so research on auditory menus has recently increased with mobile devices as well as desktop computers. Despite the potential importance of auditory displays on touch screen devices, little research has been attempted to enhance the effectiveness of auditory menus for those devices. In the present study, I investigated how advanced auditory cues enhance auditory menu navigation on a touch screen smartphone, especially for new input gestures such as tapping, wheeling, and flicking methods for navigating a one-dimensional menu. Moreover, I examined if advanced auditory cues improve user experience, not only for visuals-off situations, but also for visuals-on contexts. To this end, I used a novel auditory menu enhancement called a "spindex" (i.e., speech index), in which brief audio cues inform the users of where they are in a long menu. In this study, each item in a menu was preceded by a sound based on the item's initial letter. One hundred and twenty two undergraduates navigated through an alphabetized list of 150 song titles. The study was a split-plot design with manipulated auditory cue type (text-to-speech (TTS) alone vs. TTS plus spindex), visual mode (on vs. off), and input gesture style (tapping, wheeling, and flicking). Target search time and subjective workload for the TTS + spindex were lower than those of the TTS alone in all input gesture types regardless of visual type. Also, on subjective ratings scales, participants rated the TTS + spindex condition higher than the plain TTS on being 'effective' and 'functionally helpful'. The interaction between input methods and output modes (i.e., auditory cue types) and its effects on navigation behaviors was also analyzed based on the two-stage navigation strategy model used in auditory menus. Results were discussed in analogy with visual search theory and in terms of practical applications of spindex cues.

Common thought about cinema calls to mind an audience seated in a darkened theatre watching projected moving images that unfold a narrative onto a single screen. Cinema is much more than this. There is a significant history of artists experimenting with the moving image outside of its familiar setting in a movie theatre. These investigations are often referred to as "expanded cinema". This dissertation proposes a genre of expanded cinema called haptic cinema, an approach to interactive narrative that emphasizes material object sensing, identification and management; viewer's interaction with material objects; multisequential narrative; and the presentation of visual and audio information through multiple displays to create a sensorially rich experience for viewers. The interactive digital media tabletop is identified as one platform on which to develop haptic cinema. This platform supports a subgenre of haptic cinema called tabletop cinema. Expanded cinema practices are analyzed for their contributions to haptic cinema. Based on this theoretical and artistic research, the thesis claims that haptic cinema contributes to the historical development of expanded cinema and interactive cinema practices. I have identified the core properties of a haptic cinema practice during the process of designing, developing and testing a series of haptic cinema projects. These projects build on and make use of methods and conventions from tangible interfaces, tangible narratives and tabletop computing.

The touchscreen user interface is an inherently dynamic device that is becoming ubiquitous. The touchscreen’s ability to adapt to the user’s needs makes it superior to more traditional haptic devices in many ways. Most touchscreen devices come with a very large array of sensors already included in the package. This gives engineers the means to develop human-machine interfaces that are very intuitive to use. This thesis presents research that was done to develop a best touchscreen interface for driving an industrial crane for novice users. To generalize the research, testing also determined how touchscreen interfaces compare to the traditional joystick in highly dynamic tracking situations using a manual tracking experiment. Three separate operator studies were conducted to investigate touchscreen control of cranes. The data indicates that the touchscreen interfaces are superior to the traditional push-button control pendent and that the layout and function of the graphical user interface on the touchscreen plays a roll in the performance of the human operators. The touchscreen interface also adds great promise for allowing users to navigate through interactive textbooks. Therefore, this thesis also presents developments directed at creating the next generation of engineering textbooks. Nine widgets were developed for an interactive mechanical design textbook that is meant to be delivered via tablet computers. Those widgets help students improve their technical writing abilities, introduce them to tools they can use in product development, as well as give them knowledge in how some dynamical systems behave. In addition two touchscreen applications were developed to aid the judging of a mechanical design competition.

The venerable desktop metaphor is beginning to show signs of strain in supporting modern knowledge work. Traditional desktop systems were not designed to support the sheer number of simultaneous windows, information resources, and collaborative contexts that have become commonplace in contemporary knowledge work. Even though the desktop has been slow to evolve, knowledge workers still consistently manage multiple tasks, collaborate effectively among colleagues or clients, and manipulate information most relevant to their current task by leveraging the spatial organization of their work area. The potential exists for desktop workspaces to better support these knowledge work practices by leveraging the unifying construct of activity. Semantically-meaningful activities, conceptualized as a collection of tools (applications, documents, and other resources) within a social and organizational context, offer an alternative orientation for the desktop experience that more closely corresponds to knowledge workers' objectives and goals. In this research, I unpack some of the foundational assumptions of desktop interface design and propose an activity-centered model for organizing the desktop interface based on empirical observations of real-world knowledge work practice, theoretical understandings of cognition and activity, and my own experiences in developing two prototype systems for extending the desktop to support knowledge work. I formalize this analysis in a series of key challenges for the research and development of activity-based systems. In response to these challenges, I present the design and implementation of a third research prototype, the Giornata system, that emphasizes activity as a primary organizing principle in GUI-based interaction, information organization, and collaboration. I conclude with two evaluations of the system. First, I present findings from a longitudinal deployment of the system among a small group of representative knowledge workers; this deployment constitutes one of the first studies of how activity-based systems are adopted and appropriated in a real-world context. Second, I provide an assessment of the technologies that enable and those that pose barriers to the development of activity-based computing systems.

Thesis (M.S.)--State University of New York at Binghamton, Department of Computer Science, Thomas J. Watson School of Engineering and Applied Science, 2006. / Includes bibliographical references.

Thesis (M.Info.Tech.-Res.)--University of Wollongong, 2007. / Typescript. Includes bibliographical references.

Thesis (M.Eng.(Computer Engineering))--University of Pretoria, 2008. / Summaries in Afrikaans and English. Includes bibliographical references (leaves 77-83).

Thesis (Ph.D. (Computer Science)) - Rhodes University, 2009.