This thesis presents a novel two stage multimodal speech enhancement system, making use of both visual and audio information to filter speech, and explores the extension of this system with the use of fuzzy logic to demonstrate proof of concept for an envisaged autonomous, adaptive, and context aware multimodal system. The design of the proposed cognitively inspired framework is scalable, meaning that it is possible for the techniques used in individual parts of the system to be upgraded and there is scope for the initial framework presented here to be expanded. In the proposed system, the concept of single modality two stage filtering is extended to include the visual modality. Noisy speech information received by a microphone array is first pre-processed by visually derived Wiener filtering employing the novel use of the Gaussian Mixture Regression (GMR) technique, making use of associated visual speech information, extracted using a state of the art Semi Adaptive Appearance Models (SAAM) based lip tracking approach. This pre-processed speech is then enhanced further by audio only beamforming using a state of the art Transfer Function Generalised Sidelobe Canceller (TFGSC) approach. This results in a system which is designed to function in challenging noisy speech environments (using speech sentences with different speakers from the GRID corpus and a range of noise recordings), and both objective and subjective test results (employing the widely used Perceptual Evaluation of Speech Quality (PESQ) measure, a composite objective measure, and subjective listening tests), showing that this initial system is capable of delivering very encouraging results with regard to filtering speech mixtures in difficult reverberant speech environments. Some limitations of this initial framework are identified, and the extension of this multimodal system is explored, with the development of a fuzzy logic based framework and a proof of concept demonstration implemented. Results show that this proposed autonomous,adaptive, and context aware multimodal framework is capable of delivering very positive results in difficult noisy speech environments, with cognitively inspired use of audio and visual information, depending on environmental conditions. Finally some concluding remarks are made along with proposals for future work.

No description available.

No description available.

No description available.

Various roles for operators in human-machine systems have been proposed. This thesis shows that all of these views have in common the fact that operators perform best when given feedback that matches their intent. Past studies have shown that position control is superior to rate control except when operating large-workspace and/or dynamically slow manipulators and for exact tracking tasks. Operators of large-workspace and/or dynamically slow manipulators do not receive immediate position feedback. To remedy this lack of position feedback, a ghost arm overlay was displayed to operators of a dynamically slow manipulator, giving feedback that matches their intent. Operators performed several simple one- and two-dimensional tasks (point-to-point motion, tracking, path following) with three different controllers (position control with and without a ghost, rate control) to indicate how task conditions influence operator intent. Giving the operator position feedback via the ghost significantly increased performance with the position controller and made it comparable to performance with the rate control. These results were further validated by testing coordinated position control with and without a ghost arm and coordinated rate control on an excavator simulator. The results show that position control with the ghost arm is comparable, but not superior to rate control for the dynamics of our excavator example. Unlike previous work, this research compared the fuel efficiencies of different HMIs, as well as the time efficiencies. This work not only provides the design law of matching the feedback to the operator intent, but also gives a guideline for when to choose position or rate control based on the speed of the system.

As automated technologies continue to advance, they will be perceived more as collaborative team members and less as simply helpful machines. Expectations of the likely performance of others play an important role in how their actual performance is judged (Stephan, 1985). Although user expectations have been expounded as important for human-automation interaction, this factor has not been systematically investigated. The purpose of the current study was to examine the effect older and younger adults expectations of likely automation performance have on human-automation interaction. In addition, this study investigated the effect of different automation errors (false alarms and misses) on dependence, reliance, compliance, and trust in an automated system. Findings suggest that expectancy effects are relatively short lived, significantly affecting reliance and compliance only through the first experimental block. The effects of type of automation error indicate that participants in a false alarm condition increase reliance and decrease compliance while participants in a miss condition do not change their behavior. The results are important because expectancies must be considered when designing training for human-automation interaction. In addition, understanding the effects of type of automation errors is crucial for the design of automated systems. For example, if the automation is designed for diverse and dynamic environments where automation performance may fluctuate, then a deeper understanding of automation functioning may be needed by users.

As the tasks humans perform become more complicated and the technology manufactured to support those tasks becomes more adaptive, the relationship between humans and automation transforms into a collaborative system. In this system each member depends on the input of the other to reach a predetermined goal beneficial to both parties. Studying the human/automation dynamic as a social team provides a new set of variables affecting performance previously unstudied by automation researchers. One such variable is the shared mental model (Mathieu, Heffner, Goodwin, Salas, & Cannon-Bowers, 2000). This study examined the relationship between mental model quality and collaborative system performance within the domain of a navigation task. Participants navigated through a simulated city with the help of a navigational system performing at two levels of accuracy; 70% and 100%. Participants with robust mental models of the task environment identified automation errors when they occurred and optimally navigated to destinations. Conversely, users with vague mental models were less likely to identify automation errors, and chose inefficient routes to destinations. Thus, mental model quality proved to be an efficient predictor of navigation performance. Additionally, participants with no mental model performed as well as participants with vague mental models. The difference in performance was the number and type of errors committed. This research is important as it supports previous assertions that humans and automated systems can work as teammates and perform teamwork (Nass, Fog, & Moon, 2000). Thus, other variables found to impact human/human team performance might also affect human/automation team performance just as this study explored the effects of a primarily human/human team performance variable, the mental model. Additionally, this research suggests that a training program creating a weak, inaccurate, or incomplete mental model in the user is equivalent to no training program in terms of performance. Finally, through a qualitative model, this study proposes mental model quality affects the constructs of user self confidence and trust in automation. These two constructs are thought to ultimately determine automation usage (Lee & Moray, 1994). To validate the model a follow on study is proposed to measure automation usage as mental model quality changes.

While some 30 years ago the addition of computers to the human-machine environment was considered only for routine tasks in support tasks in support of the human, the balance has dramatically shifted to the computer now being able to perform almost any task the human is willing to delegate. Advances in automation and especially Artificial Intelligence have enabled the formation of rather unique teams with human and (electronic) machine members. Such teams are still led by the human with the machine as a subordinate associate or assistant, but as these become more complex, the automation that the human has to interact with is becoming increasingly intelligent and capable. / This thesis proposes a conceptual framework for automation and human-machine teaming that is based on developments in military aviation under the headings of Pilot's Associate or Crew Assistant. The starting point is the introduction of the machine-assistant into the traditional situation where the human is solely responsible for all activities. Analogies from classical control theory will be used to identify the boundaries, interfaces and tasks to be shared between the human and machine-assistant. Several schemes for task classification and allocation aim to establish a complementary relationship which will allow the human to stay in command and to utilise assistance of the machine to its fullest potential. Task management and coordination are requirements emphasised by the introduction of automation. Their correct implementation is extremely important in dealing with abnormal and high-workload situations. / The framework will be placed in the context of Cognitive Systems Engineering and Human Centered Automation. Both disciplines have been developed to provide answers to the problems associated with the aggressive introduction of automation into the traditional human-machine environment. Both disciplines also arrive at similar approaches which refocus on the human as the most important element in this environment. Several concepts of cognitive engineering and cognitive automation provide a theoretical reference for the proposed framework. / The conceptual framework highlights that the machine-assistant interactions are complex and that these interactions should exhibit intelligent behaviour in order to remain transparent to the human-operator at all times. Artificial Intelligence and Advanced Information Processing are technologies which are expected to be able to handle this complexity, to support a sophisticated human-machine dialogue and to minimise the cognitive gap between human and machine. Intelligent Agents, and in particular agents that apply the Belief-Desire-Intention architecture, have become attractive options to implement machine-assistants. Several areas for further research, such as human-agent teaming architectures, human-agent coordination and agent learning will be discussed. / Thesis (PhDElectronicEngineering)--University of South Australia, 2003.

While some 30 years ago the addition of computers to the human-machine environment was considered only for routine tasks in support tasks in support of the human, the balance has dramatically shifted to the computer now being able to perform almost any task the human is willing to delegate. Advances in automation and especially Artificial Intelligence have enabled the formation of rather unique teams with human and (electronic) machine members. Such teams are still led by the human with the machine as a subordinate associate or assistant, but as these become more complex, the automation that the human has to interact with is becoming increasingly intelligent and capable. / This thesis proposes a conceptual framework for automation and human-machine teaming that is based on developments in military aviation under the headings of Pilot's Associate or Crew Assistant. The starting point is the introduction of the machine-assistant into the traditional situation where the human is solely responsible for all activities. Analogies from classical control theory will be used to identify the boundaries, interfaces and tasks to be shared between the human and machine-assistant. Several schemes for task classification and allocation aim to establish a complementary relationship which will allow the human to stay in command and to utilise assistance of the machine to its fullest potential. Task management and coordination are requirements emphasised by the introduction of automation. Their correct implementation is extremely important in dealing with abnormal and high-workload situations. / The framework will be placed in the context of Cognitive Systems Engineering and Human Centered Automation. Both disciplines have been developed to provide answers to the problems associated with the aggressive introduction of automation into the traditional human-machine environment. Both disciplines also arrive at similar approaches which refocus on the human as the most important element in this environment. Several concepts of cognitive engineering and cognitive automation provide a theoretical reference for the proposed framework. / The conceptual framework highlights that the machine-assistant interactions are complex and that these interactions should exhibit intelligent behaviour in order to remain transparent to the human-operator at all times. Artificial Intelligence and Advanced Information Processing are technologies which are expected to be able to handle this complexity, to support a sophisticated human-machine dialogue and to minimise the cognitive gap between human and machine. Intelligent Agents, and in particular agents that apply the Belief-Desire-Intention architecture, have become attractive options to implement machine-assistants. Several areas for further research, such as human-agent teaming architectures, human-agent coordination and agent learning will be discussed. / Thesis (PhDElectronicEngineering)--University of South Australia, 2003.

Thesis (M. S.)--Psychology, Georgia Institute of Technology, 2008. / Committee Chair: Arthur D. Fisk; Committee Member: Gregory M. Corso; Committee Member: Wendy A. Rogers.