Relevance assessing is an important part of information retrieval (IR) evaluation in addition to being something that all users of IR systems must do as part of their search for relevant documents. In this thesis, we present a user study conducted to understand the relevance judging behaviour of assessors when the prevalence of relevant documents in a set of documents to be judged is varied. In our user study, we collected judgements of participants on document sets of three different prevalence levels. The prevalence levels that we used were low (0.1), balanced (0.5) and high (0.9). We found that participants who judged documents at the 0.9 level made the most mistakes, and participants who judged documents at the 0.5 level made the least mistakes. We did not find a statistically significant difference in judging quality between 0.1 and 0.5 prevalence levels.

This research study examined the design of a virtual keyboard that can be used for text entry with a rotary controller, particularly when users may differ in age and experience with a particular system. I specifically examined the shape and letter arrangement on the virtual keyboard to help determine the best features to use in a design. Two keyboard shapes, an Oval and a Plus, were selected to represent different aspects of the shape. Two keyboard arrangements, Alphabetic and a Standard QWERTY-based ordering, were selected to represent a well-known and less familiar arrangement. In the experiment, older and younger adults entered words over two consecutive days. Most of the time, they used either the Oval or the Plus, but they also used the alternate shape at specific points during their practice session to allow assessment of their ability to transfer what they had learned. At the end of the second day, they also used a variation of the practiced arrangement to examine how well they had learned the letter arrangement. Text entry performance on both shapes improved as a function of practice, demonstrating that participants could learn even unfamiliar devices and virtual keyboards to complete a word entry task. No overall shape effects were found for any level of performance, but shape did affect how participants learned and performed the word entry task. In particular, unique visual features on a shape may facilitate memorization of letter/visual cue mappings. These shape features are particularly important for older adults, as younger adults seem to develop a mental model that helps them memorize letter locations on either shape. With practice, older adults could achieve optimal performance levels with an Alphabetic keyboard on the Plus shape that has the more visually unique corners. In general, alphabetic ordering is best not only because it helped visual search, but also because it facilitated better movement planning. Overall, designers should consider creating unique visual features on a virtual keyboard that will blend with the compatibility and allowed movements for the selected device to create an effective virtual keyboard.

With the development and evolution of technology and aviation safety management, the aviation accident rate of the world has been reducing year by year, and the aviation safety situation appears to be improving. Nevertheless, in the analysis of the causes of aviation accidents of local and foreign civil aircrafts happened in recent years, the proportion of the cause being human factors of pilot is as high as 90%. And the two major accidents happened to the Navy in the past 10 years were also related to the human error of pilot. In other words, human factors are is the greatest threats to aviation safety, and should be taken as the core work of aviation safety management in aviation field. The study refers to the literature about the related human factors of the local and foreign civil aircraft and naval aircraft accidents, and takes the SHELL model as the classification criterion of the course structure. As a result, the study has screened and summarized 7 constructs, including ¡§brief introduction of human factors of aviation accidents,¡¨ ¡§aviation physiology of pilot,¡¨ ¡§aviation psychology of pilot,¡¨ ¡§interaction between pilot and other people,¡¨ ¡§relationship between pilot and equipment,¡¨ ¡§relationship between pilot and software,¡¨ and ¡§relationship betweel pilot and environment.¡¨ Under these 7 constructs, the paper designs 60 training subjects. The research approach of the paper adopts two-stage questionnaire survey. In the first stage, aviation experts screened out 7 constructs with 50 more important subjects. After that, survey is made on the helicopter pilots of the Navy and National Airborne Service Corps. Verbal evaluation function is adopted to analyze different constructs as well as the relative weights and sequence of different courses in these constructs. Furthermore, the study develops for the Navy three courses, namely initial training, specialized training and recurrent training, which are applicable to different training targets. Finally, the study refers to the related literature about human factors of civil aviation, and proposes some improvement strategies for the human factors management of naval pilot in order to ensure success of the training of the study. The study sets a 35-hour initial training course, a 26-hour specialized training course, and a 15-hour recurrent training course. Recurrent training course has to be taken regularly every half year. The skill practices of both specialized training and recurrent training can be implemented by the Line Oriented flight training or by the way of role playing so as to let pilots experience their practice in crew resource management. In addition, the study proposes to the Navy some improvement strategies in the aspects of human factors operation doctrine, organization, training and supervision. These improvement strategies and the training courses of the study can be provided to the Navy as a reference for future use in achieving the goal of great improvement on aviation safety.

With the advances in technology, clutter of mechanical indicators in the aircraft cockpit is replaced with digital displays. This revolution does not make only visual changes, but also changes the use of the cockpit design. Cockpit automation has changed cockpit design philosophy with many promised benefits such as improvements in the precision, improved system safety, efficiency of operations, less workload etc. However, to achieve perfect design has not been fulfilled yet. Despite providing innovation and easiness, cockpit automation brings about some &ldquo / Human Factors&rdquo / problems because of lack of support of human-machine interaction and cooperation. In this study, advantages and disadvantages of the cockpit automation will be discussed according to a survey that is conducted to pilots who fly with automated cockpits in Turkey about how automation affects them. The main purpose of this study is to contribute to the modifications of current cockpit systems and development of new design philosophy for advanced flight decks by gathering data from pilots&#039 / attitudes on cockpit automation philosophy.

Everyday technologies are intended for use by everyone with no specific training and minimal instructions. Prior research (e.g., Norman, 2002; Polson&Lewis, 1990) suggests that these technologies are usable if users can leverage their prior experience. However, different users will leverage difference experiences to operate the same technologies (Blackler, Popovic,&Mahar, 2003a). This dissertation systematically examined use of prior knowledge in the operation of everyday technology by diverse users, specifically users of different ages and experience levels. In Study 1 encounters with everyday technologies were self-reported by younger adults, older adults with low technology experience, and older adults with high technology experience. Comparisons of technology repertoires for each participant group indicated similar usage between younger adults and high tech older adults that differed in expected domains. Low tech older adults used fewer technologies, but overall they used more than expected across domains. Prior experience generally helped participants have successful encounters, but in some cases introduced problems. In Study 2 video recorded observations were made during participant interactions with exemplar everyday technologies. Participants with more relevant experience generally performed better. Older adults exhibited more inter-individual variability in their performance levels. Appropriate use of prior experience, an unassuming approach to the interaction, and using information on the technology generally led to more successful performance. Results from both studies can provide theoretical and practical support for more effective design that reflects how the target population will use their prior experience.

Although widely used, little is known about the nature of expertise involved in functional task analysis, methods used to discover and represent a task structure in terms of goals and subgoals. Training studies indicated that learning task analysis is not trivial. To counter the "task analysis is an art" explanation, this dissertation approached task analysis as a skill acquisition problem that can be understood through scientific inquiry. Two studies were designed to capture and characterize experienced and novice performance. Professional (Study 1) and novice (Study 2) task analysts conducted task analyses on six tasks from two domains (cooking, communication). Master task analyses were created for each task and served as a basis for analysis. Some similar patterns to the task analysis products and errors were observed for the hierarchy dimensions (breadth and depth of analysis), subgoal focus, and versatility. However, differences in separating subgoals (verb-noun pairs) were observed and may be further investigated in the future. Future directions could also focus on understanding the association between the general approach (breadth and depth-first) and the characteristics of the task analysis products. Skill components of functional task analysis were derived from the findings in both studies conducted for this dissertation.

Joint attention, a crucial component in interaction and an important milestone in human development, has drawn a lot of attention from the robotics community recently. Robotics researchers have studied and implemented joint attention for robots for the purposes of achieving natural human-robot interaction and facilitating social learning. Most previous work on the realization of joint attention in the robotics community has focused only on responding to joint attention and/or initiating joint attention. Responding to joint attention is the ability to follow another's direction of gaze and gestures in order to share common experience. Initiating joint attention is the ability to manipulate another's attention to a focus of interest in order to share experience. A third important component of joint attention is ensuring, where by the initiator ensures that the responders has changed their attention. However, to the best of our knowledge, there is no work explicitly addressing the ability for a robot to ensure that joint attention is reached by interacting agents. We refer to this ability as ensuring joint attention and recognize its importance in human-robot interaction. We propose a computational model of joint attention consisting of three parts: responding to joint attention, initiating joint attention, and ensuring joint attention. This modular decomposition is supported by psychological findings and matches the developmental timeline of humans. Infants start with the skill of following a caregiver's gaze, and then they exhibit imperative and declarative pointing gestures to get a caregiver's attention. Importantly, as they aged and social skills matured, initiating actions often come with an ensuring behavior that is to look back and forth between the caregiver and the referred object to see if the caregiver is paying attention to the referential object. We conducted two experiments to investigate joint attention in human-robot interaction. The first experiment explored effects of responding to joint attention. We hypothesize that humans will find that robots responding to joint attention are more transparent, more competent, and more socially interactive. Transparency helps people understand a robot's intention, facilitating a better human-robot interaction, and positive perception of a robot improves the human-robot relationship. Our hypotheses were supported by quantitative data, results from questionnaire, and behavioral observations. The second experiment studied the importance of ensuring joint attention. The results confirmed our hypotheses that robots that ensure joint attention yield better performance in interactive human-robot tasks and that ensuring joint attention behaviors are perceived as natural behaviors by humans. The findings suggest that social robots should use ensuring joint attention behaviors.

Multiple-task environments are pervasive in a variety of workplaces; many jobs require several concurrent, time-sensitive tasks be done in one task space. One concern in these multiple-task environments is attention allocation: To perform well, the operator must be able to know when and where to look. Otherwise, he or she will not be aware of the status of each task or be able to complete them. To aid these jobs, automation has been developed to support attention allocation: Auditory and visual alerts draw attention to where the system determines it is needed. However, imperfect automation may complicate the aid by introducing misses and false alarms to which the operator must also attend. Researchers studying these environments and automation's purview within them have focused on a variety of different topics. Some examples include: different types of automation (alerts, decision aid systems, etc.), levels of reliability (0-100% reliable), what automation supports (attention allocation to situation awareness to performance), and how automation affects multiple task environments (two tasks to many). Because attention had not been directly studied in relation to imperfect automation reliability in multiple-task environments, I decided to analyze the effects of different levels of automation reliability on visual attention allocation and how removal of that automation changed those effects. To study this, I helped to develop the Simultaneous Task Environment Platform (STEP), a program to study and test participants' behavior in multiple-task environments. The STEP program enabled me to vary the frequency and criticality (number of points gained/lost) of the different tasks to disambiguate how automation was affecting the participants. In the study, participants were trained on all four tasks of the STEP system, had the automation explained to them, and then were asked to gain as many points a trial as possible. There were three between-subject conditions; a system where ~70% of the automated alerts were reliable, one where ~90% of the alerts were reliable, and one where the participants received no automated aid at all. The automation was designed to support visual attention allocation. The participants interacted with the system and automation for twenty-four trials, divided into six blocks over two days, at which point they transferred to a system with no automation at all. To better understand exactly how the participants interacted with the system, I measured the number of times they accessed each task (attention allocation, as well as a measure of workload) and the number of points they scored (task performance). Mixed ANOVAs for these two measures, as well as a derived measure of efficiency (points scored per window opened), were conducted crossing automation condition with Block (to measure how the participants changed with experience) and task (to measure how certain tasks' attributes affected the way they were acted upon). Overall, the automation provided a benefit in terms of reduced workload and improved task performance. Participants in the automated conditions opened fewer windows and performed better. This also meant higher efficiency for those conditions. Experience affected conditions differentially. Those in the no automation condition increased their score but also the number of windows opened, causing their efficiency to stay the same. The 70% reliable condition was similar, with a minor point increase and no significant window decrease, resulting in no significant efficiency gain. The 90% reliable condition gained little score boost, but opened fewer windows by the end of the experiment, becoming more efficient. The frequency and criticality of tasks affected both the windows opened and the points scored across conditions, as participants in the two automated conditions opened fewer windows and scored relatively more points on those tasks worth many points that did not appear often. This increased their efficiency on those tasks, but also caused them to suffer greater when the automation was taken away. In the transfer trials, those participants in the automated conditions experienced both a workload increase and a performance decrease. These were centered on the two high-criticality/low-frequency tasks, as the other two tasks showed only small or no change between normal and transfer trials. These results show that automation at different levels of reliability affects the behavior of the operator of that system differentially based on the attributes of the tasks the operator must oversee. Tasks that happen often and are only important when aggregated over many are not aided by automation as much as those tasks that happen rarely and are critical every time they appear. When automation fails, however, those same tasks that are aided the most suffer the most, whereas those that do not get much aid do not suffer as much. Designers of automated systems should consider the type of tasks to be automated and their attributes, as well as the effects of increasing or decreasing the reliability of the automation when designing automation to provide support to system operators.

The objective of this thesis is to generate a discrete alphabet of low-level robotic controllers rich enough to mimic the actions of high-level users using the robot for a specific task. This alphabet will be built through the analysis of various user data sets in a modified version of the motion description language, MDLe. It can then be used to mimic the actions of a future user attempting to perform the task by calling scaled versions of the controls in the alphabet, potentially reducing the amount of data required to be transmitted to the robot, with minimal error. In this thesis, theory is developed that will allow the construction of such an alphabet, as well as its use to mimic new actions. A MATLAB algorithm is then built to implement the theory. This is followed by an experiment in which various users drive a Khepera robot through different courses with a joystick. The thesis concludes by presenting results which suggest that a relatively small group of users can generate an alphabet capable of mimicking the actions of other users, while drastically reducing bandwidth.

The vision of personal robot assistants continues to become more realistic with technological advances in robotics. The increase in the capabilities of robots, presents boundless opportunities for them to perform useful tasks for humans. However, it is not feasible for engineers to program robots for all possible uses. Instead, we envision general-purpose robots that can be programmed by their end-users. Learning from Demonstration (LfD), is an approach that allows users to program new capabilities on a robot by demonstrating what is required from the robot. Although LfD has become an established area of Robotics, many challenges remain in making it effective and intuitive for naive users. This thesis contributes to addressing these challenges in several ways. First, the problems that occur in teaching-learning interactions between humans and robots are characterized through human-subject experiments in three different domains. To address these problems, two mechanisms for guiding human teachers in their interactions are developed: embodied queries and teaching heuristics. Embodied queries, inspired from Active Learning queries, are questions asked by the robot so as to steer the teacher towards providing more informative demonstrations. They leverage the robot's embodiment to physically manipulate the environment and to communicate the question. Two technical contributions are made in developing embodied queries. The first is Active Keyframe-based LfD -- a framework for learning human-segmented skills in continuous action spaces and producing four different types of embodied queries to improve learned skills. The second is Intermittently-Active Learning in which a learner makes queries selectively, so as to create balanced interactions with the benefits of fully-active learning. Empirical findings from five experiments with human subjects are presented. These identify interaction-related issues in generating embodied queries, characterize human question asking, and evaluate implementations of Intermittently-Active Learning and Active Keyframe-based LfD on the humanoid robot Simon. The second mechanism, teaching heuristics, is a set of instructions given to human teachers in order to elicit more informative demonstrations from them. Such instructions are devised based on an understanding of what constitutes an optimal teacher for a given learner, with techniques grounded in Algorithmic Teaching. The utility of teaching heuristics is empirically demonstrated through six human-subject experiments, that involve teaching different concepts or tasks to a virtual agent, or teaching skills to Simon. With a diverse set of human subject experiments, this thesis demonstrates the necessity for guiding humans in teaching interactions with robots, and verifies the utility of two proposed mechanisms in improving sample efficiency and final performance, while enhancing the user interaction.