State-of-the-art domestic robot assistants are essentially autonomous mobile manipulators capable of exerting human-scale precision grasps. To maximize utility and economy, non-technical end-users would need to be nearly as efficient as trained roboticists in control and collaboration of manipulation task behaviors. However, it remains a significant challenge given that many WIMP-style tools require superficial proficiency in robotics, 3D graphics, and computer science for rapid task modeling and recovery. But research on robot-centric collaboration has garnered momentum in recent years; robots are now planning in partially observable environments that maintain geometries and semantic maps, presenting opportunities for non-experts to cooperatively control task behavior with autonomous-planning agents exploiting the knowledge. However, as autonomous systems are not immune to errors under perceptual difficulty, a human-in-the-loop is needed to bias autonomous-planning towards recovery conditions that resume the task and avoid similar errors. In this work, we explore interactive techniques allowing non-technical users to model task behaviors and perceive cooperatively with a service robot under robot-centric collaboration. We evaluate stylus and touch modalities that users can intuitively and effectively convey natural abstractions of high-level tasks, semantic revisions, and geometries about the world. Experiments are conducted with 'pick-and-place' tasks in an ideal 'Blocks World' environment using a Kinova JACO six degree-of-freedom manipulator. Possibilities for the architecture and interface are demonstrated with the following features; (1) Semantic 'Object' and 'Location' grounding that describe function and ambiguous geometries (2) Task specification with an unordered list of goal predicates, and (3) Guiding task recovery with implied scene geometries and trajectory via symmetry cues and configuration space abstraction. Empirical results from four user studies show our interface was much preferred than the control condition, demonstrating high learnability and ease-of-use that enable our non-technical participants to model complex tasks, provide effective recovery assistance, and teleoperative control.

In anticipation of the complex, dynamic battlefields of the future, military operations are increasingly demanding robots with increased autonomous capabilities to support soldiers. Effective communication is necessary to establish a common ground on which human-robot teamwork can be established across the continuum of military operations. However, the types and format of communication for mixed-initiative collaboration is still not fully understood. This study explores two approaches to communication in human-robot interaction, transparency and communication pattern, and examines how manipulating these elements with a robot teammate affects its human counterpart in a collaborative exercise. Participants were coupled with a computer-simulated robot to perform a cordon-and-search-like task. A human-robot interface provided different transparency types - about the robot's decision making process alone, or about the robot's decision making process and its prediction of the human teammate's decision making process - and different communication patterns - either conveying information to the participant or both conveying information to and soliciting information from the participant. This experiment revealed that participants found robots that both conveyed and solicited information to be more animate, likeable, and intelligent than their less interactive counterparts, but working with those robots led to more misses in a target classification task. Furthermore, the act of responding to the robot led to a reduction in the number of correct identifications made, but only when the robot was solely providing information about its own decision making process. Findings from this effort inform the design of next-generation visual displays supporting human-robot teaming.

There are numerous sensors from which to choose when designing a mobile robot: ultrasonic, infrared, radar, or laser range finders, video, collision detectors, or beacon based systems such as the Global Positioning System. In order to meet the need for reliability, accuracy, and fault tolerance, mobile robot designers often place multiple sensors on the same platform, or combine sensor data from multiple platforms. The combination of the data from multiple sensors to improve reliability, accuracy, and fault tolerance is termed Sensor Fusion.;The types of robotic sensors are as varied as the properties of the environment that need to be sensed. to reduce the complexity of system software, Roboticists have found it highly desirable to adopt a common interface between each type of sensor and the system responsible for fusing the information. The process of abstracting the essential properties of a sensor is called Sensor Virtualization.;Sensor virtualization to date has focused on abstracting the properties shared by sensors of the same type. The approach taken by T. Henderson is simply to expose to the fusion system only the data from the sensor, along with a textual label describing the sensor. We extend Henderson's work in the following manner. First, we encapsulate both the fusion algorithm and the interface layer in the virtual sensor. This allows us to build multi-tiered virtual sensor hierarchies. Secondly, we show how common fusion algorithms can be encapsulated in the virtual sensor, facilitating the integration and replacement of both physical and virtual sensors. Finally, we provide a physical proof of concept using monostatic sonars, vector sonars, and a laser range-finder.

The genetic algorithm (GA) is a robust search technique which has been theoretically and empirically proven to provide efficient search for a variety of problems. Due largely to the semantic and expressive limitations of adopting a bitstring representation, however, the traditional GA has not found wide acceptance in the Artificial Intelligence community. In addition, binary chromosones can unevenly weight genetic search, reduce the effectiveness of recombination operators, make it difficult to solve problems whose solution schemata are of high order and defining length, and hinder new schema discovery in cases where chromosome-wide changes are required.;The research presented in this dissertation describes a grammar-based approach to genetic algorithms. Under this new paradigm, all members of the population are strings produced by a problem-specific grammar. Since any structure which can be expressed in Backus-Naur Form can thus be manipulated by genetic operators, a grammar-based GA strategy provides a consistent methodology for handling any population structure expressible in terms of a context-free grammar.;In order to lend theoretical support to the development of the syntactic GA, the concept of a trace schema--a similarity template for matching the derivation traces of grammar-defined rules--was introduced. An analysis of the manner in which a grammar-based GA operates yielded a Trace Schema Theorem for rule processing, which states that above-average trace schemata containing relatively few non-terminal productions are sampled with increasing frequency by syntactic genetic search. Schemata thus serve as the "building blocks" in the construction of the complex rule structures manipulated by syntactic GAs.;As part of the research presented in this dissertation, the GEnetic Rule Discovery System (GERDS) implementation of the grammar-based GA was developed. A comparison between the performance of GERDS and the traditional GA showed that the class of problems solvable by a syntactic GA is a superset of the class solvable by its binary counterpart, and that the added expressiveness greatly facilitates the representation of GA problems. to strengthen that conclusion, several experiments encompassing diverse domains were performed with favorable results.

This thesis presents such a new production system architecture, called a palimpsest parser, that adapts LR parsing technology to the process of controlled production system interpretation. Two unique characteristics of this architecture facilitate the construction and execution of large production systems: the rate at which productions fire is independent of production system size, and the modularity inherent in production systems is preserved and enhanced. In addition, individual productions may be evaluated in either a forward or backward direction, production systems can be integrated with other production systems and procedural programs, and production system modules can be compiled into libraries and used by other production systems.;Controlled production systems are compiled into palimpsest parsers as follows. Initially, the palimpsest transformation is applied to all productions to transform them into context-free grammar rules with associated disambiguation predicates and semantics. This grammar and the control grammar are then concatenated and compiled into modified LR(0) parse tables using conventional parser generation techniques. the resulting parse tables, disambiguation predicates, and semantics, in conjunction with a modified LR(0) parsing algorithm, constitute a palimpsest parser. When executed, this palimpsest parser correctly interprets the original controlled production system. Moreover, on any given cycle, the palimpsest parser only attempts to instantiate those productions that are allowed to fire by the control language grammar. Tests conducted with simulated production systems have consistently exhibited firing rates in excess of 1000 productions per second on a conventional microcomputer.

Blackboard systems are a natural progression of Artificial Intelligence based systems into a more powerful problem solving technique. They provide a way for several highly specialized knowledge sources to cooperate to solve large, complex problems. Blackboard systems incorporate the concepts developed by rule-based and expert systems programmers and include the ability to add conventionally coded knowledge sources. The small and specialized knowledge sources are easier to develop and test, and are hosted on hardware specifically suited to the task that they are solving.;Designing and developing blackboard systems is a difficult process. The designer is attempting to balance several conflicting goals and achieve a high degree of concurrent knowledge source execution while maintaining both knowledge and semantic consistency on the blackboard. Blackboard systems have not attained their apparent potential because no established tools or methods exist to guide in their construction or analyze their performance.;The Formal Model for Blackboard Systems was developed to provide a formal method for describing a blackboard system. The formal model outlines the basic components of a blackboard system, and how the components interact. A set of blackboard system design tools has been developed and validated for implementing systems that are expressed using the formal model. The tools are used to test and refine a proposed blackboard system design before the design is implemented. The set of blackboard system design tools consists of a Knowledge Source Organizer, a Knowledge Source Input/Output Connectivity Analyzer, and a validated Blackboard System Simulation Model. My preliminary research has shown that the level of independence and specialization of the knowledge sources directly affects the performance of blackboard systems. Using the design, simulation, and analysis tools I developed a concurrent object-oriented blackboard system that is faster, more efficient, and more powerful than existing systems. The use of the design and analysis tools provided the highly specialized and highly independent knowledge sources required for my concurrent blackboard system to achieve its design goals.

Artificial social intelligence is a step towards human-like human-computer interaction. One important milestone towards building socially intelligent systems is enabling computers with the ability to process and interpret the social signals of humans in the real world. Social signals include a wide range of emotional responses from a simple smile to expressions of complex affects. This dissertation revolves around computational models for social signal processing in the wild, using multimodal signals with an emphasis on the visual modality. We primarily focus on complex affect recognition with a strong interest in curiosity. In this dissertation,we ?rst present our collected dataset, EmoReact. We provide detailed multimodal behavior analysis across audio-visual signals and present unimodal and multimodal classi?cation models for affect recognition. Second, we present a deep multimodal fusion algorithm to fuse information from visual, acoustic and verbal channels to achieve a uni?ed classi?cation result. Third, we present a novel system to synthesize, recognize and localize facial occlusions. The proposed framework is based on a three-stage process: 1) Synthesis of naturalistic occluded faces, which include hand over face occlusions as well as other common occlusions such as hair bangs, scarf, hat, etc. 2) Recognition of occluded faces and differentiating between hand over face and other types of facial occlusions. 3) Localization of facial occlusions and identifying the occluded facial regions. Region of facial occlusion, plays an important role in recognizing affect and a shift in location can result in a very different interpretation, e.g., hand over chin can indicate contemplation, while hand over eyes may show frustration or sadness. Finally, we show the importance of considering facial occlusion type and region in affect recognition through achieving promising results in our experiments.

A core pursuit of artificial intelligence is the comprehension of human behavior. Imbuing intelligent agents with a good human behavior model can help them understand how to behave intelligently and interactively in complex situations. Due to the increase in data availability and computational resources, the development of machine learning algorithms for duplicating human cognitive abilities has made rapid progress. To solve difficult scenarios, learning-based methods must search for solutions in a predefined but large space. Along with implementing a smart exploration strategy, the right representation for a task can help narrow the search process during learning. This dissertation tackles three important aspects of machine intelligence: 1) prediction, 2) exploration, and 3) representation. More specifically we develop new algorithms for 1) predicting the future maneuvers or outcomes in pilot training and computer architecture applications; 2) exploration strategies for reinforcement learning in game environments and 3) scene representations for autonomous driving agents capable of handling large numbers of dynamic entities. This dissertation makes the following research contributions in the area of representation learning. First, we introduce a new time series representation for flight trajectories in intelligent pilot training simulations. Second, we demonstrate a method, Temporally Aware Embedding (TAE) for learning an embedding that leverages temporal information extracted from data retrieval series. Third, the dissertation introduces GRAD (Graph Representation for Autonomous Driving) that incorporates the future location of neighboring vehicles into the decision-making process. We demonstrate the usage of our models for pilot training, cache usage prediction, and autonomous driving; however, believe that our new time series representations can be applied to many other types of modeling problems.

With the growing impact of artificial intelligence, the topic of fairness in AI has received increasing attention. Artificial intelligence is observed to have caused unanticipated negative consequences. In this dissertation, we address two critical aspects regarding human-centered artificial intelligence (HCAI), a new paradigm for developing artificial intelligence that is ethical, fair, and helps to improve the human condition. In the first part of this dissertation, we investigate the effect that AI curation of contents by social media platforms has on an online discussions, by studying a polarized discussion in the Twitter network. We then develop a network communication model that simulates a polarized discussion, and propose two inoculation strategies to reverse the negative effects of polarization. Next we address the problem where AI might inadvertently result in increasing social inequalities. In doing so, a generative adversarial network is proposed to generate synthetic tabular datasets that are fair with respect to protected attributes such as race, sex, etc. Finally, an encoder-decoder network is developed to modify image datasets in order to improve fair attribute classification while maintaining classification accuracy. The contributions of this dissertation include: 1) understanding the effects of AI algorithms on societal well-being in terms of polarization and inequalities arising from the use of these algorithms in (a) curating content for users in an online social network and (b) decision making in areas of significant impact on human life; 2) addressing some of these concerns by a) providing a model to generate synthetic data, leading to training fair classifiers, b) providing an image encoder-decoder network that achieves superior fairness-accuracy trade-off, with the advantage that it does not rely on modifying downstream classifiers, hence making it suitable to be deployed in an automated machine learning pipeline with lower cost, and c) providing solutions to address polarization/influence related concern.

The Deep Neural Networks (DNN) have become the main contributor in the field of machine learning (ML). Specifically in the computer vision (CV), there are applications like image and video classification, object detection and tracking, instance segmentation and visual question answering, image and video generation are some of the applications from many that DNNs have demonstrated magnificent progress. To achieve the best performance, the DNNs usually require a large number of labeled samples, and finding the optimal solution for such complex models with millions of parameters is a challenging task. It is known that, the data are not uniformly distributed on the sample space, rather they are residing on a low-dimensional manifold embedded in the ambient space. In this dissertation, we specifically investigate the effect of manifold assumption on various applications in computer vision. First we propose a novel loss sensitive adversarial learning (LSAL) paradigm in training GAN framework that is built upon the assumption that natural images are lying on a smooth manifold. It benefits from the geodesic of samples in addition to the distance of samples in the ambient space to differentiate between real and generated samples. It is also shown that the discriminator of a GAN model trained based on LSAL paradigm is also successful in semi-supervised classification of images when the number of labeled images are limited. Then we propose a novel Capsule projection Network (CapProNet) that models the manifold of data through the union of subspace capsules in the last layer of a CNN image classifier. The CapProNet idea has been further extended to the general framework of Subspace Capsule Network that not only does model the deformation of objects but also parts of objects through the hierarchy of sub- space capsules layers. We apply the subspace capsule network on the tasks of (semi-) supervised image classification and also high resolution image generation. Finally, we verify the reliability of DNN models by investigating the intrinsic properties of the models around the manifold of data to detect maliciously trained Trojan models.