The philosophy of research at the University of Texas – Robotics Research Group (RRG) is towards creating a foundation for an open architecture, reconfigurable intelligent machines to meet wide breadth of operational needs. An intelligent system is the one which has complete knowledge of its operating characteristics at all times (updated in real-time) and it can make on-the-fly decisions to adapt itself to the different conditions or present the best possible options to the human decision maker under specified and ranked criteria. The reality of all complex system is that they are inherently non-linear with coupled parameters. The traditional approach dealing with such systems assumes linearized models, imposing conservative bounds on the operational domain and thus limiting performance capability of the system. Recent advancements in sensor technology and availability of computational resources (embedded processing) at low cost have made real-time intelligent control feasible for complex systems. The computational intelligence envisioned in modern intelligent machines will enhance the system performance and will provide capabilities such as criteria based control, identification of incipient faults, condition based maintenance, fault tolerance, and ability to monitor performance parameters in real-time. The first step in this process is to equip a system with a comprehensive suite of sensors. These sensors will provide real-time data and awareness about both, the internal system states and the external/environmental operating conditions. The aim of this work is to establish an argument in favor of using multiple sensors in all complex electro-mechanical systems. The report discusses numerous benefits of a multi-sensor environment with suitable examples and attempts to justify its pressing need in all the existing complex mechanical systems. Case studies for a multi-sensor environment in railroad freight cars and vehicle systems are presented. Sensing requirements in freight train and vehicle systems are evaluated and suitable sensor technology and commercial sensor options are suggested for decision makers. In addition to benefits, challenges in a multi-sensor environment such as sensor noise, cabling complexities, signal processing, communication, data validation and data management, sensor fusion, information integration, maintenance etc. are addressed and best practices to alleviate these complexities are discussed in the report. This effort lays out a foundation for developing a multi-sensor system and will enable computational intelligence and structured decision making in the system. / text

ITC/USA 2009 Conference Proceedings / The Forty-Fifth Annual International Telemetering Conference and Technical Exhibition / October 26-29, 2009 / Riviera Hotel & Convention Center, Las Vegas, Nevada / Several navigation technologies exist, which can facilitate the generation of Time Space Positioning Information (TSPI) in urban environments. These include GPS, image-based localization, radio-based localization and dead reckoning. This paper first presents a basic overview of these techniques including advantages and limitations of each. We present an approach to localization in urban environments, based on environment learning and collaborative navigation using multiple homogeneous and non-homogeneous localization technologies, fused to form a multi-sensor system.

This thesis work is motivated by the potential and promise of image fusion technologies in the multi sensor image fusion system and applications. With specific focus on pixel level image fusion, the process after the image registration is processed, we develop graphic user interface for multi-sensor image fusion software using Microsoft visual studio and Microsoft Foundation Class library. In this thesis, we proposed and presented some image fusion algorithms with low computational cost, based upon spatial mixture analysis. The segment weighted average image fusion combines several low spatial resolution data source from different sensors to create high resolution and large size of fused image. This research includes developing a segment-based step, based upon stepwise divide and combine process. In the second stage of the process, the linear interpolation optimization is used to sharpen the image resolution. Implementation of these image fusion algorithms are completed based on the graphic user interface we developed. Multiple sensor image fusion is easily accommodated by the algorithm, and the results are demonstrated at multiple scales. By using quantitative estimation such as mutual information, we obtain the experiment quantifiable results. We also use the image morphing technique to generate fused image sequence, to simulate the results of image fusion. While deploying our pixel level image fusion algorithm approaches, we observe several challenges from the popular image fusion methods. While high computational cost and complex processing steps of image fusion algorithms provide accurate fused results, they also makes it hard to become deployed in system and applications that require real-time feedback, high flexibility and low computation ability

abstract: Metaphor as a way of thinking permeates daily life. It affects how people understand and experience everything. It also plays an important role in artistic creation. The idea of creating highly personal but commonly understood metaphors was central to the research and creation of Ink. I created this work to find out how I—as a Chinese artist with unique personal experiences, educational experiences, and cultural perspectives—can explore metaphors that would resonate with predominantly Western audiences. This research specifically addressed the metaphorical meanings of the colors black and white and drew from my visual artistry to compose dances, stage setting, and costume design. / Dissertation/Thesis / Masters Thesis Dance 2017

This dissertation first conducts a study on raw-data level cooperative perception for enhancing the detection ability of self-driving systems for connected autonomous vehicles (CAVs). A LiDAR (Light Detection and Ranging sensor) point cloud-based 3D object detection method is deployed to enhance detection performance by expanding the effective sensing area, capturing critical information in multiple scenarios and improving detection accuracy. In addition, a point cloud feature based cooperative perception framework is proposed on edge computing system for CAVs. This dissertation also uses the features' intrinsically small size to achieve real-time edge computing, without running the risk of congesting the network. In order to distinguish small sized objects such as pedestrian and cyclist in 3D data, an end-to-end multi-sensor fusion model is developed to implement 3D object detection from multi-sensor data. Experiments show that by solving multiple perception on camera and LiDAR jointly, the detection model can leverage the advantages from high resolution image and physical world LiDAR mapping data, which leads the KITTI benchmark on 3D object detection. At last, an application of cooperative perception is deployed on edge to heal the live map for autonomous vehicles. Through 3D reconstruction and multi-sensor fusion detection, experiments on real-world dataset demonstrate that a high definition (HD) map on edge can afford well sensed local data for navigation to CAVs.

Real-time quality monitoring continues to gain interest within the manufacturing domain as new and faster sensors are being developed. Unfortunately, most quality monitoring solutions are still based on collecting data from the end product. From a process improvement point of view, it is definitely more advantageous to proactively monitor quality directly in the process instead of the product, so that the consequences of a defective part can be minimized or even eliminated. In this dissertation, new methods for in-line process monitoring are explored using multiple sensors. In the first case, a new cutting force-based monitoring methodology was developed to detect out of control conditions in a broaching operation. The second part of this thesis focusses on the development of a test bed for monitoring the tool condition in a turning operation. The constructed test bed includes the combination of multiple sensors signals including, temperature, vibrations, and energy measurements. Here, the proposed SPC strategy integrates sensor data with engineering knowledge to produce quick, reliable results using proven profile monitoring techniques. While, the already existing methods are based on raw process data which requires more features to monitor without any loss of information. This technique is straight forward and able to monitor the process comprehensively with less number of features. Consequently, this also adds to the group of tools that are available for the practitioner. / Master of Science

In dimensional metrology, contact and non-contact measurement methods each have their own respective strengths and weaknesses. Touch-trigger probes have low uncertainty, and perform well inside deep holes, but have a relatively slow data acquisition speed. By contrast, non-contact digitizers collect high density surface point clouds in seconds, and are much less likely to suffer from sensor collision with the part, but have a higher uncertainty than touch probes. In sheet metal forming, iterative design of the stamping die is needed due to the springback of the sheet metal part. Holes or other features of first article parts may be significantly out of tolerance, so the tactile measurement path created from the Computer Aided Design (CAD) nominal has to be adjusted to avoid cosine error. In more serious cases, probe collisions or missed touches may occur. When measuring holes in thin sheet metal, determination of the touch probe path height is also a challenge if the actual surface location differs from the nominal. To solve this problem and seize the complimentary advantages of contact and non-contact measurement methods, a multi-sensor blue Light Emitting Diode (LED) snapshot sensor and touch-trigger probe inspection system was developed, and affixed to a Coordinate Measuring Machine (CMM). The tactile measurement path was adjusted according to the approximate positions and sizes of the features obtained from the scanner data. The system includes an in-house designed calibration target for scanner calibration and a lightweight 2-axis rotary table for multiple-orientation scanning as well. Software in programming language C for interacting with the scanner and the CMM was developed. A sample stamped sheet metal automobile part was experimentally measured. This system is currently applied to an orthogonal CMM. Suggested future works include implementation on non-Cartesian CMMs, such as articulated arm CMMs, or Computer Numerical Control (CNC) machine tools. / Thesis / Master of Applied Science (MASc)

Sensors are getting smaller, inexpensive and sophisticated, with an increased availability. Compared to 25 years ago, an object tracking system now can easily achieve twice the accuracy, a much larger coverage and fault tolerance, without any significant changes in the overall cost. This is possible by simply employing more than just one sensor and processing measurements from individual sensors sequentially (or even in a batch form). %This is the centralized scheme of multi-sensor target tracking wherein the sensors send their individual detections to a central facility, where tracking related tasks such as data association, filtering, and track management etc. are performed. This is also perhaps the simplest solution for a multi-sensor approach and also optimal in the sense of minimum mean square error (MMSE) among all other multi-sensor scenario. In sophisticated sensors, the number of detections can reach thousands in a single frame. The communication and computation load for gathering all such detections at the fusion center will hamper the system's performance while also being vulnerable to faults. A better solution is a distributed architecture wherein the individual sensors are equipped with processing capabilities such that they can detect measurements, extract clutter, form tracks and transmit them to the fusion center. The fusion center now fuses tracks instead of measurements, due to which this scheme is commonly termed track-level fusion. In addition to sub-optimality, the track-level fusion suffers from a very coarse problem, which occurs due to correlations between the tracks to be fused. Often, in realistic scenarios, the cross-correlations are unknown, without any means to calculate them. Thus, fusion cannot be performed using traditional methods unless extra information is transmitted from the fusion center. This thesis proposes a novel and generalized method of fusing any two probability density functions (pdf) such that a positive cross-correlation exists between them. In modern tracking systems, the tracks are essentially pdfs and not necessarily Gaussian. We propose harmonic mean density based fusion and prove that it obeys all the necessary requirements of being a viable fusion mechanism. We show that fusion in this case is a classical example of agreement between the fused and participating densities based on average $\chi^2$ divergence. Compared to other such fusion techniques in the literature, the HMD performs exceptionally well. Transmitting covariance matrices in distributed architecture is not always possible in cases for e.g. tactical and automotive systems. Fusion of tracks without the knowledge of uncertainty is another problem discussed in the thesis. We propose a novel technique for local covariance reconstruction at the fusion center with the knowledge of estimates and a vector of times when update has occurred at local sensor node. It has been shown on a realistic scenario that the reconstructed covariance converges to the actual covariance, in the sense of Frobenius norm, making fusion without covariance, possible. / Thesis / Doctor of Philosophy (PhD)

Networked, multi-sensor array systems have proven to be advantageous in the sensor world. A large amount of research has been conducted with these systems, with a main interest in data fusion. Intelligently processing the large amounts of data collected by these systems is required in order to fully utilize the benefits of a multi-sensor array system. A robust but flexible simulation environment would provide a platform for accurately comparing current and future data fusion theories. This thesis proposes a simulator model for testing fusion theories for these acoustic multi-sensor networks. An iterative, lossless data fusion algorithm was presented as the model for simulation development. The arrangement and orientation of objects in the simulation environment, as well as most other system parameters are defined by the user before the simulation runs. The sensor data, including noise, is generated at the appropriate time delay and propagation loss before being processed by a delay and sum beamformer and a matched filter. The resulting range-Doppler maps are modified to probability density functions, and translated to a single point of reference. The data is then combined into a single world model. An iterative process is used to filter out false targets and amplify true target detections. Data is fused from each multi-sensor array and from each simulation run. Target amplitudes are gained if they are present in all combined world models, and are otherwise reduced. This thesis presents the results of the fusion algorithm used, including multiple iterations, to prove the algorithms effectiveness. / Master of Science

abstract: Modern intelligent transportation systems (ITS) make driving more efficient, easier, and safer. Knowledge of real-time traffic conditions is a critical input for operating ITS. Real-time freeway traffic state estimation approaches have been used to quantify traffic conditions given limited amount of data collected by traffic sensors. Currently, almost all real-time estimation methods have been developed for estimating laterally aggregated traffic conditions in a roadway segment using link-based models which assume homogeneous conditions across multiple lanes. However, with new advances and applications of ITS, knowledge of lane-based traffic conditions is becoming important, where the traffic condition differences among lanes are recognized. In addition, most of the current real-time freeway traffic estimators consider only data from loop detectors. This dissertation develops a bi-level data fusion approach using heterogeneous multi-sensor measurements to estimate real-time lane-based freeway traffic conditions, which integrates a link-level model-based estimator and a lane-level data-driven estimator. Macroscopic traffic flow models describe the evolution of aggregated traffic characteristics over time and space, which are required by model-based traffic estimation approaches. Since current first-order Lagrangian macroscopic traffic flow model has some unrealistic implicit assumptions (e.g., infinite acceleration), a second-order Lagrangian macroscopic traffic flow model has been developed by incorporating drivers’ anticipation and reaction delay. A multi-sensor extended Kalman filter (MEKF) algorithm has been developed to combine heterogeneous measurements from multiple sources. A MEKF-based traffic estimator, explicitly using the developed second-order traffic flow model and measurements from loop detectors as well as GPS trajectories for given fractions of vehicles, has been proposed which gives real-time link-level traffic estimates in the bi-level estimation system. The lane-level estimation in the bi-level data fusion system uses the link-level estimates as priors and adopts a data-driven approach to obtain lane-based estimates, where now heterogeneous multi-sensor measurements are combined using parallel spatial-temporal filters. Experimental analysis shows that the second-order model can more realistically reproduce real world traffic flow patterns (e.g., stop-and-go waves). The MEKF-based link-level estimator exhibits more accurate results than the estimator that uses only a single data source. Evaluation of the lane-level estimator demonstrates that the proposed new bi-level multi-sensor data fusion system can provide very good estimates of real-time lane-based traffic conditions. / Dissertation/Thesis / Doctoral Dissertation Industrial Engineering 2015