No description available.

<div> <div> <div> <p>Billions of “smart” things in our lives have been equipped with various sensors. Current devices, such as smartphones, smartwatches, tablets, and VR/AR headsets, are equipped with a variety of embedded sensors, e.g. accelerometer, gyroscope, magnetometer, camera, GPS sensor, etc. Based on these sensor data, many technologies have been developed to track human motion at different granularities and to enable new applications. This dissertation examines two challenging problems in human motion tracking. One problem is the ID association issue when utilizing external sensors to simultaneously track multiple people. Although an “outside” system can track all human movements in a designated area, it needs to digitally associate each tracking trajectory to the corresponding person, or say the smart device carried by that person, to provide customized service based on the tracking results. Another problem is the inaccuracy caused by limited sensing information when merely using the embedded sensors located on the devices being tracked. Since sensor data may contain inevitable noises and there is no external beacon used as a reference point for calibration, it is hard to accurately track human motion only with internal sensors.</p><p>In this dissertation, we focus on applying multimodal sensing to perform human motion tracking in mobile systems. To address the two above problems separately, we conduct the following research works. (1) The first work seeks to enable public cameras to send personalized messages to people without knowing their phone addresses. We build a system which utilizes the users’ motion patterns captured by the cameras as their communication addresses, and depends on their smartphones to locally compare the sensor data with the addresses and to accept the correct messages. To protect user privacy, the system requires no data from the users and transforms the motion patterns into low-dimensional codes to prevent motion leaks. (2) To enhance distinguishability and scalability of the camera-to-human communication system, we introduce context features which include both motion patterns and ambience features (e.g. magnetic field, Wi-Fi fingerprint, etc.) to identify people. The enhanced system achieves higher association accuracy and is demonstrated to work with dense people in a retailer, with a fixed-length packet overhead. The first two works explore the potential of widely deployed surveillance cameras and provide a generic underlay to various practical applications, such as automatic audio guide, indoor localization, and sending safety alerts. (3) We close this dissertation with a fine-grained motion tracking system which aims to track the positions of two hand-held motion controllers in a mobile VR system. To achieve high tracking accuracy without external sensors, we introduce new types of information, e.g. ultrasonic ranging among the headset and the controllers, and a kinematic arm model. Effectively fusing this additional information with inertial sensing generates accurate controller positions in real time. Compared with commodity mobile VR controllers which only support rotational tracking, our system provides an interactive VR experience by letting the user actually move the controllers’ positions in a VR scene. To summarize, this dissertation shows that multimodal sensing can further explore the potential power in sensor data and can take sensor-based applications to the next generation of innovation.</p><div><br></div></div></div></div><div><div><div> </div> </div> </div>

Although sensor planning in computer vision has been a subject of research for over two decades, a vast majority of the research seems to concentrate on two particular applications in a rather limited context of laboratory and industrial workbenches, namely 3D object reconstruction and robotic arm manipulation. Recently, increasing interest is engaged in research to come up with solutions that provide wide-area autonomous surveillance systems for object characterization and situation awareness, which involves portable, wireless, and/or Internet connected radar, digital video, and/or infrared sensors. The prominent research problems associated with multisensor integration for wide-area surveillance are modality selection, sensor planning, data fusion, and data exchange (communication) among multiple sensors. Thus, the requirements and constraints to be addressed include far-field view, wide coverage, high resolution, cooperative sensors, adaptive sensing modalities, dynamic objects, and uncontrolled environments. This article summarizes a new survey and analysis conducted in light of these challenging requirements and constraints. It involves techniques and strategies from work done in the areas of sensor fusion, sensor networks, smart sensing, Geographic Information Systems (GIS), photogrammetry, and other intelligent systems where finding optimal solutions to the placement and deployment of multimodal sensors covering a wide area is important. While techniques covered in this survey are applicable to many wide-area environments such as traffic monitoring, airport terminal surveillance, parking lot surveillance, etc., our examples will be drawn mainly from such applications as harbor security and long-range face recognition.

<p>Mobile systems, such as smartphones, wearables (e.g., smartwatches, AR/VR headsets),<br> and IoT devices, have come a long way from being just a method of communication to<br> sophisticated sensing devices that monitor and control several aspects of our lives. These<br> devices have enabled several useful applications in a wide range of domains ranging from<br> healthcare and finance to energy and agriculture industries. While such advancement has<br> enabled applications in several aspects of human life, it has also made these devices an<br> interesting target for adversaries.<br> In this dissertation, I specifically focus on how the various sensors on mobile devices can<br> be exploited by adversaries to violate users’ privacy and present methods to use sensors<br> to improve the security of these devices. My thesis posits that multi-modal sensing can be<br> leveraged to enhance the security and privacy of mobile systems.<br> In this, first, I describe my work that demonstrates that human interaction with mobile de-<br> vices and their accessories (e.g., stylus pencils) generates identifiable patterns in permissionless<br> mobile sensors’ data, which reveal sensitive information about users. Specifically, I developed<br> S3 to show how embedded magnets in stylus pencils impact the mobile magnetometer sensor<br> and can be exploited to infer a users incredibly private handwriting. Then, I designed LocIn<br> to infer a users indoor semantic location from 3D spatial data collected by mixed reality<br> devices through LiDAR and depth sensors. These works highlight new privacy issues due to<br> advanced sensors on emerging commodity devices.<br> Second, I present my work that characterizes the threats against smartphone authentication<br> and IoT device pairing and proposes usable and secure methods to protect against these threats.<br> I developed two systems, FaceRevelio and IoTCupid, to enable reliable and secure user and<br> device authentication, respectively, to protect users’ private information (e.g., contacts,<br> messages, credit card details) on commodity mobile and allow secure communication between<br> IoT devices. These works enable usable authentication on diverse mobile and IoT devices<br> and eliminate the dependency on sophisticated hardware for user-friendly authentication.</p>