Back to search

Georgia Institute of Technology

Analyzing Health-related Behaviors Using First-person Vision

Abstract

dc:description.abstract

Wearable sensors are increasingly affordable and easy to deploy, and as a result they are widely-used in mobile health applications. Sensors such as accelerometers and gyroscopes are broadly-used due to their low cost, small size, and relatively low power consumption. More recently, it has become feasible to deploy wearable cameras in mobile health applications, with two primary benefits: 1) when cameras are head-mounted, they can capture the health-related visual attention of the participant which is difficult to measure via other sensing modalities; 2) cameras can capture health-related behaviors under a wide range of conditions and contexts. When cameras are combined with other wearable sensors, the video can be used to curate labeled examples of behaviors under field conditions, and thereby drive the development of machine learning-based models, provided that the sensor data is time-synchronized. The goal of this thesis is to develop methods for analyzing video from wearable cameras, an area known as egocentric vision, that can enable both automatically-derived measures of behavior and the automatic synchronization of video with other sensing modalities in order to facilitate the large scale collection of labelled training data for use in building machine learning models. I begin by describing a model for egocentric action recognition which leverages the shared motion and appearance properties of different actions to enable zero shot learning (the model can predict novel actions that it has not been trained on.) Second, I develop a method to analyze video from a head-worn camera and quantify the participant's attention to screens (e.g. monitors, smart phones, etc.), without using an eye-tracker. Finally, I present a method based on a weighted kernel density estimation approach to automatically synchronize the timestamps of a wearable camera and wearable accelerometer. The method is able to estimate the time offset between multiple modalities of sensor data collected from devices mounted on different locations in the natural field environment, without introducing additional participant burden.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Electrical and Computer Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2021

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Zhang, Yun
Advisors dc:contributor.advisor
  • Clements, Mark A.
  • Rehg, James M.
Committee members dc:contributor.committeemember
  • Inan, Omer T.
  • Abowd, Gregory D.
  • Ploetz, Thomas
  • Alshurafa, Nabil

Subjects

dc:subject × 5

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
http://hdl.handle.net/1853/65031

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Zhang, Yun. Analyzing Health-related Behaviors Using First-person Vision. Doctoral thesis, Georgia Institute of Technology, 2021. http://hdl.handle.net/1853/65031