Back to search

Georgia Institute of Technology

Multimodal assessment of neuropsychiatric disorders using audiovisual recordings

Abstract

dc:description.abstract

Over one billion people worldwide live with a neuropsychiatric disorder, yet most do not have access to adequate diagnosis and care. Accurate, fast, and accessible detection of those disorders is critical to early and effective interventions. Over the last decade, digitally administered assessments have emerged as one of the most promising approaches. Moreover, the increasing use of telemedicine in psychiatry and neurology in recent years presented an unprecedented opportunity to use audiovisual data for accessible neuropsychiatric assessments without the limitation of geographical location and specialized hardware. This dissertation describes the use of low-cost audiovisual data collected from in-lab and remote mobile devices to assess neuropsychiatric conditions by extracting and combining various behavioral and physiological indicators. First, we showed that facial and speech emotions can be effectively estimated from audiovisual data collected in interviews and used for major depressive disorder evaluation. Then, we presented the automated assessment of cognitive impairment using facial emotions and viewing behaviors recognized from videos passively collected from a mobile device. Lastly, we further improved the scalability and accessibility by extracting facial, vocal, linguistic, and cardiovascular features from audiovisual data collected remotely from heterogeneous mobile devices and validating them in both clinician-rated and self-rated mental health condition evaluation.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Biomedical Engineering (Joint GT/Emory Department)
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Jiang, Zifan
Advisor dc:contributor.advisor
  • Clifford, Gari D.
Committee members dc:contributor.committeemember
  • Mitchell, Cassie S.
  • Dyer, Eva L.
  • Haider, Bilal
  • Bahrami Rad, Ali

Subjects

dc:subject × 10

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/76786

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Jiang, Zifan. Multimodal assessment of neuropsychiatric disorders using audiovisual recordings. Doctoral thesis, Georgia Institute of Technology, 2023. https://hdl.handle.net/1853/76786