Back to search

Cornell University

Towards Comprehensive Visual Understanding

Abstract

dc:description.abstract

An image is worth a thousand words, conveying information that goes beyond the visual content therein. Traditional computer vision tasks focus on the recognition of tangible properties of images, such as objects and scenes. Relatively little attention has been paid to tasks that involve private states where subjectivity analysis is relevant. This area includes detecting cyberbullying and hate speech, identifying emotions, and understanding rhetoric and intentions. This dissertation presents our work in exploring new challenges and approaches towards comprehensive visual understanding, with both subjectivity and objectivity in images in mind. Specifically, on the challenge side, we focus on a specific aspect of subjectivity: the intent behind social media images. We introduce an intent dataset, Intentonomy, annotated with 28 intent categories derived from a social psychology taxonomy. We then systematically study whether, and to what extent, commonly used visual information, i.e., object and context, contribute to human intent understanding. On the approach side, we present three approaches: (1) an intent classifier that attends to object and context classes in images as well as textual information in the form of hashtags; (2) a streamlined pre-training method that uses pseudo labels derived from human responses to social media posts. (3) a parameter-efficient transfer learning method for adapting ever-increasing pre-trained vision models. We find our dataset to be very challenging for visual recognition systems and our approaches to be empirically effective on representative visual understanding tasks.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Information Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Information Science
Grantor
Cornell University
Year dc:date.issued
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Jia, Menglin
Chair dc:contributor.chair
  • Cardie, Claire
Committee members dc:contributor.committeemember
  • Belongie, Serge
  • Mimno, David

Subjects

dc:subject × 2

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
16176649
Dc Identifier Other
ProQuest Submission ID: 13469
ProQuest Publication ID: 30248533

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Jia, Menglin. Towards Comprehensive Visual Understanding. Doctor of Philosophy thesis, Cornell University, 2023. https://hdl.handle.net/1813/114059