Back to search

University of Sheffield

Beyond Words: Analyzing Social Media with Text and Images

Abstract

dc:description.abstract

People express their opinions and experiences through text and images in social media platforms. Analyzing social media content has several applications in natural language processing such as sentiment analysis, hate speech detection, fact checking and sarcasm detection. Combining text and images from social media posts is challenging due to weak visual-text relationships. For instance, a post with the text: Feeling on top of the world after acing my final exams! and a picture of a group of friends at the beach. The image and the text are weakly related as the image does not directly align with the academic context, potentially leading to confusion or misinterpretation of the intended message. Thus, effectively modeling text and images from social media posts is crucial for advancing natural language understanding. This thesis proposes a number of new challenging multimodal classification tasks: point-of-interest (POI) type prediction, political advertisements analysis, and influencer content analysis. First, we introduce POI type prediction which consists of inferring the type of location from which a social media message was posted such as a park or a restaurant. This task is relevant to study a place's identity and has applications such as POI visualization and recommendation. Second, we analyze political advertisements by introducing two new datasets containing political ads labeled by the sponsor's ideology (conservative, liberal), and the sponsor type (political party, third party); and we experiment with multimodal models for advertisement classification. Analyzing political ads is important for researching the characteristics of online campaigns (e.g. voter targeting, non-party campaigns and misinformation) on a large scale. Next, we perform an extensive analysis of influencer content including multimodal approaches for identifying commercial posts, i.e., content that is monetized. Automatically detecting influencer commercial posts is of utmost importance for addressing issues related to transparency and regulatory compliance, such as misleading advertising. Finally, this thesis also presents novel methods for tackling the challenges of modeling text and visual content in social media. We propose two auxiliary losses, Image-Text Contrastive which encourages the model to capture the underlying dependencies in multimodal posts; and Image-Text Matching to enable visual and language alignment.

Degree

thesis:*
Name dc:type.qualificationname
Ph.D
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Sheffield
Year dc:date.issued
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Sanchez Villegas, Danae
Advisor dc:contributor.advisor
  • Aletras, Nikolaos
ORCID person:identifier.orcid

Rights

Language dc:language.dcterms_iso639_2
eng

Identifiers

dc:identifier.*
Dc Identifier Grantnumber
EP/S023062/1

Chain of custody

source
Harvested from
White Rose University Consortium
Base URL
etheses.whiterose.ac.uk/cgi/oai2
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
related terms
citation

Sanchez Villegas, Danae. Beyond Words: Analyzing Social Media with Text and Images. doctoral thesis, University of Sheffield, 2023.