Back to search

Virginia Tech

Scam Shield: A Hierarchical Framework for Robust and Efficient Adversarial Scam Detection

Abstract

dc:description.abstract

Can we trust large language models (LLMs) to accurately detect scams? This thesis investigates the vulnerabilities of LLMs when facing adversarial scam messages. We created a dataset with fine-grained labels, including original scam messages, adversarial scam messages, and non-scam messages. These labels allow us to examine detection performance across different scam types in addition to the binary classification task. We evaluated GPT, Claude, and LLaMA on this dataset. Under the selected rewriting conditions, the evaluated models miss scam messages even when the deceptive intent is preserved. To address this problem, we propose a Hierarchical Scam Detection System (HSDS). The system combines four lightweight classifiers, Decision Tree, XGBoost, Random Forest, and K-Nearest Neighbors, with a fine-tuned LLaMA 3.1 8B Instruct model. The classifiers first predict whether each message is a scam. When all four agree, the system returns their prediction. When they disagree, the message is sent to the language model. We fine-tuned this model with Low-Rank Adaptation (LoRA) using a domain-specific adversarial dataset. The fine-tuned model achieved higher detection scores with zero-shot prompting than the evaluated few-shot baselines. The full HSDS achieved an accuracy of 0.90 and an F1-score of 0.90, with higher scores than the evaluated proprietary LLM baselines. It also reduced average inference time by approximately 56.7% compared with the fine-tuned model alone. Romance scams remained difficult to detect. Further work could examine whether semantic and contextual features improve detection of these emotionally manipulative messages.

Degree

thesis:*
Name thesis:degree_name
Master of Science
Level thesis:degree_level
masters
Discipline thesis:degree_discipline
Computer Science & Applications
Department dc:contributor.department
Computer Science and#38; Applications
Grantor dc:publisher
Virginia Tech
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Chang, Chen-Wei
Chair dc:contributor.committeechair
  • Lu, Chang Tien
Committee members dc:contributor.committeemember
  • Cho, Jin-Hee
  • Purohit, Hemant

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • In Copyright
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
vt_gsexam:48066

Chain of custody

source
Harvested from
Virginia Tech
Base URL
vtechworks.lib.vt.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Chang, Chen-Wei. Scam Shield: A Hierarchical Framework for Robust and Efficient Adversarial Scam Detection. masters thesis, Virginia Tech, 2026. https://hdl.handle.net/10919/143986