Virginia Tech
Scam Shield: A Hierarchical Framework for Robust and Efficient Adversarial Scam Detection
Abstract
dc:description.abstractCan we trust large language models (LLMs) to accurately detect scams? This thesis investigates the vulnerabilities of LLMs when facing adversarial scam messages. We created a dataset with fine-grained labels, including original scam messages, adversarial scam messages, and non-scam messages. These labels allow us to examine detection performance across different scam types in addition to the binary classification task. We evaluated GPT, Claude, and LLaMA on this dataset. Under the selected rewriting conditions, the evaluated models miss scam messages even when the deceptive intent is preserved. To address this problem, we propose a Hierarchical Scam Detection System (HSDS). The system combines four lightweight classifiers, Decision Tree, XGBoost, Random Forest, and K-Nearest Neighbors, with a fine-tuned LLaMA 3.1 8B Instruct model. The classifiers first predict whether each message is a scam. When all four agree, the system returns their prediction. When they disagree, the message is sent to the language model. We fine-tuned this model with Low-Rank Adaptation (LoRA) using a domain-specific adversarial dataset. The fine-tuned model achieved higher detection scores with zero-shot prompting than the evaluated few-shot baselines. The full HSDS achieved an accuracy of 0.90 and an F1-score of 0.90, with higher scores than the evaluated proprietary LLM baselines. It also reduced average inference time by approximately 56.7% compared with the fine-tuned model alone. Romance scams remained difficult to detect. Further work could examine whether semantic and contextual features improve detection of these emotionally manipulative messages.
Degree
thesis:*- Name thesis:degree_name
- Master of Science
- Level thesis:degree_level
- masters
- Discipline thesis:degree_discipline
- Computer Science & Applications
- Department dc:contributor.department
- Computer Science and#38; Applications
- Grantor dc:publisher
- Virginia Tech
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Chang, Chen-Wei
- Chair dc:contributor.committeechair
-
- Lu, Chang Tien
- Committee members dc:contributor.committeemember
-
- Cho, Jin-Hee
- Purohit, Hemant
Subjects
dc:subject × 5Rights
dc:rights- Statement dc:rights
-
- In Copyright
- Licence dc:rights.uri
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Other
- vt_gsexam:48066