Eastern Washington University
Evaluating LLMs for CPE Identification in IoT Reconnaissance
Abstract
dc:description.abstractVulnerability identification during penetration testing relies on rigid string-matching to map network scan data to Common Platform Enumeration (CPE) identifiers and downstream Common Vulnerabilities and Exposures (CVEs). The approach frequently fails on physical Internet of Things (IoT) devices, which produce non-standard, irregular service banners that resist deterministic parsing. Large Language Models can reason through these fuzzy associations, but cloud-hosted models introduce cost, latency, and operational security concerns when processing reconnaissance data from live networks. This thesis asks whether locally-hosted open-weight Large Language Models (LLMs) can perform this task well enough to be useful, and how performance varies with model scale, family, and prompt design. I evaluate 36 LLMs across three deployment tiers (frontier hosted APIs, Ollama Cloud, and local vLLM), plus Nmap’s native CPE extraction as the no-LLM baseline, on a curated corpus of Nmap covering 16 analyzed IoT device entries drawn from an 18-entry testbed: 14 physical entries and 4 derived entries, with one physical and one derived entry excluded for unusable scan or run data. Three findings shape the result. Model family is the dominant source of variance, with a wider spread than prompt design or scan-suite choice produces. The best truly-local model approaches frontier accuracy on a multi-GPU workstation but does not on single-GPU consumer hardware. The most operationally dangerous failure is not a malformed string but a confident, well-formed CPE that retrieves the wrong vulnerability list.
Degree
thesis:*- Name thesis:degree_name
- Master of Science (MS) in Cyber Defense
- Level thesis:degree_level
- Thesis
- Discipline thesis:degree_discipline
- Computer Science and Electrical Engineering
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Davisson, Christopher
- Contributors dc:contributor
-
- Sanmeet Kaur
- Stuart Steiner
- Lynnae Daniels
Subjects
dc:subject × 7Rights
dc:rights- Statement dc:rights
-
- Access is available to all users
- Licence dc:rights.license
Identifiers
dc:identifier.*- Repository record dc:identifier
- https://dc.ewu.edu/theses/1009