Back to search

Eastern Washington University

Evaluating LLMs for CPE Identification in IoT Reconnaissance

Abstract

dc:description.abstract

Vulnerability identification during penetration testing relies on rigid string-matching to map network scan data to Common Platform Enumeration (CPE) identifiers and downstream Common Vulnerabilities and Exposures (CVEs). The approach frequently fails on physical Internet of Things (IoT) devices, which produce non-standard, irregular service banners that resist deterministic parsing. Large Language Models can reason through these fuzzy associations, but cloud-hosted models introduce cost, latency, and operational security concerns when processing reconnaissance data from live networks. This thesis asks whether locally-hosted open-weight Large Language Models (LLMs) can perform this task well enough to be useful, and how performance varies with model scale, family, and prompt design. I evaluate 36 LLMs across three deployment tiers (frontier hosted APIs, Ollama Cloud, and local vLLM), plus Nmap’s native CPE extraction as the no-LLM baseline, on a curated corpus of Nmap covering 16 analyzed IoT device entries drawn from an 18-entry testbed: 14 physical entries and 4 derived entries, with one physical and one derived entry excluded for unusable scan or run data. Three findings shape the result. Model family is the dominant source of variance, with a wider spread than prompt design or scan-suite choice produces. The best truly-local model approaches frontier accuracy on a multi-GPU workstation but does not on single-GPU consumer hardware. The most operationally dangerous failure is not a malformed string but a confident, well-formed CPE that retrieves the wrong vulnerability list.

Degree

thesis:*
Name thesis:degree_name
Master of Science (MS) in Cyber Defense
Level thesis:degree_level
Thesis
Discipline thesis:degree_discipline
Computer Science and Electrical Engineering

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Davisson, Christopher
Contributors dc:contributor
  • Sanmeet Kaur
  • Stuart Steiner
  • Lynnae Daniels

Subjects

dc:subject × 7

Rights

dc:rights
Statement dc:rights
  • Access is available to all users

Identifiers

dc:identifier.*
Repository record dc:identifier
https://dc.ewu.edu/theses/1009

Chain of custody

source
Harvested from
Eastern Washington University
Base URL
dc.ewu.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Davisson, Christopher. Evaluating LLMs for CPE Identification in IoT Reconnaissance. Thesis thesis, https://dc.ewu.edu/theses/1009