Virginia Tech
Streamlining SOC Analyst Diagnosis Through Workflow Extraction from Demonstrations
Abstract
dc:description.abstractSecurity Operations Center (SOC) analysts triage SIEM alerts and identify which alerts belong to which kind of attack, a bottleneck in incident response since alert volume far ex- ceeds what analysts can read. Hand-authored detection rules are brittle; single-shot Large Language Model (LLM) calls are non-deterministic and unauditable. This thesis introduces demonstration-driven workflow generation (DDW): a four-step pipeline (demonstration → chat log → distillation → executable workflow) that converts one analyst's investigative rea- soning into a deterministic, reusable detector under an input policy that excludes labels and phase names. The analyst uses deterministic analytical tools (rule-frequency aggregation, record sampling, full-text inspection, MITRE ATTandCK lookup) to identify the attack phase of an anonymised log subset once; the chat log is distilled into a three-node JSON workflow (aggregate, sample, classify) that runs on new scenarios with one parameter. On eight AIT- ADS scenarios and a 136-cell matrix totalling 2.6 million Wazuh records, DDW achieves macro-F1 = 0.971 on the two demonstrable phases (dirb, wpscan), cell accuracy 99.3%, and Wilson 95 % CI [67.6%, 100.0%]. On the same two phases Sigma reaches 0.733 and ReAct reaches 0.899; a within-method ablation isolates the distillation pathway as decisive (agent- loop 0.971 vs single-shot 0.322). DDW also surfaces demonstrability as a dataset-level signal property: phases on which demonstration fails are the same phases on which both reference baselines underperform.
Degree
thesis:*- Name thesis:degree_name
- Master of Science
- Level thesis:degree_level
- masters
- Discipline thesis:degree_discipline
- Computer Science & Applications
- Department dc:contributor.department
- Computer Science and#38; Applications
- Grantor dc:publisher
- Virginia Tech
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Yadav, Siddharth
- Chairs dc:contributor.committeechair
-
- MATSUO, SHINICHIRO
- Cameron, Melissa
- Committee member dc:contributor.committeemember
-
- Meng, Na
Subjects
dc:subject × 7Rights
dc:rights- Statement dc:rights
-
- In Copyright
- Licence dc:rights.uri
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Other
- vt_gsexam:47301