Back to search

Virginia Tech

Streamlining SOC Analyst Diagnosis Through Workflow Extraction from Demonstrations

Abstract

dc:description.abstract

Security Operations Center (SOC) analysts triage SIEM alerts and identify which alerts belong to which kind of attack, a bottleneck in incident response since alert volume far ex- ceeds what analysts can read. Hand-authored detection rules are brittle; single-shot Large Language Model (LLM) calls are non-deterministic and unauditable. This thesis introduces demonstration-driven workflow generation (DDW): a four-step pipeline (demonstration → chat log → distillation → executable workflow) that converts one analyst's investigative rea- soning into a deterministic, reusable detector under an input policy that excludes labels and phase names. The analyst uses deterministic analytical tools (rule-frequency aggregation, record sampling, full-text inspection, MITRE ATTandCK lookup) to identify the attack phase of an anonymised log subset once; the chat log is distilled into a three-node JSON workflow (aggregate, sample, classify) that runs on new scenarios with one parameter. On eight AIT- ADS scenarios and a 136-cell matrix totalling 2.6 million Wazuh records, DDW achieves macro-F1 = 0.971 on the two demonstrable phases (dirb, wpscan), cell accuracy 99.3%, and Wilson 95 % CI [67.6%, 100.0%]. On the same two phases Sigma reaches 0.733 and ReAct reaches 0.899; a within-method ablation isolates the distillation pathway as decisive (agent- loop 0.971 vs single-shot 0.322). DDW also surfaces demonstrability as a dataset-level signal property: phases on which demonstration fails are the same phases on which both reference baselines underperform.

Degree

thesis:*
Name thesis:degree_name
Master of Science
Level thesis:degree_level
masters
Discipline thesis:degree_discipline
Computer Science & Applications
Department dc:contributor.department
Computer Science and#38; Applications
Grantor dc:publisher
Virginia Tech
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Yadav, Siddharth
Chairs dc:contributor.committeechair
  • MATSUO, SHINICHIRO
  • Cameron, Melissa
Committee member dc:contributor.committeemember
  • Meng, Na

Subjects

dc:subject × 7

Rights

dc:rights
Statement dc:rights
  • In Copyright
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
vt_gsexam:47301

Chain of custody

source
Harvested from
Virginia Tech
Base URL
vtechworks.lib.vt.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Yadav, Siddharth. Streamlining SOC Analyst Diagnosis Through Workflow Extraction from Demonstrations. masters thesis, Virginia Tech, 2026. https://hdl.handle.net/10919/143214