Back to search

University of the Aegean

Effective detection of lateral movement in network security: leveraging Sysmon logs and machine learning

Abstract

dc:description

Attackers employ lateral movement (LM) to spread from an initial entry point, gain further network access, and ultimately exfiltrate valuable data. This process, particularly challenging on any computing platform, commonly involves pivoting through systems and escalating privileges. This PhD thesis offers a holistic methodology for detecting LM, leveraging a rich combination of comprehensive literature survey and extensive experimental results. Particularly, the detection is achieved through supervised and unsupervised machine learning (ML) techniques, applied effectively to both imbalanced and balanced data samples.This work makes a pivotal contribution to the field by not only leveraging but also creating the LMD dataset, which to our knowledge is the first to exclusively contain Sysmon logs detailing 15 diverse LM attack techniques captured from the MS Windows platform. The thesis presents a comprehensive overview of this specialized area and its associated methodologies, critically examining the potential of standard ML for detection. In detail, the research addresses five core questions pertaining to LM-detection. First, it investigates the appropriate methodology for conducting a comprehensive survey on LM identification, primarily from an Intrusion Detection System (IDS) perspective, to bridge a significant gap in existing literature. Second, it determines, based on expert analysis and relevant literature, the definitive criteria for establishing the most effective initial features for Sysmon's event monitoring tool, suitable for implementation as custom rules within its config.xml configuration file. Third, the thesis outlines how a functional configuration file can be systematically generated, utilizing these identified features, to maximize the identification of various LM attack variants. Fourth, it explores whether a proven methodology exists for the creation of ML-based LM-oriented IDSs, incorporating the evaluation of multiple supervised and unsupervised ML algorithms, including both shallow and Deep Neural Network (DNN) techniques. Finally, the research critically assesses the extent to which sample distribution within a benchmark dataset impacts the performance evaluation metrics of LM-oriented IDS models, irrespective of whether they employ shallow or DNN architectures. To answer these questions, we initially conducted a systematic survey of the topic. We then leveraged the MITRE ATT&CK knowledge base of adversary tactics and techniques, focusing initially on the execution of the nine most common LM methods. Experiments on a properly configured testbed revealed a significant number of interrelated networking features. We implemented these features as custom rules in Sysmon's config.xml file. Furthermore, we capitalized on the rich, LM-oriented corpus, LMD, which we created as the first of its kind specifically focusing on LM samples and enhanced with the execution of six more LM tactics. We elaborated on feature selection, data preprocessing, and feature importance processes for configuring both supervised (SML) and unsupervised (UML) ML models, employing both multiclass and binary classification schemes. Compared to relevant literature, and considering the highly unbalanced corpus and a multiclass classification problem, SML reported superior scores with F1-score and AUC metrics of 99.41% and 99.84%, respectively. The best UML performer also showed promising results, achieving around 94.7% AUC and 93% F1-score for the best shallow model, and 95.2% AUC and 93.8% F1-score for the best DNN model. To further improve these metrics, we devised and evaluated a two-stage ML model, which surpassed the previous best score by approximately 3.5%. Finally, we re-adopted a multiclass supervised approach to scrutinize the effects of oversampling, undersampling, and hybrid-sampling techniques across 13 ML algorithms (nine shallow and four DNN techniques) using the LMD corpus. Our key findings revealed that balanced versions of the dataset generally improved performance. Shallow models trained on resampled data achieved a marginal convergence of roughly +0.05% in AUC and F1-score compared to the imbalanced scenario. Notably, DNN models exhibited a more substantial performance gain of around 3.5% across most balancing techniques. Furthermore, analysis of false positive rate (FPR) and false negative rate (FNR) revealed crucial trade-offs. While some balanced datasets led to near-zero FNR with ensemble methods like Bagging, others, particularly with DNNs and techniques like ADASYN, showed a higher propensity for false alarms. These observations underscore the critical role of data balancing in optimizing LM-focused IDS performance and highlight the varying impact of different techniques on the FPR/FNR trade-off for shallow versus DNN models.

Degree

thesis:*
Grantor dc:publisher
University of the Aegean
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Smiliotopoulos, Christos
  • Σμηλιοτόπουλος, Χρήστος

Subjects

dc:subject × 23

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/59397

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Smiliotopoulos, Christos; Σμηλιοτόπουλος, Χρήστος. Effective detection of lateral movement in network security: leveraging Sysmon logs and machine learning. University of the Aegean, 2025. http://hdl.handle.net/10442/hedi/59397