Back to search

Georgia Institute of Technology

Multi-method data augmentation for enhancing LLM performance in low-resource aerospace NER tasks

Abstract

dc:description.abstract

As digital transformation progresses across various sectors, systems engineering is also transitioning from document-based practices to Model-Based Systems Engineering (MBSE). Aligned with this transition, there is a growing need for Named Entity Recognition (NER) methods capable of extracting machine-readable entities from requirements written in natural language (NL) due to their usefulness in identifying and classifying data belonging to target entity types. Among the various approaches developed for NER, methods based on Large Language Model (LLM) have recently gained prominence due to the emergence of versatile pre-trained LLMs. Since these pre-trained LLMs are trained on vast amount of general data, recent researches accordingly focus on fine-tuning these models for domain-specific tasks such as NER for aerospace requirements. However, fine-tuning LLMs for domain-specific NER tasks is often hindered by challenges associated with data preparation. First, some domains lack sufficient open-source data due to concerns related to security or intellectual property. Another challenge arises from the labor-intensive pre-processing, where every token in the dataset must be paired with a corresponding entity label (token-label pairing). Aerospace requirements clearly illustrate both challenges: their confidential nature limits data availability, and the annotation process demands not only extensive effort but also domain expertise to ensure accurate token-label pairing when dealing with complex systems and domain specific jargon. To address these challenges, data augmentation presents a solution that is both promising and cost-efficient by generating synthetic data without requiring additional data collection or manual annotation. Among the commonly adopted augmentation methods, Synonym Replacement (SR) and Label-wise Token Replacement (LwTR) were selected for this study due to their compatibility with the fine-tuning process of LLMs. To explain, these methods introduce semantic variability at the token level, which is essential for improving generalization while preserving the token-to-entity label structure required for NER fine-tuning. However, even with these methods, improving domain-specific NER performance remains challenging. This is because the optimal level of semantic variability and the effectiveness of each augmentation method vary across entity types. This highlights the need for a more adaptive strategy that accounts for entity-specific characteristics during augmentation. Moreover, these differences among entity types often make it difficult for a single augmentation method to provide consistent and sufficient performance gains across all entities. As the data becomes more complex with multiple target entity types, relying on a single method increases the risk of suboptimal improvement for certain entities if its linguistic transformation style is not well-suited to those entities. This underscores the need to integrate multiple augmentation techniques to ensure adequate semantic diversity and effective learning across all target entity types during fine-tuning. To maximize the benefits of data augmentation, this study proposes a multi-method, entity-adaptive augmentation strategy aimed at enhancing the performance of fine-tuned LLMs without increasing manual labeling overhead. The strategy integrates SR and LwTR by identifying optimal replacement rate configurations for each entity type, informed by a detailed analysis of entity-wise performance trends under different augmentation settings. This approach enables fine-grained control over variability by aligning the augmentation method and replacement ratio with the linguistic characteristics of each entity type. Through this tailored combination of augmentation techniques, the resulting synthetic datasets are better suited to improving the representational diversity and generalization capabilities of the model, ultimately leading to enhanced NER performance in the target domain. Empirical evaluations conducted on aerospace requirements demonstrated the effectiveness of the proposed approach. In terms of overall weighted F1 score, single-method augmentation using SR and LwTR yielded performance gains of 5.99% and 5.73%, respectively, relative to the baseline model. Notably, the multi-method strategy achieved an additional improvement of 2.12% beyond the best-performing single-method result. These results validate the advantage of adapting augmentation strategies to domain-specific entity characteristics and confirm the complementary effects of combining multiple techniques. The proposed method thus provides a cost-effective and scalable solution for improving NER performance in low-resource settings such as aerospace requirements, supporting the structured extraction and classification of natural language specifications into machine-readable formats suitable for MBSE workflows.

Degree

thesis:*
Level thesis:degree_level
Masters
Department dc:contributor.department
Aerospace Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Cho, Kwangmin
Advisor dc:contributor.advisor
  • Mavris, Dimitri N.
Committee members dc:contributor.committeemember
  • Kennedy, Graeme J.
  • Fischer, Olivia J.
  • Sung, Woongje

Subjects

dc:subject × 5

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/78701

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Cho, Kwangmin. Multi-method data augmentation for enhancing LLM performance in low-resource aerospace NER tasks. Masters thesis, Georgia Institute of Technology, 2025. https://hdl.handle.net/1853/78701