In manufacturing, uptime and system reliability are paramount, placing high demands on automation technologies such as robotic systems. Failures in these systems cause considerable disruptions and incur significant costs. Traditional troubleshooting methods require extensive manual analysis by experts of log files, system data, application information, and problem descriptions. This process is labor-intensive and time-consuming, often resulting in prolonged downtimes and increased customer dissatisfaction, leading to heavy financial losses for companies. This research explores the application of Large Language Models (LLMs) like MistralLite and Mixtral-8*7B to automate root cause prediction from log data. We employed various fine-tuning methods, including full fine-tuning, Low-Rank Adaptation (LoRA), and Quantized Low Rank Adaptation (QLoRA), on these decoder-only models. Beyond using perplexity as an evaluation metric, the study incorporates GPT-4 as-a-judge to assess model performance. Additionally, the research uses complex prompting techniques to aid in the extraction of root causes from problem descriptions using GPT-4 and utilizes vector embeddings to analyze the importance of features in root cause prediction. The findings demonstrate that LLMs, when fine-tuned, can assist in identifying root causes from log data, with the smaller MistralLite model showing superior performance compared to the larger Mixtral model, challenging the notion that larger models are inherently better. The results also indicate that different training adaptations yield varied effectiveness, with QLoRA adaptation performing best for MistralLite and full fine-tuning proving most effective for Mixtral. This suggests that a tailored approach to model adaptation is necessary for optimal performance. Additionally, employing GPT-4 with Chain of Thought (CoT) prompting has demonstrated the capability to extract reasonable root causes from solved issues using this technique. The analysis of feature vector embeddings provides insights into the significant features, enhancing our understanding of the underlying patterns and relationships in the data.
| Identifer | oai:union.ndltd.org:UPSALLA1/oai:DiVA.org:liu-205285 |
| Date | January 2024 |
| Creators | Mandakath Gopinath, Aswath |
| Publisher | Linköpings universitet, Statistik och maskininlärning |
| Source Sets | DiVA Archive at Upsalla University |
| Language | English |
| Detected Language | English |
| Type | Student thesis, info:eu-repo/semantics/bachelorThesis, text |
| Format | application/pdf |
| Rights | info:eu-repo/semantics/openAccess |
Page generated in 0.0018 seconds