Modern semiconductor manufacturing is a complex process with a multitude of software applications. This application landscape has to be constantly monitored, since the communication and access patterns provide important insights. Because of the high event rates of the equipment log data stream in modern factories, big-data tools are required for scalable state and history analytics. The choice of suitable big-data solutions and their technical realization remains a challenging task.
This thesis compares big-data architectures and discovers solutions for log-data ingest, enrichment, analytics and visualization. Based on the use cases and requirements of developers working in this field, a comparison of a custom assembled stack and a complete solution is made. Since the complete stack is a preferable solution, Datadog, Grafana Loki and the Elastic 8 Stack are selected for a more detailed study. These three systems are implemented and compared based on the requirements. All three systems are well suited for big-data logging and fulfill most of the requirements, but show different capabilities when implemented and used.:1 Introduction
1.1 Motivation
1.2 Structure
2 Fundamentals and Prerequisites
2.1 Logging
2.1.1 Log level
2.1.2 CSFW log
2.1.3 SECS log
2.2 Existing system and data
2.2.1 Production process
2.2.2 Log data in numbers
2.3 Requirements
2.3.1 Functional requirements
2.3.2 System requirements
2.3.3 Quality requirements
2.4 Use Cases
2.4.1 Finding specific communication sequence
2.4.2 Watching system changes
2.4.3 Comparison with expected production path
2.4.4 Enrichment with metadata
2.4.5 Decoupled log analysis
3 State of the Art and Potential Software Stacks
3.1 State of the art software stacks
3.1.1 IoT flow monitoring system
3.1.2 Big-Data IoT monitoring system
3.1.3 IoT Cloud Computing Stack
3.1.4 Big-Data Logging Architecture
3.1.5 IoT Energy Conservation System
3.1.6 Similarities of the architectures
3.2 Selection of software stack
3.2.1 Components for one layer
3.2.2 Software solutions for the stack
4 Analysis and Implementation
4.1 Full stack vs. a custom assembled stack
4.1.1 Drawbacks of a custom assembled stack
4.1.2 Advantages of a complete solution
4.1.3 Exclusion of a custom assembled stack
4.2 Selection of full stack solutions
4.2.1 Elastic vs. Amazon
4.2.2 Comparison of Cloud-Only-Solutions
4.2.3 Comparison of On-Premise-Solutions
4.3 Implementation of selected solutions
4.3.1 Datadog
4.3.2 Grafana Loki Stack
4.3.3 Elastic 8 Stack
5 Comparison
5.1 Comparison of components
5.1.1 Collection
5.1.2 Analysis
5.1.3 Visualization
5.2 Comparison of requirements
5.2.1 Functional requirements
5.2.2 System requirements
5.2.3 Quality requirements
5.3 Results
6 Conclusion and Future Work
6.1 Conclusion
6.2 Future Work / Die moderne Halbleiterfertigung ist ein komplexer Prozess mit einer Vielzahl von Softwareanwendungen. Diese Anwendungslandschaft muss ständig überwacht werden, da die Kommunikations- und Zugriffsmuster wichtige Erkenntnisse liefern. Aufgrund der hohen Ereignisraten des Logdatenstroms der Maschinen in modernen Fabriken werden Big-Data-Tools für skalierbare Zustands- und Verlaufsanalysen benötigt. Die Auswahl geeigneter Big-Data-Lösungen und deren technische Umsetzung ist eine anspruchsvolle Aufgabe.
Diese Arbeit vergleicht Big-Data-Architekturen und untersucht Lösungen für das Sammeln, Anreicherung, Analyse und Visualisierung von Log-Daten. Basierend auf den Use Cases und den Anforderungen von Entwicklern, die in diesem Bereich arbeiten, wird ein Vergleich zwischen einem individuell zusammengestellten Stack und einer Komplettlösung vorgenommen. Da die Komplettlösung vorteilhafter ist, werden Datadog, Grafana Loki und der Elastic 8 Stack für eine genauere Untersuchung ausgewählt. Diese drei Systeme werden auf der Grundlage der Anforderungen implementiert und verglichen. Alle drei Systeme eignen sich gut für Big-Data-Logging und erfüllen die meisten Anforderungen, zeigen aber unterschiedliche Fähigkeiten bei der Implementierung und Nutzung.:1 Introduction
1.1 Motivation
1.2 Structure
2 Fundamentals and Prerequisites
2.1 Logging
2.1.1 Log level
2.1.2 CSFW log
2.1.3 SECS log
2.2 Existing system and data
2.2.1 Production process
2.2.2 Log data in numbers
2.3 Requirements
2.3.1 Functional requirements
2.3.2 System requirements
2.3.3 Quality requirements
2.4 Use Cases
2.4.1 Finding specific communication sequence
2.4.2 Watching system changes
2.4.3 Comparison with expected production path
2.4.4 Enrichment with metadata
2.4.5 Decoupled log analysis
3 State of the Art and Potential Software Stacks
3.1 State of the art software stacks
3.1.1 IoT flow monitoring system
3.1.2 Big-Data IoT monitoring system
3.1.3 IoT Cloud Computing Stack
3.1.4 Big-Data Logging Architecture
3.1.5 IoT Energy Conservation System
3.1.6 Similarities of the architectures
3.2 Selection of software stack
3.2.1 Components for one layer
3.2.2 Software solutions for the stack
4 Analysis and Implementation
4.1 Full stack vs. a custom assembled stack
4.1.1 Drawbacks of a custom assembled stack
4.1.2 Advantages of a complete solution
4.1.3 Exclusion of a custom assembled stack
4.2 Selection of full stack solutions
4.2.1 Elastic vs. Amazon
4.2.2 Comparison of Cloud-Only-Solutions
4.2.3 Comparison of On-Premise-Solutions
4.3 Implementation of selected solutions
4.3.1 Datadog
4.3.2 Grafana Loki Stack
4.3.3 Elastic 8 Stack
5 Comparison
5.1 Comparison of components
5.1.1 Collection
5.1.2 Analysis
5.1.3 Visualization
5.2 Comparison of requirements
5.2.1 Functional requirements
5.2.2 System requirements
5.2.3 Quality requirements
5.3 Results
6 Conclusion and Future Work
6.1 Conclusion
6.2 Future Work
| Identifer | oai:union.ndltd.org:DRESDEN/oai:qucosa:de:qucosa:82088 |
| Date | 11 November 2022 |
| Creators | Tiede, David |
| Contributors | Dienel, René, Habich, Dirk, Technische Universität Dresden |
| Source Sets | Hochschulschriftenserver (HSSS) der SLUB Dresden |
| Language | English |
| Detected Language | English |
| Type | info:eu-repo/semantics/acceptedVersion, doc-type:masterThesis, info:eu-repo/semantics/masterThesis, doc-type:Text |
| Rights | info:eu-repo/semantics/openAccess |
Page generated in 0.0026 seconds