Democritus University of Thrace (DUTH)
Innovative technologies for smart production: augmented reality, computer vision and machine learning in manufacturing maintenance and waste segregation
Abstract
dc:descriptionThe Fourth Industrial Revolution, also known as ''Industry 4.0'' or ''I4.0'', marked the beginning of the 21st century with the digitalisation of industrial processes and the transformation of corporations. Recently, the European Commission introduced the pillars of ''Industry 5.0'' or ''I5.0'' that promote sustainability, human-centricity, and resilience in digitised intelligent plants. The new I5.0 ecosystem makes use of the technologies and advances of I4.0, guaranteeing human participation while ensuring green and sustainable value chains. The smart factory of the new era is characterised by robots and machines working alongside humans, achieving sustainability through innovative segregation techniques and minimising defects and faulty products through production success. On those aspects, the technologies of Augmented Reality (AR), Machine Vision, and Machine Learning (ML) are crucial as they assist humans in manufacturing operations, whereas advanced sensing technologies combined with intelligent capabilities reduce production defects and improve waste segregation flows by separating material agnostic feedstocks. Smart factories are rapidly adopting AR technologies for a wide range of applications. Approximately 25% of their recent I4.0 implementations involve remote assistance and maintenance, the remaining concentrating on visualisation and simulation applications or robot programming and human-robot interaction. Maintenance is one of the most essential aspects of the production life cycle, accounting for 60-70% of its total costs. Consequently, improving its efficacy and performance is a top priority, resulting in the implementation of AR in a number of maintenance applications with the goal of helping humans solve complex unplanned problems. To evaluate the quality of manufactured goods in an environment where the objective is to achieve defect rates close to zero, machine vision systems or manual inspection procedures are used. In industries such as automotive manufacturing, aerospace, and electronics, where precision and safety are essential, defect rates are low, whereas in the Fast-Moving Consumer Goods (FMCG) industry, companies prioritise product quality but still encounter defects due to ingredient issues, manufacturing errors, or packaging errors. Therefore, it is crucial to implement machine vision systems with ML capabilities in in-line quality control processes in order to minimise defects and maintain customer satisfaction. Despite production defects, the world is producing twice as much plastic waste as it did two decades ago, with an estimated 3.4 billion tonnes of waste to be produced by 2050. The vast majority of it ends up in landfills, being burnt or seeping into the environment; only 9% is properly recycled. Barriers such as the existence of complex fractions in waste streams (i.e. pollutants, multi-layer materials, a mixture of plastics, other refuse) that cannot be processed efficiently contribute to the low rate of waste recycling. Therefore, there is an imperative need for a new sorting solution that employs machine vision and ML to accurately identify the type of plastic, thus increasing the amount of waste that is being valorized.Hence, throughout the entire production life cycle, there exists a requirement for an advanced machine vision system with ML capabilities to perceive and comprehend the surrounding environment, enabling it to make autonomous decisions without the need for human intervention. The motivation behind this dissertation is to highlight the role of machine vision technology in the smart environment by improving the current state in the categories of human-centricity, resiliency, and sustainability while proposing a technology maturity assessment framework for the components required by the most advanced machine vision systems. Thus, this thesis introduces a vision-based method for identifying products within production moving lines, an AR maintenance assistant for low-skilled operators that identify assets and propose maintenance steps, a vision-based method for identifying products within dairy production moving lines under ZDM approach, and a multi-modal deep learning methodology capable of exploiting spatial and spectral information from multiple optical modalities, while a novel sorting system is introduced, improving the plastic waste segregation and the wood by-products upcycling processes. This dissertation is organised into six chapters.Let us begin with Chapter 1 where firstly the influence of I4.0 in the production ecosystem is presented, highlighting opportunities for the business and research community. Following the research gap extracted shows that the role of machine vision technology in smart production has not been looked into in-depth, a nine-phase Systematic Literature Review (SLR) strategy was conducted. Since the automotive factory is the leading industry in automated vision systems, the entire automotive manufacturing life cycle from first-tier suppliers up to recycling was reviewed, identifying 59 different vision applications. The analysis revealed that the architectural components of Industry 5.0 machine vision systems can be classified into nine functional categories and four verticals. Additionally, the processing techniques employed in these systems are structured into eight levels, ranging from threshold-based methods to more advanced reinforcement learning approaches. Furthermore, the relevance between machine vision and the technological pillars of I4.0 is analysed, such as the effort on I5.0 and the importance of facilitating efficient smart factory operations.According to the analysis performed in Section 2, machine vision technology aligns with the resilience pillar of I5.0 by offering vision-based solutions to identify products and faults on fast-moving production lines. However, the majority of current production facilities are ageing and lack these characteristics. To provide a standard method for retrofitting machines and integrating with smart factory systems, in Chapter 2, an architecture based on the Zero-Defect Manufacturing (ZDM) strategy was developed, as well as a vision-based method for identifying products within dairy production moving lines, where state-of-the-art architectures (YOLO and RCNN) were compared in the custom public available YoGOdataset. As concerns the human-centric pillar, in Chapter 3, an efficient AR system is proposed to locate components in a manufacturing plant and visualise maintenance instructions for the corresponding failure modes on the mobile devices of the operator. The system detects the asset of interest on the image plane, employing a robust tracking algorithm to track its position and scale across subsequent frames.Based on the calculated position and scale values, the system provides step-by-step instructions for maintenance procedures as the operator approaches the machine, allowing easy navigation through maintenance steps.In Chapter 4, a novel multi-sensor Cyber-Physical Sorting System (CPSS) capable of autonomously characterising and classifying materials in real-time without human intervention is presented. In the CPSS, a new approach for Multi-modal Classifier capable of categorizing waste is proposed, while a detector is utilised to identify objects within moving lines. Specifically, multiple parallel auto-encoders are used to extract spatio-spectral information from RGB, VIS, and NIR sensors and project it into a common latent space so that it can be decoded and the class of each object predicted. To support this deep architecture, the dataset, called Multispectral Mixed Waste Dataset (MMWD), was created which contains multispectral data from the visible (16 bands), near infrared (25 bands) regions of the electromagnetic spectrum and RGB (3 bands) data from polypropylene (PP), polyethene terephthalate (PET), low-density polyethene (LDPE), high-density polyethene (HDPE), medium-density fibreboard (MDF), melamine faced chipboards (MFC) and oak veneer (Al Vene).The dissertation concludes in Chapter 5. This chapter includes a proposed maturity assessment framework to support the evaluation of the machine vision systems that can be included in the I5.0 ecosystem. This maturity framework will aid in the evaluation of each industrial vision-based approach in order to determine where it falls and how it can be improved in the future so that it can be incorporated into a smart factory. Finally, discussion and future suggestions are also presented, which aim to extend the research described here.The contribution of this dissertation to the state-of-the-art is clear and proven by our publication list, as follows:•An analysis regarding the role of machine vision in the smart ecosystem, which summarises 59 distinct applications while defining the architectural components of visual-based systems for the smart environment, is the first conducted in this field. •A low-cost mobile-based AR maintenance application was introduced that locates the asset inside the complex manufacturing shop floor using feature matching and tracking techniques. •A vision-based method for identifying products within dairy production moving lines under ZDM approach is proposed, while the publicly available YoGOdataset (labelled for Coco and Yolo architectures) was created including yoghurt cups with LOT numbers within a production line. •An expandable multi-sensor cyber-physical sorting system capable of separating various wastes based on a trained model was first introduced.•A labelled dataset, called MMWD, comprised of synchronised RGB and multi-spectral data was introduced for the first time, comprising 7 categories of plastic (PET, PP, HDPE, LDPE) and wood (MDF.MFV, Oak Veneer) wastes. •An innovative deep learning multi-modal approach is proposed that utilizes the auto-Encoders framework to fuse spectral and spatial information from RGB, VIS and NIR modalities, improving waste segregation classifiers •A technology maturity assessment framework for machine vision systems that can be fitted into smart production is given for the first time.
Degree
thesis:*- Grantor dc:publisher
- Democritus University of Thrace (DUTH)
- Year dc:date
- 2023
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Konstantinidis, Fotios
- Κωνσταντινίδης, Φώτιος
Subjects
dc:subject × 32- Βιομηχανία 4.0
- Βιομηχανία 5.0
- Μηχανική όραση
- έξυπνη παραγωγή
- Βιωσιμότητα
- Επαυξημένη πραγματικότητα
- Πολυτροπική μεθοδολογία
- Βαθιά μάθηση
- Κυβερνο-φυσικό σύστημα διαλογής
- Διαλογή αποβλήτων
- Industry 4.0
- Industry 5.0
- Machine vision
- Smart production
- Sustainability
- Augmented reality
- Multi-modal architecture
- Deep learning
- Cyber-physical sorting system
- Waste segregation
- Άλλες Επιστήμες Μηχανικού και Τεχνολογίες
- Επιστήμες Μηχανικού και Τεχνολογία
- Μηχανική και Τεχνολογίες, άλλοι τομείς
- Τεχνολογία απεικόνισης και Φωτογραφική τεχνολογία
- Επιστήμη Ηλεκτρολόγου Μηχανικού, Ηλεκτρονικού Μηχανικού, Μηχανικού Η/Υ
- Ρομποτική
- Other Engineering and Technologies
- Engineering and Technology
- Engineering and Technologies, miscellaneous
- Imaging Science and Photographic Technology
- Electrical Engineering, Electronic Engineering, Information Engineering
- Robotics
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/54657