The application of augmented reality (AR) technology for assembly guidance is a novel approach in the traditional manufacturing domain. In this paper, we propose an AR approach for assembly guidance using a virtual interactive tool that is intuitive and easy to use. The virtual interactive tool, termed the Virtual Interaction Panel (VirIP), involves two tasks: the design of the VirIPs and the real-time tracking of an interaction pen using a Restricted Coulomb Energy (RCE) neural network. The VirIP includes virtual buttons, which have meaningful assembly information that can be activated by an interaction pen during the assembly process. A visual assembly tree structure (VATS) is used for information management and assembly instructions retrieval in this AR environment. VATS is a hierarchical tree structure that can be easily maintained via a visual interface. This paper describes a typical scenario for assembly guidance using VirIP and VATS. The main characteristic of the proposed AR system is the intuitive way in which an assembly operator can easily step through a pre-defined assembly plan/sequence without the need of any sensor schemes or markers attached on the assembly components. / Singapore-MIT Alliance (SMA)

<div>The manufacturing industry has begun its transition into a digital age, where data-driven decisions aim to improve product quality, output, and efficiency. Decisions made based on manufacturing data can help identify key problem areas in an assembly line and mitigate any defects from progressing through to the next step in the assembly process. But what if the products’ as manufactured data was inaccurate or didn’t exist at all? Decisions based on incorrect data can lead to defective parts being passed as good parts, costing manufacturers millions of dollars in rework or recalls. When specifically referring to mechanically fastened assemblies, products that experience rotation, like an aircraft propeller, or compress to create a seal, like an oil pipe flange, all require specific torque pattern sequences to be followed during assembly. When incorrectly torqued, the parts can have catastrophic failures resulting in consumer injury or ecological contamination. This paper outlines the development and feasibility of a system and its components for tracking and error-proofing the assembly of bolted joints in an industrial environment.</div><div>Using a machine vision system, the system traces the tool location relative to the mechanical fastener and records which order the fasteners were torqued in, if an error is detected, the system does not allow the user to progress through the assembly process, notifying if an error is detected. The system leverages open source machine learning algorithms from TensorFlow2 and OpenCv, that allow efficient object detection model training. The proposed system was tested using a series of tests and evaluated using the STEP method. The data collected aims to understand the system's feasibility and effectiveness in an industrial setting. </div><div>The tests aim to understand the effectiveness of the system under standard and variable industrial work conditions. Using the STEP method and other statistical analysis, an evaluation matrix was completed, ranking the system's ability to successfully meet all predetermined benchmarks and successfully record the torque pattern used to assemble apart</div>

Mixed Reality (MR) is an emerging paradigm in industry. While MR equipment and software have taken great technological strides in past years, standardized methods and workflows for developing MR systems for industry have not been widely adopted for many tasks. This thesis proposes a dynamic MR system for an assembly process. Optical recognition methods are explored to drive the application logic. The systemis developed using the Unity platform for the HoloLens 2. The software tools Vuforia Engine and Mixed Reality Toolkit (MRTK) are utilized. The project work concludes with an application capable of guiding users using graphics and audio. Successful methods are realized for calibrating the application logic for dynamic object positions,as well as for validating user actions. Experiments are conducted to validate the system. Subjects complete a different assembly process using paper instructions as guidance before using the MR application. Qualitative results regarding the MR experience are obtained through a questionnaire subjects answer, where the experience using paper instructions serves as a benchmark. Data obtained from an experienced user completing the assembly process is used as a quantitative benchmark for system performance measures. All subjects were able to complete the assembly tasks correctly using the MR application. Results show significantly better system performance for the experienced user compared to subjects unfamiliar with the MR system. Vuforia Engine recognition tools successfully tracked individual components that meet a specific criterion. Methods for validating user actions using Vuforia Engine software tools and the HoloLens’s internal hand tracking capabilities resulted in a high validation success rate. The thesis concludes effective training methods for the specific assembly scenario, although not robust for general implementation. / Mixed Reality (MR) är ett framväxande paradigm inom industrin. Medan tillbehör och programvara för MR har gjort enorma framsteg under det senaste decenniet, har standardiserade metoder och arbetsflöden för utveckling av MR applikationer i industriella kontexter inte använts i lika stor utsträckning. Det här examensarbetet utvecklar och proponerar en dynamisk MR applikation för en monteringsprocess. Optiska valideringsmetoder utforskas för att använda applikationen. Applikationen är utvecklad med hjälp av Unity game engine för HoloLens 2. Programvaran Vuforia Engine och MRTK är utnyttjad. Projektarbetet resulterade i en applikation som kan vägleda användare med hjälp av ljud och grafik. Framgångsrika metoder implementerades för att kalibrera applikationslogiken av dynamisk objektspositionering, samt för att validera användarens rörelser. Ett experiment utfördes för att validera MR applikationen där deltagare genomförde en monteringsprocess med hjälp av pappersinstruktioner, vilket används som ett kvalitativt riktmärke. Mätningar av en erfaren applikationsanvändare har använts som ett kvantitativt riktmärke för mätning av systemmässigt utförande. Alla deltagare kunde utföra monteringsuppgifterna korrekt med hjälp av MR applikationen. Resultaten visar betydligt bättre utförande för den erfarna användaren jämfört med personer som inte är bekanta med MR systemet. Spårning av enskilda objekt med hjälp av Vuforia Engine igenkänningsverktyg var framgångsrikt för komponenter som uppfyller ett specifikt kriterium. Metoder för att validera användarens rörelser med programvaran Vuforia Engine samt HoloLens interna handspårningsfunktion gav mycket framgångsrika resultat vid validering. Sammanfattningsvis kom studien fram till effektiva upplärningsmetoder för det här monteringsscenariot, även om de inte var robusta nog för generell implementering.