Nowadays inspections of civil engineering structures are performed manually at close range to be able to assess damages. This requires specialized equipment that tends to be expensive and to produce closure of the bridge. Furthermore, manual inspections are time-consuming and can often be a source or risk for the inspectors. Moreover, manual inspections are subjective and highly dependent on the state of mind of the inspector which reduces the accuracy of this kind of inspections. Image-based inspections using cameras or unmanned aerial vehicles (UAV) combined with image processing have been used to overcome the challenges of traditional manual inspections. This type of inspection has also been studied with the use of machine learning algorithms to improve the detection of damages, in particular cracks. This master’s thesis presents an approach that combines different aspects of the inspection, from the data acquisition, through the crack detection to the quantification of essential parameters. To do this, both digital cameras and a UAV have been used for data acquisition. A convolutional neural network (CNN) for the identification of cracks is used and subsequently, different quantification methods are explored to determine the width and length of the cracks. The results are compared with control measures to determine the accuracy of the method. The results present low to no false negatives when using the CNN to identify cracks. The quantification of the identified cracks is performed obtaining the highest accuracy estimation for 0.2mm cracks.

To address some of the challenges associated with aging bridge infrastructure, this dissertation explores the development and evaluation of a novel tool for bridge inspections leveraging Augmented Reality (AR) and computer vision (CV) technologies to facilitate measurements. Named the Wearable Inspection Report Management System (WIRMS), the system supports various data entry methods and an adaptable automation workflow for defect measurements, showcasing AR's potential to improve bridge inspection efficiency and accuracy. Within this context, the work's main research goal is to understand the difference in performance between traditional field data collection methods (i.e. pen and paper) and automated methods like spoken data entry and CV-based structural defect measurements. In case of CV assistance, emphasis was placed on human-computer interaction (HCI) to understand whether partial, collaborative automation could address some of the limitations of fully automated inspection methods. The project began with comprehensive data collection through interviews, surveys, and observations at bridge sites, which informed the creation of a Virtual Reality (VR) prototype. An initial user study tested the feasibility of using voice commands for data entry in the AR environment but found it impractical. A second user study focused on optimizing interaction methods for virtual concrete crack measurements by testing different degrees of automated CV assistance. As part of this effort, major technical contributions were made to back-end technologies and CV algorithms to improve human-machine collaboration and ensure the accuracy of measurements. Results were mixed, with larger degrees of automation resulting in significant reductions in inspection time and perceived workload, but also significant increases in the amount of measurement error. The latter result is strongly associated with a lack of field robustness of CV methods, which can under-perform if conditions are not ideal. In general, hybrid techniques which allow the user to correct CV results were seen as the most favorable. Field validations with bridge inspectors showed promising potential for practical field implementation, though further refinement is needed for broader deployment. Overall, the research establishes a viable path for making AR a central component to future inspection practices, including digital data collection, automation, data analytics, and other technologies currently in development. / Doctor of Philosophy / This dissertation investigates the development of an innovative tool designed to transform bridge inspections using Augmented Reality (AR) technology, incorporating advanced computer vision (CV) techniques to assist with measurements. The project began with thorough data collection, including interviews and observational studies at bridge sites, which directly influenced the tool's design. A prototype was initially created in a Virtual Reality (VR) environment to refine the functionalities needed for AR application. The resulting AR system supports various interactive methods for documenting and measuring bridge defects, showcasing how AR can streamline and enhance traditional bridge inspection processes. However, challenges remain, particularly in accurately measuring certain types of defects, indicating that some traditional tools are still necessary. Despite these challenges, early tests with bridge inspectors have been promising, suggesting that AR could significantly improve the efficiency and accuracy of bridge inspections. The research demonstrates a clear path forward for further development, with the potential to revolutionize how bridge inspections are conducted.

Esta pesquisa propôs uma contribuição aos procedimentos de inspeção de pontes por meio de um plano de monitoramento de curta duração do comportamento dinâmico do tabuleiro de pontes rodoviárias de concreto com a utilização de receptores GPS de 100 Hz associado à diversas técnicas de filtragem de sinais. O estudo foi conduzido em uma ponte em serviço localizada na rodovia Fernão Dias (BR-381). O procedimento de inspeção proposto baseou-se no uso de dois sinais de satélites GPS, por meio da aplicação do Método Residual de Fase (MRF), e da análise temporal dos resíduos da dupla diferença de fase a partir da Fast Fourier Transform (FFT), Continuous Wavelet Transform (CWT), Passa-Faixa Chebyshev do Tipo I. A descrição do comportamento dinâmico do tabuleiro do vão central da ponte se deu por meio da extração dos valores de frequência e amplitude das deflexões verticais a partir dos dados GPS filtrados (resíduos), em três períodos de amostragem de 1 minuto. Os valores de frequência variaram de 0,5 Hz a 8 Hz nos três períodos de amostragens e também nos filtros FFT, CWT e Passa-Faixa Chebyshev do Tipo I. Os valores de amplitude de deslocamento vertical máximo ficaram em torno de 6 mm. Estes valores coincidiram com os valores de frequência e amplitude de deslocamento vertical registrados pela instrumentação clássica com acelerômetros, transdutores de deslocamento vertical, modelagem por elementos finitos e prova de carga estática e dinâmica realizadas sobre tabuleiro do vão central da ponte instrumentado. Por fim, pode-se afirmar que os procedimentos propostos, além de poderem ser utilizados como uma etapa preliminar de inspeção de tabuleiro de pontes rígidas possibilitaram a detecção de deslocamentos dinâmicos verticais milimétricos e suas frequências de vibração. / This research proposed a contribution for bridge inspection procedures at of a short-term monitoring plan of the dynamic behavior of the concrete road bridge with the use of 100 Hz GPS receivers associated with various signal filtering techniques. The study was conducted in the service bridge located on the highway Fernão Dias (BR-381). The procedure inspection proposed was based on the use of two satellite GPS signals, by applying the Phase Residual Method (PRM), and the temporal analysis of the residuals of the double difference phase from the Fast Fourier Transform (FFT), Continuous Wavelet Transform (CWT), Type I Chebyshev Band-Pass. The dynamic behavior of the central span of the bridge was described by extracting the frequency and amplitude values of the vertical deflections from the filtered GPS data (residues), in three sampling periods of 1 minute. The frequency values found ranged from 0.5 Hz to 8 Hz in the three sampling periods as well as in the FFT, CWT and Chebyshev Type I Band-Pass filters. The maximum vertical displacement peak values were around 6 mm. These values coincided with the vertical displacement and amplitude values recorded by the classical instrumentation with accelerometers, vertical displacement transducers, finite element modeling and static and dynamic load test performed on the central span deck of the instrumented bridge. Finally, it could affirm that the procedures proposed, besides being able to be used as a preliminary step of inspection of the rigid bridge\'s deck enabled the detection of millimetric vertical dynamic displacements and their frequencies of vibration.

Esta pesquisa propôs uma contribuição aos procedimentos de inspeção de pontes por meio de um plano de monitoramento de curta duração do comportamento dinâmico do tabuleiro de pontes rodoviárias de concreto com a utilização de receptores GPS de 100 Hz associado à diversas técnicas de filtragem de sinais. O estudo foi conduzido em uma ponte em serviço localizada na rodovia Fernão Dias (BR-381). O procedimento de inspeção proposto baseou-se no uso de dois sinais de satélites GPS, por meio da aplicação do Método Residual de Fase (MRF), e da análise temporal dos resíduos da dupla diferença de fase a partir da Fast Fourier Transform (FFT), Continuous Wavelet Transform (CWT), Passa-Faixa Chebyshev do Tipo I. A descrição do comportamento dinâmico do tabuleiro do vão central da ponte se deu por meio da extração dos valores de frequência e amplitude das deflexões verticais a partir dos dados GPS filtrados (resíduos), em três períodos de amostragem de 1 minuto. Os valores de frequência variaram de 0,5 Hz a 8 Hz nos três períodos de amostragens e também nos filtros FFT, CWT e Passa-Faixa Chebyshev do Tipo I. Os valores de amplitude de deslocamento vertical máximo ficaram em torno de 6 mm. Estes valores coincidiram com os valores de frequência e amplitude de deslocamento vertical registrados pela instrumentação clássica com acelerômetros, transdutores de deslocamento vertical, modelagem por elementos finitos e prova de carga estática e dinâmica realizadas sobre tabuleiro do vão central da ponte instrumentado. Por fim, pode-se afirmar que os procedimentos propostos, além de poderem ser utilizados como uma etapa preliminar de inspeção de tabuleiro de pontes rígidas possibilitaram a detecção de deslocamentos dinâmicos verticais milimétricos e suas frequências de vibração. / This research proposed a contribution for bridge inspection procedures at of a short-term monitoring plan of the dynamic behavior of the concrete road bridge with the use of 100 Hz GPS receivers associated with various signal filtering techniques. The study was conducted in the service bridge located on the highway Fernão Dias (BR-381). The procedure inspection proposed was based on the use of two satellite GPS signals, by applying the Phase Residual Method (PRM), and the temporal analysis of the residuals of the double difference phase from the Fast Fourier Transform (FFT), Continuous Wavelet Transform (CWT), Type I Chebyshev Band-Pass. The dynamic behavior of the central span of the bridge was described by extracting the frequency and amplitude values of the vertical deflections from the filtered GPS data (residues), in three sampling periods of 1 minute. The frequency values found ranged from 0.5 Hz to 8 Hz in the three sampling periods as well as in the FFT, CWT and Chebyshev Type I Band-Pass filters. The maximum vertical displacement peak values were around 6 mm. These values coincided with the vertical displacement and amplitude values recorded by the classical instrumentation with accelerometers, vertical displacement transducers, finite element modeling and static and dynamic load test performed on the central span deck of the instrumented bridge. Finally, it could affirm that the procedures proposed, besides being able to be used as a preliminary step of inspection of the rigid bridge\'s deck enabled the detection of millimetric vertical dynamic displacements and their frequencies of vibration.

The forecast of the next upcoming years estimates a growth of demand in transport. As the railway sector in Europe has developed over many years, the infrastructure presents performance issues because of, among other factors, asset maintenance activities being difficult and time consuming. There are currently 4000 railway bridges in Sweden managed by Trafikverket which are submitted to inspections at least every six years. The most common survey is done visually to determine the physical and functional condition of the bridges as well as finding damages that may exist on them. Because visual inspection is a subjective evaluation technique, the results of these bridge inspections may vary from inspector to inspector. The data collection is time consuming and written in standard inspection reports which may not provide sufficient visualization of damages. The inspector also needs to move around the bridge at close distance which could lead to unsafe working conditions. 3D modelling technology is becoming more and more common. Methods such as Close Ranged Photogrammetry (CRP) and Terrestrial Laser Scanning (TLS) are starting to be used for architecture and heritage preservation as well as engineering applications. Infrared (IR) scanning is also showing potential in creating 3D models but has yet not been used for structural analysis and inspections. A result from these methods is a point cloud, a 3D representation of a model in points that can be used for creating as-built Building Information Modeling (BIM)-models. In this study, the authors put these three methods to test to see if IR scanning and CRP are suitable ways, such as TLS is, to gather data for 3D-reconstruction of concrete railway bridges in fast, safe and non-disturbing ways. For this, the three technologies are performed on six bridges chosen by Trafikverket. The further aim is to determine if the 3D-reconstructions can be used for acquiring BIM-information to, among other things, create as-built drawings and to perform structural evaluations. As a result from the study, IR scanning and CRP show great potential as well as TLS in 3D-reconstruction of concrete railway bridges in fast, safe and non-disturbing ways. Still, there is a need of development regarding the technologies before we can start to rely on them completely.

Inspections of bridges are today performed ocularly by an inspector at arm’s lengths distance to evaluate damages and to assess its current condition. Ocular inspections often require specialized equipment to aid the inspector to reach all parts of the bridge. The current state of practice for bridge inspection is therefore considered to be time-consuming, costly, and a safety hazard for the inspector. The purpose of this thesis has been to develop a method for automated modeling of bridges from point cloud data. Point clouds that have been created through photogrammetry from a collection of images acquired with an Unmanned Aerial Vehicle (UAV). This thesis has been an attempt to contribute to the long-term goal of making bridge inspections more efficient by using UAV technology. Several methods for the identification of structural components in point clouds have been evaluated. Based on this, a method has been developed to identify planar surfaces using the model-fitting method Random Sample Consensus (RANSAC). The developed method consists of a set of algorithms written in the programming language Python. The method utilizes intersection points between planes as well as the k-Nearest-Neighbor (k-NN) concept to identify the vertices of the structural elements. The method has been tested both for simulated point cloud data as well as for real bridges, where the images were acquired with a UAV. The results from the simulated point clouds showed that the vertices were modeled with a mean deviation of 0.13− 0.34 mm compared to the true vertex coordinates. For a point cloud of a rectangular column, the algorithms identified all relevant surfaces and were able to reconstruct it with a deviation of less than 2 % for the width and length. The method was also tested on two point clouds of real bridges. The algorithms were able to identify many of the relevant surfaces, but the complexity of the geometries resulted in inadequately reconstructed models. / Besiktning av broar utförs i dagsläget okulärt av en inspektör som på en armlängds avstånd bedömer skadetillståndet. Okulär besiktning kräver därmed ofta speciell utrustning för att inspektören ska kunna nå samtliga delar av bron. Detta resulterar i att det nuvarande tillvägagångssättet för brobesiktning beaktas som tidkrävande, kostsamt samt riskfyllt för inspektören. Syftet med denna uppsats var att utveckla en metod för att modellera broar på ett automatiserat sätt utifrån punktmolnsdata. Punktmolnen skapades genom fotogrammetri, utifrån en samling bilder tagna med en drönare. Uppsatsen har varit en insats för att bidra till det långsiktiga målet att effektivisera brobesiktning genom drönarteknik. Flera metoder för att identifiera konstruktionselement i punktmoln har undersökts. Baserat på detta har en metod utvecklats som identifierar plana ytor med regressionsmetoden Random Sample Consensus (RANSAC). Den utvecklade metoden består av en samling algoritmer skrivna i programmeringsspråket Python. Metoden grundar sig i att beräkna skärningspunkter mellan plan samt använder konceptet k-Nearest-Neighbor (k-NN) för att identifiera konstruktionselementens hörnpunkter. Metoden har testats på både simulerade punktmolnsdata och på punktmoln av fysiska broar, där bildinsamling har skett med hjälp av en drönare. Resultatet från de simulerade punktmolnen visade att hörnpunkterna kunde identifieras med en medelavvikelse på 0,13 − 0,34 mm jämfört med de faktiska hörnpunkterna. För ett punktmoln av en rektangulär pelare lyckades algoritmerna identifiera alla relevanta ytor och skapa en rekonstruerad modell med en avvikelse på mindre än 2 % med avseende på dess bredd och längd. Metoden testades även på två punktmoln av riktiga broar. Algoritmerna lyckades identifiera många av de relevanta ytorna, men geometriernas komplexitet resulterade i bristfälligt rekonstruerade modeller.

The purpose of this research was to develop a new index for concrete bridge deck management in Utah. Data were collected in the summer of 2005 from 15 concrete bridge decks in the vicinity of Salt Lake City. The decks ranged from 2 to 21 years in age and were all constructed using epoxy-coated rebar. Visual inspection, sounding, Schmidt hammer testing, half-cell potential testing, and chloride concentration testing were performed on six 6-ft by 6-ft test areas randomly distributed within the single lane closed to traffic on each deck, and testing protocols followed American Society for Testing and Materials standards to the extent possible. Collected data were analyzed using statistics, and age, cover, and half-cell potential were ultimately selected for inclusion in a new Utah Bridge Deck Index (UBDI); these variables effectively reflect chloride-induced corrosion mechanisms active on Utah bridge decks, are highly correlated to delamination distresses, and are relatively easy to measure compared to chloride concentration. At the request of Utah Department of Transportation (UDOT) personnel, the UBDI equation was structured around a deduct system using a 100-point scale similar to the sufficiency rating system, in which a perfect bridge deck receives a score of 100. Coefficients were selected based largely on the judgment of the researchers and the UDOT personnel involved in the research, and threshold values for maintenance, rehabilitation, and replacement (MR&R) options were specified to be the same as those associated with the standard sufficiency ratings. The UBDI and corresponding MR&R recommendation were then provided for each of the bridge decks tested in this research; nine of the decks are recommended for preventive treatment, and six are recommended for rehabilitation. In addition, the possibility of treatment applications was considered, leading to required adjustments in the UBDI calculation; the treatment options that were considered include an epoxy seal, an HPC overlay, and an asphalt membrane overlay. Four case scenarios were developed to demonstrate the response of the revised UBDI equation to these treatments. Finally, as aids for UDOT personnel implementing this research, charts were created to facilitate rapid determination of the required number of half-cell potential and concrete cover measurements for different levels of reliability and tolerance. The UBDI developed in this research is recommended for implementation by UDOT personnel as a tool for optimizing the timing of MR&R treatments on concrete bridge decks similar to those evaluated in this project. In measuring cover and half-cell potential values, UDOT personnel should utilize the sampling guidelines presented in this report to ensure adequate characterization of each deck. Furthermore, to facilitate the inclusion of treatment effects in the UBDI, UDOT personnel should establish a policy of recording the types and dates of all MR&R treatments applied to bridge decks. As performance data are collected for specific treatments over time, the treatment lives proposed in this research for epoxy seals, HPC overlays, and asphalt membrane overlays should be revised as needed, and information for other treatments may be added. In addition, to maximize the predictive capabilities of the UBDI, more accurate relationships between half-cell potential values and deck age should be developed for estimating future deck condition.

<p>  </p> <p>The inspection of highway bridge structures in the United States is a task critical to the national transportation system. Inspection images contain abundant visual information that can be exploited to streamline bridge assessment and management tasks. However, historical inspection images often go unused in subsequent assessments as they are disorganized and unlabeled. Further, due to the lack of GPS metadata and visual ambiguity, it is often difficult for other inspectors to identify the location on the bridge where past images were taken. While many approaches are being considered toward fully- or semi-automated methods for bridge inspection, there are research opportunities to develop practical tools for inspectors to make use of those images already in a database. In this study, a deep learning-based image similarity technique is combined with image geolocation data to localize and retrieve historical inspection images based on a current query image. A Siamese convolutional neural network (SCNN) is trained and validated on a gathered dataset of over 1,000 real world bridge deck images collected by the Indiana Department of Transportation. A composite similarity (CS) metric is created for effective image ranking and the overall method is validated on a subset of eight bridge’s images. The results show promise for implementation into existing databases and for other similar structural inspections, showing up to an 11-fold improvement in successful image retrieval when compared to random image selection.</p>

Common Objects in Context for bridge inspection (COCO-Bridge) was introduced for use by unmanned aircraft systems (UAS) to assist in GPS denied environments, flight-planning, and detail identification and contextualization, but has far-reaching applications such as augmented reality (AR) and other artificial intelligence (AI) platforms. COCO-Bridge is an annotated dataset which can be trained using a convolutional neural network (CNN) to identify specific structural details. Many annotated datasets have been developed to detect regions of interest in images for a wide variety of applications and industries. While some annotated datasets of structural defects (primarily cracks) have been developed, most efforts are individualized and focus on a small niche of the industry. This effort initiated a benchmark dataset with a focus on structural details. This research investigated the required parameters for detail identification and evaluated performance enhancements on the annotation process. The image dataset consisted of four structural details which are commonly reviewed and rated during bridge inspections: bearings, cover plate terminations, gusset plate connections, and out of plane stiffeners. This initial version of COCO-Bridge includes a total of 774 images; 10% for evaluation and 90% for training. Several models were used with the dataset to evaluate model overfitting and performance enhancements from augmentation and number of iteration steps. Methods to economize the predictive capabilities of the model without the addition of unique data were investigated to reduce the required number of training images. Results from model tests indicated the following: additional images, mirrored along the vertical-axis, provided precision and accuracy enhancements; increasing computational step iterations improved predictive precision and accuracy, and the optimal confidence threshold for operation was 25%. Annotation recommendations and improvements were also discovered and documented as a result of the research. / MS / Common Objects in Context for bridge inspection (COCO-Bridge) was introduced to improve a drone-conducted bridge inspection process. Drones are a great tool for bridge inspectors because they bring flexibility and access to the inspection. However, drones have a notoriously difficult time operating near bridges, because the signal can be lost between the operator and the drone. COCO-Bridge is an imagebased dataset that uses Artificial Intelligence (AI) as a solution to this particular problem, but has applications in other facets of the inspection as well. This effort initiated a dataset with a focus on identifying specific parts of a bridge or structural bridge elements. This would allow a drone to fly without explicit direction if the signal was lost, and also has the potential to extend its flight time. Extending flight time and operating autonomously are great advantagesfor drone operators and bridge inspectors. The output from COCO-Bridge would also help the inspectors identify areas that are prone to defects by highlighting regions that require inspection. The image dataset consisted of 774 images to detect four structural bridge elements which are commonly reviewed and rated during bridge inspections. The goal is to continue to increase the number of images and encompass more structural bridge elements in the dataset so that it may be used for all types of bridges. Methods to reduce the required number of images were investigated, because gathering images of structural bridge elements is challenging,. The results from model tests helped build a roadmap for the expansion and best-practices for developing a dataset of this type.

Bridge inspection and infrastructure inspection are critical steps in the lifecycle of the built environment. Emerging technologies and data are driving factors which are disrupting the traditional processes for conducting these inspections. Because inspections are mainly conducted visually by human inspectors, this paper focuses on improving the visual inspection process with data-driven approaches. Data driven approaches, however, require significant data, which was sparse in the existing literature. Therefore, this research first examined the present state of the existing data in the research domain. We reviewed hundreds of image-based visual inspection papers which used machine learning to augment the inspection process and from this, we compiled a comprehensive catalog of over forty available datasets in the literature and identified promising, emerging techniques and trends in the field. Based on our findings in our review we contributed six significant datasets to target gaps in data in the field. The six datasets comprised of structural material segmentation, corrosion condition state segmentation, crack detection, structural detail detection, and bearing condition state classification. The contributed datasets used novel annotation guidelines and benefitted from a novel semi-automated annotation process for both object detection and pixel-level detection models. Using the data obtained from our collected sources, task-appropriate deep learning models were trained. From these datasets and models, we developed a change detection algorithm to monitor damage evolution between two inspection videos and trained a GAN-Inversion model which generated hyper-realistic synthetic bridge inspection image data and could forecast a future deterioration state of an existing bridge element. While the application of machine learning techniques in civil engineering is not wide-spread yet, this research provides impactful contribution which demonstrates the advantages that data driven sciences can provide to more economically and efficiently inspect structures, catalog deterioration, and forecast potential outcomes. / Doctor of Philosophy / Bridge inspection and infrastructure inspection are critical steps in the lifecycle of the built environment. Emerging technologies and data are driving factors which are disrupting the traditional processes for conducting these inspections. Because inspections are mainly conducted visually by human inspectors, this paper focuses on improving the visual inspection process with data-driven approaches. Data driven approaches, however, require significant data, which was sparse in the existing literature. Therefore, this research first examined the present state of the existing data in the research domain. We reviewed hundreds of image-based visual inspection papers which used machine learning to augment the inspection process and from this, we compiled a comprehensive catalog of over forty available datasets in the literature and identified promising, emerging techniques and trends in the field. Based on our findings in our review we contributed six significant datasets to target gaps in data in the field. The six datasets comprised of structural material detection, corrosion condition state identification, crack detection, structural detail detection, and bearing condition state classification. The contributed datasets used novel labeling guidelines and benefitted from a novel semi-automated labeling process for the artificial intelligence models. Using the data obtained from our collected sources, task-appropriate artificial intelligence models were trained. From these datasets and models, we developed a change detection algorithm to monitor damage evolution between two inspection videos and trained a generative model which generated hyper-realistic synthetic bridge inspection image data and could forecast a future deterioration state of an existing bridge element. While the application of machine learning techniques in civil engineering is not widespread yet, this research provides impactful contribution which demonstrates the advantages that data driven sciences can provide to more economically and efficiently inspect structures, catalog deterioration, and forecast potential outcomes.