Back to search

University of British Columbia

Construction safety inspection with large pre-trained vision-language models

Abstract

dc:description

Construction safety inspections typically involve a human inspector iden tifying safety concerns on-site. With the rise of powerful large pre-trained Vision Language Models (VLMs), researchers are exploring their use for tasks such as detecting safety rule violations from on-site images. However, the question “whether these large models trained on data scraped from the internet understand concepts related to construction safety” is yet answered. Evaluating this capability is critical before deploying pre-trained models on real construction sites, especially in zero-shot settings (i.e., directly deploying models without training). In this thesis, I attempt to evaluate artificial intelligence (AI)’s capability of understanding safety concepts in zero-shot settings by deploying a state-of-the-art (SOTA) pre-trained VLM to match 443 construction safety posters collected from the internet (i.e., visual inputs) with detailed descriptions of the poster written manually by the researchers (i.e., natural language inputs). Moreover, there is a lack of open datasets to comprehensively evaluate and further fine-tune VLMs in construction safety inspection. Current applications of VLMs use small, supervised datasets, limiting their applicability in tasks they are not directly trained for. In this thesis, I propose the ConstructionSite 10k, featuring 10,013 construction site images with annotations for three inter-connected tasks, including image captioning, safety rule violation visual question answering (VQA), and construction element visual grounding. My subsequent evaluation of current SOTA large pre-trained generative VLMs shows strong generalization abilities in zero-shot and few shot settings. This dataset allows researchers to train and evaluate their own VLMs with new architectures and techniques, providing a valuable benchmark for construction safety inspection. The aforementioned dataset and testing framework pave the way for large-scale evaluation and deployment of these models onsite.

Degree

thesis:*
Name thesis:degree_name
Master of Applied Science - MASc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Civil Engineering
Grantor dc:publisher
University of British Columbia
Year dc:date
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Chen, Xuezheng
Contributors dc:contributor
  • Swei, Omar

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/89945

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Chen, Xuezheng. Construction safety inspection with large pre-trained vision-language models. master's thesis, University of British Columbia, 2024. http://hdl.handle.net/2429/89945