{"id":{"repo_id":"ets-quebec","oai_identifier":"oai:espace.etsmtl.ca:3313"},"canonical_url":"https://search.ndltd.org/etd/ets-quebec/oai:espace.etsmtl.ca:3313","repository":{"repo_id":"ets-quebec","name":"ETS (Quebec)","base_url":"https://espace.etsmtl.ca/cgi/oai2"},"display":{"title":"Deep visual-infrared fusion for multimodal person re-identification in the wild","abstract":"Person re-identification (ReID) is a crucial video surveillance task, allowing one to match images of individuals captured by non-overlapping cameras. This task poses significant challenges due to factors such as varying camera positions, capture conditions (e.g., illumination, weather, background), complex body shapes, and diverse clothing styles. These factors result in a wide range of potential capture conditions leading to datasets that only cover a small fraction of the potential scenarios, and consequently to models’ learning and evaluation data unsuited to the design of a robust framework. Under these constraints, a cost-effective model must be built to capture the complex and discriminative personal features while allowing to perform real-time data processing. Among the previous aspects, the visible modality commonly used in traditional ReID frameworks is highly dependent on the prevailing luminosity. Low-light conditions can severely affect the quality of captured scenes, resulting in inaccurate ReID. This introduces an additional obstacle in the person ReID task, compounded by the potential presence of noisy or blurry captures. Infrared cameras can mitigate the issues caused by lighting conditions because they do not rely on light information for scene encoding, but do not capture color information and are similarly affected by sensor encoding issues. Therefore, visible and infrared sensors are versatile and relevant sensors in the context of person ReID, but relying solely on one of these modalities compromises the effectiveness of the framework in outdoor conditions or under complex capture conditions. In this thesis, the multimodal setting and especially the fusion of visible and infrared (V-I) modalities are considered to address these challenges. By having a distinct encoding process while capturing videos from the same individuals, V-I cameras allow for correlated captures while limiting the effect of eventual encoding issues. Chapter 1 provides some background on deep learning models and techniques for person ReID. Then, a review of multimodal fusion techniques and real-world data for person ReID is provided in Chapter 2, allowing us to assess the key challenges in the area. As such, multiple aspects must be considered. First, multimodal fusion models are seen to neglect modality-specific features, mostly focusing on shared knowledge instead, and consequently missing an important part of discriminant information. In addition, recent approaches emphasize the need to deepen evaluation protocols by artificially corrupting datasets but also to implement specific learning strategies in this regard. However, while such approaches have been explored and shown to be effective in the unimodal setting, this has yet to be done for fusion algorithms, similarly affected by real-world unpredictability. This work presents a novel multimodal model architecture Chapter 3 to fully exploit modality knowledge and handle real-world data. The model is composed of three backbones, two concentrate on extracting modality-specific features, while the third leverage shared knowledge from a fused modality representation. Furthermore, attention-based approaches are investigated to enable dynamic feature selection, which is likely suitable for multimodal feature fusion under challenging operational conditions. These conditions are reproduced through the proposed V-I corrupted datasets that replicate realistic and highly challenging conditions for both co-located and not co-located camera scenarios, allowing an in-depth model evaluation. For co-located cameras, eventual corruptions correlations are considered, not expected, and consequently not applied for not co-located cameras since each V and I cameras are at distinct locations. Finally, a multimodal data augmentation that enhances the multimodal model’s capacity for generalization is proposed and works at promoting collaboration among modalities and priming the model to face modality-specific local or global corruptions. The use of three datasets and two corrupted evaluation scenarios through twenty V and I corruptions allowed us to show that the multimodal ReID strategy can improve ReID accuracy while conserving moderate system complexity. Specifically, with the appropriate learning approach, the proposed multimodal model can outperform related state-of-the-art systems under ideal and challenging noisy real-world conditions.","abstract_html":"Person re-identification (ReID) is a crucial video surveillance task, allowing one to match images of individuals captured by non-overlapping cameras. This task poses significant challenges due to factors such as varying camera positions, capture conditions (e.g., illumination, weather, background), complex body shapes, and diverse clothing styles. These factors result in a wide range of potential capture conditions leading to datasets that only cover a small fraction of the potential scenarios, and consequently to models’ learning and evaluation data unsuited to the design of a robust framework. Under these constraints, a cost-effective model must be built to capture the complex and discriminative personal features while allowing to perform real-time data processing. Among the previous aspects, the visible modality commonly used in traditional ReID frameworks is highly dependent on the prevailing luminosity. Low-light conditions can severely affect the quality of captured scenes, resulting in inaccurate ReID. This introduces an additional obstacle in the person ReID task, compounded by the potential presence of noisy or blurry captures. Infrared cameras can mitigate the issues caused by lighting conditions because they do not rely on light information for scene encoding, but do not capture color information and are similarly affected by sensor encoding issues. Therefore, visible and infrared sensors are versatile and relevant sensors in the context of person ReID, but relying solely on one of these modalities compromises the effectiveness of the framework in outdoor conditions or under complex capture conditions. In this thesis, the multimodal setting and especially the fusion of visible and infrared (V-I) modalities are considered to address these challenges. By having a distinct encoding process while capturing videos from the same individuals, V-I cameras allow for correlated captures while limiting the effect of eventual encoding issues. Chapter 1 provides some background on deep learning models and techniques for person ReID. Then, a review of multimodal fusion techniques and real-world data for person ReID is provided in Chapter 2, allowing us to assess the key challenges in the area. As such, multiple aspects must be considered. First, multimodal fusion models are seen to neglect modality-specific features, mostly focusing on shared knowledge instead, and consequently missing an important part of discriminant information. In addition, recent approaches emphasize the need to deepen evaluation protocols by artificially corrupting datasets but also to implement specific learning strategies in this regard. However, while such approaches have been explored and shown to be effective in the unimodal setting, this has yet to be done for fusion algorithms, similarly affected by real-world unpredictability. This work presents a novel multimodal model architecture Chapter 3 to fully exploit modality knowledge and handle real-world data. The model is composed of three backbones, two concentrate on extracting modality-specific features, while the third leverage shared knowledge from a fused modality representation. Furthermore, attention-based approaches are investigated to enable dynamic feature selection, which is likely suitable for multimodal feature fusion under challenging operational conditions. These conditions are reproduced through the proposed V-I corrupted datasets that replicate realistic and highly challenging conditions for both co-located and not co-located camera scenarios, allowing an in-depth model evaluation. For co-located cameras, eventual corruptions correlations are considered, not expected, and consequently not applied for not co-located cameras since each V and I cameras are at distinct locations. Finally, a multimodal data augmentation that enhances the multimodal model’s capacity for generalization is proposed and works at promoting collaboration among modalities and priming the model to face modality-specific local or global corruptions. The use of three datasets and two corrupted evaluation scenarios through twenty V and I corruptions allowed us to show that the multimodal ReID strategy can improve ReID accuracy while conserving moderate system complexity. Specifically, with the appropriate learning approach, the proposed multimodal model can outperform related state-of-the-art systems under ideal and challenging noisy real-world conditions.","abstract_has_math":false,"creators":["Josi, Arthur"],"institution":"École de technologie supérieure","degree_name":"Mémoire de maîtrise électronique","degree_level":"masters","degree_discipline":null,"degree_department":null,"school":null,"contributors":[],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2023,"date_issued":"2023-08-16","date_published":"2023-08-16","updated_at":"2026-11-18T22:09:57Z","subjects":["réseaux neuronaux profonds, fusion multimodale, images corrompues, augmentation de données, ré-identification visible-infrarouge de personnes"],"languages":["en"],"rights":[],"rights_urls":[],"identifier_entries":[{"key":"dc:identifier","label":"Identifier","values":["INSERT-YOUR-LIBRARY-ID-HERE3313"],"render_values":[{"text":"INSERT-YOUR-LIBRARY-ID-HERE3313","href":null,"code":true}]}],"person_identifier_entries":[]},"links":{"outbound_url":null,"outbound_label":null,"outbound_source":null,"record_url":null,"record_label":null,"record_kind":null,"record_source":null,"download_url":"https://espace.etsmtl.ca/id/eprint/3313/1/JOSI_Arthur.pdf","download_label":"Download PDF","download_source":"dc:identifier","access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://espace.etsmtl.ca/cgi/oai2?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Aespace.etsmtl.ca%3A3313","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:creator","label":"Author","values":["Josi, Arthur"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2023-08-16"]},{"key":"dc:publisher","label":"Institution","values":["École de technologie supérieure"]},{"key":"dc:type","label":"Dc Type","values":["Electronic Thesis or Dissertation"]},{"key":"thesis:degree_level","label":"Degree Level","values":["masters"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Mémoire de maîtrise électronique"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["École de technologie supérieure"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["réseaux neuronaux profonds, fusion multimodale, images corrompues, augmentation de données, ré-identification visible-infrarouge de personnes"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["INSERT-YOUR-LIBRARY-ID-HERE3313","https://espace.etsmtl.ca/id/eprint/3313/1/JOSI_Arthur.pdf"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Person re-identification (ReID) is a crucial video surveillance task, allowing one to match images of individuals captured by non-overlapping cameras. This task poses significant challenges due to factors such as varying camera positions, capture conditions (e.g., illumination, weather, background), complex body shapes, and diverse clothing styles. These factors result in a wide range of potential capture conditions leading to datasets that only cover a small fraction of the potential scenarios, and consequently to models’ learning and evaluation data unsuited to the design of a robust framework. Under these constraints, a cost-effective model must be built to capture the complex and discriminative personal features while allowing to perform real-time data processing. Among the previous aspects, the visible modality commonly used in traditional ReID frameworks is highly dependent on the prevailing luminosity. Low-light conditions can severely affect the quality of captured scenes, resulting in inaccurate ReID. This introduces an additional obstacle in the person ReID task, compounded by the potential presence of noisy or blurry captures. Infrared cameras can mitigate the issues caused by lighting conditions because they do not rely on light information for scene encoding, but do not capture color information and are similarly affected by sensor encoding issues. Therefore, visible and infrared sensors are versatile and relevant sensors in the context of person ReID, but relying solely on one of these modalities compromises the effectiveness of the framework in outdoor conditions or under complex capture conditions. In this thesis, the multimodal setting and especially the fusion of visible and infrared (V-I) modalities are considered to address these challenges. By having a distinct encoding process while capturing videos from the same individuals, V-I cameras allow for correlated captures while limiting the effect of eventual encoding issues. Chapter 1 provides some background on deep learning models and techniques for person ReID. Then, a review of multimodal fusion techniques and real-world data for person ReID is provided in Chapter 2, allowing us to assess the key challenges in the area. As such, multiple aspects must be considered. First, multimodal fusion models are seen to neglect modality-specific features, mostly focusing on shared knowledge instead, and consequently missing an important part of discriminant information. In addition, recent approaches emphasize the need to deepen evaluation protocols by artificially corrupting datasets but also to implement specific learning strategies in this regard. However, while such approaches have been explored and shown to be effective in the unimodal setting, this has yet to be done for fusion algorithms, similarly affected by real-world unpredictability. This work presents a novel multimodal model architecture Chapter 3 to fully exploit modality knowledge and handle real-world data. The model is composed of three backbones, two concentrate on extracting modality-specific features, while the third leverage shared knowledge from a fused modality representation. Furthermore, attention-based approaches are investigated to enable dynamic feature selection, which is likely suitable for multimodal feature fusion under challenging operational conditions. These conditions are reproduced through the proposed V-I corrupted datasets that replicate realistic and highly challenging conditions for both co-located and not co-located camera scenarios, allowing an in-depth model evaluation. For co-located cameras, eventual corruptions correlations are considered, not expected, and consequently not applied for not co-located cameras since each V and I cameras are at distinct locations. Finally, a multimodal data augmentation that enhances the multimodal model’s capacity for generalization is proposed and works at promoting collaboration among modalities and priming the model to face modality-specific local or global corruptions. The use of three datasets and two corrupted evaluation scenarios through twenty V and I corruptions allowed us to show that the multimodal ReID strategy can improve ReID accuracy while conserving moderate system complexity. Specifically, with the appropriate learning approach, the proposed multimodal model can outperform related state-of-the-art systems under ideal and challenging noisy real-world conditions."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Deep visual-infrared fusion for multimodal person re-identification in the wild"]}]}],"canonical_facts":{"dc:creator":["Josi, Arthur"],"dc:date":["2023-08-16"],"dc:description":["Person re-identification (ReID) is a crucial video surveillance task, allowing one to match images of individuals captured by non-overlapping cameras. This task poses significant challenges due to factors such as varying camera positions, capture conditions (e.g., illumination, weather, background), complex body shapes, and diverse clothing styles. These factors result in a wide range of potential capture conditions leading to datasets that only cover a small fraction of the potential scenarios, and consequently to models’ learning and evaluation data unsuited to the design of a robust framework. Under these constraints, a cost-effective model must be built to capture the complex and discriminative personal features while allowing to perform real-time data processing. Among the previous aspects, the visible modality commonly used in traditional ReID frameworks is highly dependent on the prevailing luminosity. Low-light conditions can severely affect the quality of captured scenes, resulting in inaccurate ReID. This introduces an additional obstacle in the person ReID task, compounded by the potential presence of noisy or blurry captures. Infrared cameras can mitigate the issues caused by lighting conditions because they do not rely on light information for scene encoding, but do not capture color information and are similarly affected by sensor encoding issues. Therefore, visible and infrared sensors are versatile and relevant sensors in the context of person ReID, but relying solely on one of these modalities compromises the effectiveness of the framework in outdoor conditions or under complex capture conditions. In this thesis, the multimodal setting and especially the fusion of visible and infrared (V-I) modalities are considered to address these challenges. By having a distinct encoding process while capturing videos from the same individuals, V-I cameras allow for correlated captures while limiting the effect of eventual encoding issues. Chapter 1 provides some background on deep learning models and techniques for person ReID. Then, a review of multimodal fusion techniques and real-world data for person ReID is provided in Chapter 2, allowing us to assess the key challenges in the area. As such, multiple aspects must be considered. First, multimodal fusion models are seen to neglect modality-specific features, mostly focusing on shared knowledge instead, and consequently missing an important part of discriminant information. In addition, recent approaches emphasize the need to deepen evaluation protocols by artificially corrupting datasets but also to implement specific learning strategies in this regard. However, while such approaches have been explored and shown to be effective in the unimodal setting, this has yet to be done for fusion algorithms, similarly affected by real-world unpredictability. This work presents a novel multimodal model architecture Chapter 3 to fully exploit modality knowledge and handle real-world data. The model is composed of three backbones, two concentrate on extracting modality-specific features, while the third leverage shared knowledge from a fused modality representation. Furthermore, attention-based approaches are investigated to enable dynamic feature selection, which is likely suitable for multimodal feature fusion under challenging operational conditions. These conditions are reproduced through the proposed V-I corrupted datasets that replicate realistic and highly challenging conditions for both co-located and not co-located camera scenarios, allowing an in-depth model evaluation. For co-located cameras, eventual corruptions correlations are considered, not expected, and consequently not applied for not co-located cameras since each V and I cameras are at distinct locations. Finally, a multimodal data augmentation that enhances the multimodal model’s capacity for generalization is proposed and works at promoting collaboration among modalities and priming the model to face modality-specific local or global corruptions. The use of three datasets and two corrupted evaluation scenarios through twenty V and I corruptions allowed us to show that the multimodal ReID strategy can improve ReID accuracy while conserving moderate system complexity. Specifically, with the appropriate learning approach, the proposed multimodal model can outperform related state-of-the-art systems under ideal and challenging noisy real-world conditions."],"dc:format":["application/pdf"],"dc:identifier":["INSERT-YOUR-LIBRARY-ID-HERE3313","https://espace.etsmtl.ca/id/eprint/3313/1/JOSI_Arthur.pdf"],"dc:language":["en"],"dc:publisher":["École de technologie supérieure"],"dc:subject":["réseaux neuronaux profonds, fusion multimodale, images corrompues, augmentation de données, ré-identification visible-infrarouge de personnes"],"dc:title":["Deep visual-infrared fusion for multimodal person re-identification in the wild"],"dc:type":["Electronic Thesis or Dissertation"],"thesis:degree_level":["masters"],"thesis:degree_name":["Mémoire de maîtrise électronique"],"thesis:institution_name":["École de technologie supérieure"]},"updated_at":"2026-11-18T22:09:57Z"}