Back to search

National Technical University of Athens (NTUA)

Unsupervised learning and explainable artificial intelligence in computer vision and remote sensing

Abstract

dc:description

In recent years, the use of Deep Neural Networks (DNNs) through supervised learning has led to significant advancements in computer vision and remote sensing. These systems, however, require large amounts of labeled data for high performance, making supervised learning labor-intensive, especially when collecting sufficient labels is challenging, such as in marine pollution data. Moreover, these DNNs often lack generalization to new data, excelling only in specific tasks or regions. To address these issues, unsupervised learning, particularly self-supervised learning, offers promise. DNNs trained this way can learn without labeled data, automatically generating training labels through "pretext tasks" like predicting image rotations or colorizations. However, the complexity of DNNs often leads to concerns and their characterization as "black boxes." To address this, Explainable Artificial Intelligence (XAI) methods have been developed to interpret the decisions of these complex systems. XAI seeks to explain why a model makes certain decisions, enhancing transparency and trust in these systems. XAI methods can be integrated into models for inherent interpretability or applied externally. Recent advancements in unsupervised learning employ Vision Transformers (ViT), which utilize attention layers to facilitate the interpretation of model outputs. However, the exploitation of these attention layers during the learning process has not yet been thoroughly explored. Additionally, while XAI techniques for model interpretation are well-established in computer vision, the studies in remote sensing and Earth observation are still limited. In this direction, the current dissertation explores and advances unsupervised learning techniques in self-supervised and object-centric learning, utilizing the attention layers of ViT. It also evaluates and applies XAI methods in remote sensing for land cover and marine pollution detection. In the first stage, the dissertation conducts a thorough evaluation of XAI methods aiming at improving the transparency and interpretability of deep neural network decisions in remote sensing classification. Utilizing land cover datasets like BigEarthNet and SEN12MS, the study explores various XAI techniques, such as Occlusion, Grad-CAM, and Lime, to interpret model predictions. This analysis highlights the strengths of these methods while identifying their limitations in terms of computational demands and the ability to produce high-resolution, interpretable outputs. The dissertation then focuses on unsupervised learning methods using ViT architectures that incorporate attention layers to make models inherently interpretable. Specifically, it explores the use of attention layers to improve model performance during the learning process. The dissertation particularly focuses on a sub-area of self-supervised learning known as Masked Image Modeling (MIM). In MIM, parts of an image are intentionally masked, and the model is trained to predict the hidden regions, thus enhancing its ability to recognize and interpret visual data. Traditionally, this masking is performed randomly. This dissertation introduces a novel strategy where the masking is guided by the attention maps of the last layer of a ViT. This approach, called Attention-Guided Masked Image Modeling (AttMask), accelerates the learning process and enhances performance over traditional masking strategies across various downstream tasks, with or without fine-tuning. Furthermore, this dissertation introduces the SPOT method, a novel approach that leverages Transformers’ attention layers to enhance Object-Centric Learning (OCL). SPOT adopts an encoder-decoder architecture, where the encoder generates object-specific representations known as slots. The decoder then utilizes these slots to reconstruct the image. The analysis demonstrates that the decoder’s attention maps outperform the encoder’s in object segmentation. SPOT’s innovation includes using these superior attention maps to refine the encoder through distillation. Moreover, SPOT introduces sequence permutations of image patches in the Autoregressive Transformer decoder, which amplifies the focus on initial patches, improving the learning process. This combined approach strengthens the supervisory signal and overall effectiveness, establishing SPOT as the new state-of-the-art approach in object-centric learning. In the final phase of the dissertation, the insights gained from the initial evaluation of XAI methods were applied to the critical environmental issue of marine pollution detection. In this direction, a new deep learning model named MariNeXt was developed, significantly outperforming existing baselines. Using the XAI method Grad-CAM, the dissertation not only enhances understanding and assessment of the model’s decisions beyond traditional accuracy metrics but also reveals valuable information regarding the detection of marine pollution using satellite data. Finally, aiming for a future operational solution, cloud detection algorithms crucial for the preprocessing of satellite data are evaluated in marine environments.

Degree

thesis:*
Grantor dc:publisher
National Technical University of Athens (NTUA)
Year dc:date
2024

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Kakogeorgiou, Ioannis
  • Κακογεωργίου, Ιωάννης

Subjects

dc:subject × 19

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/57115

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Kakogeorgiou, Ioannis; Κακογεωργίου, Ιωάννης. Unsupervised learning and explainable artificial intelligence in computer vision and remote sensing. National Technical University of Athens (NTUA), 2024. http://hdl.handle.net/10442/hedi/57115