National Technical University of Athens (NTUA)
Unlocking the potential of artificial intelligence for satellite image processing
Abstract
dc:descriptionArtificial intelligence (AI) encompasses the execution of tasks typically associated with intelligent entities, carried out by machines. Advances in neuroscience since the late 19th century inspired the creation of the “perceptron” in 1958, which is a mathematical model of a biological neuron. Since then, artificial neural networks (ANNs), an AI method that is inspired by the human brain, have shown great progress in various tasks. The increased computational power provided in the last decade was among the main triggers of the field. ANNs exhibit their potential mainly in “Big Data” tasks where they outperform other methods. Thus, research attention has been attracted to satellite Earth observation (EO) where large data volumes are frequently collected. The main positive points are independence from feature engineering, high flexibility, and spatial perception in image processing, while negative points are the time-consuming creation of annotations and the low interpretability. In this doctoral dissertation, the capabilities of ANNs were investigated in four EO applications: cloud masking in Sentinel-2 (S2) data, VHR change detection (CD), marine plastic litter detection through image fusion, and RGB-to-NIR image-to-image translation (ITIT). Cloud masking is a crucial pre-processing step in EO data analysis because it excludes clouds from optical imagery. Threshold-based methods, which are still the golden rule in this task, exhibit difficulties in challenging cases which include the presence of thin clouds (omission error) and bright non-cloud objects (commission error). To mitigate the above-mentioned challenges, three studies were performed in this thesis on S2 data. In the first study, a multi-layer perceptron (MLP) architecture was implemented that yielded superior results compared to state-of-the-art rule-based and muti-temporal methods in the separation of clouds from deep water spectra with noise and sunglint. Directional reflectance effects were also considered. For the purpose of the study, a relevant manual dataset was created and publicly released since equivalent datasets dο not exist in the literature. Interesting findings were the possibility of producing a positive effect when applying feature scaling by using the parameters of the test set instead of the training set, and the definition of the important bands in mitigating the spectra with noise and sunglint by employing the network weights. In the second study, a novel fine-tuning methodology for self-organizing maps (SOMs) was developed that successfully rectified the misclassified predictions of bright non-cloud spectra in land areas. SOMs are ANNs that carry topological properties. The proposed approach, applied to the output of the non-fine-tuned network, is task-independent and requires only small amounts of data. In addition, it eliminates the necessity for additional training. It is performed by pinpointing the neurons that correspond to the incorrectly predicted bright non-cloud objects and altering their labels. In the third study, a patch-to-pixel convolutional neural network (CNN) was created that effectively identified semi-transparent clouds and separated bright clouds from bright non-cloud objects. CNNs are ANNs that are inspired by the human vision. The model underwent evaluation on the first publicly available annotated cloud masking image dataset, which allowed for a robust and objective evaluation. The study further reinforced the value of CNNs in applications where spatial context is crucial and demonstrated that lightweight architectures can be successful in cloud masking. CD in the context of EO is the task of monitoring land cover transitions through time. When performed in VHR data, it is possible to detect changes in smaller objects such as buildings. However, the complexity of the task increases because of heightened within-class variance and geometric registration errors. Traditional pixel and object-based methods are more successful in high/medium resolution data where residual misregistration is less important. Recently, convolutional deep learning (DL) has contributed to the VHR CD since CNNs inherently possess spatial context perception. However, the research has predominantly concentrated on images with minor co-registration errors and the evaluation is typically not conducted on highly dissimilar test datasets. In an attempt to reduce this research gap, in this thesis, a comparative study was conducted where several state-of-the-art DL CD methods and automatic co-registration methods were assessed on VHR images with severe co-registration errors. The images were collected from European areas with versatile urban patterns and the challenges included geometric distortions and radiometric differences, as well as seasonal and vehicle-related changes. The diversity between the training sets and the study data also posed a challenge for the supervised methods. The evaluation was reinforced by a novel proposed score that provides a better understanding of the magnitude of the commission error. It was shown that an FFT-based method that uses phase correlation produced the most satisfactory co-registration results and a network called STANet outperformed the other methods in building-related changes. The STANet performance can be credited to the synergy between the spatial attention mechanism and a substantial annotated dataset. Marine litter exerts a wide spectrum of both adverse environmental and socio-economic effects. Plastic, which is its dominant component, constitutes the most significant hazard. Recent research employing satellite sensors has shown promising results in the detection of large-sized marine debris, however, the field is still in its infancy. High spatial and spectral resolutions are two critical factors in enhancing the detection and discrimination ability. However, in the present satellite sensors there are critical trade-offs in this regard. In this thesis, under the assumption that this issue could be alleviated by image fusion, two studies were performed that focused on the increase of spatial resolution in either the PRISMA or the S2 satellites. In the first study, the potential of HS satellite imagery in marine plastic litter detection was investigated for the first time through PRISMA data. The research centered on identifying small-sized targets (≤ 5 m) specifically designed for the experiment, adding an extra layer of complexity. Several literature conventional and state-of-the-art DL pansharpening approaches were evaluated. A PCA-based substitution method showed the best results as it efficiently separated plastic from water spectra without producing distortions on the output images. In the DL methods (originally introduced in the literature for VHR data), spatial distortions were encountered due to the large difference between the spatial resolutions of the PAN and the HS bands and the lack of ground-truth data. However, the importance of histogram clipping as a pre-processing step was established since random water spectra were effectively separated from the target spectra. In the absence of SWIR features, spectral VNIR characteristics were exploited and an intersection of the outputs of three novel marine plastic indexes was applied on the PCA image that detected plastics with size equal to 8% HS pixel coverage. In the second study, S2 and WV-3 images were fused since the SWIR information available in S2 and absent in WV-3 is valuable for the detection and identification of plastics. Several conventional and DL image fusion approaches were evaluated in terms of spatial and spectral accuracy on artificial plastic targets. CNMF showed the best performance overall and a GAN- and a ResNet-based model (created for the purpose of the study) outperformed all methods in terms of spectral similarity. Important findings were: a) the adequacy of VNIR WV-3 information in generating the most effective output, enhancing the chances of achieving temporally close acquisitions, b) the reinforcement of the significance of the SWIR information in detecting plastic, and c) the observation of dissimilarities in the spectral regions of S2 bands between the signatures of the various plastic materials. It is noted that the conventional image fusion methods in both studies were carried out by M. Kremezi. Image-to-image translation (ITIT) refers to an image processing technique that aims to learn the mapping functions between an input and an output image and can be either performed in a paired (co-registered input and output) or an unpaired setting. Lately, the EO community has exhibited a heightened interest in DL paired ITIT by typically employing conditional GANs (cGANs) to synthesize missing information in several applications. RGB-to-NIR ITIT, where this thesis focused on, has been either addressed indirectly in the context of creating HS outputs (spectral super-resolution) or has been exclusively directed towards vegetation applications. Regarding unpaired ITIT, it has predominantly been utilized in VHR data as an intermediary step to improve the quality of cross-domain semantic segmentation (unsupervised domain adaptation). In this thesis, attempting to contribute to the RGB-to-NIR ITIT literature, a thorough study was performed that focused on three main land cover categories (impervious, vegetation, ground) on heterogeneous bi-temporal VHR images. Through a three-stage GAN framework, both paired and unpaired data were exploited and several network configurations were explored in order to predict satisfactory NIR outputs in in- and out-domain data (do not belong to the domain of the training set). The paired data experiments, which were run in an in- and out-domain setting, showed that cGANs produced adequate NIR predictions even in out-domain cases when the domain gap was not significantly high and that instance normalization performed better than batch normalization, especially on data with low representation on the training set. The unpaired data experiments managed to enhance the NIR prediction in the vegetation category when high dissimilarities existed in the respective RGB domains.
Degree
thesis:*- Grantor dc:publisher
- National Technical University of Athens (NTUA)
- Year dc:date
- 2024
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Kristollari, Viktoria
- Κριστολλάρη, Βικτωρία
Subjects
dc:subject × 22- Βαθιά μηχανική μάθηση
- Ανίχνευση νεφών
- Ανίχνευση μεταβολών
- Πλαστικά θαλάσσια απορρίμματα
- Πρόβλεψη εγγύς υπέρυθρου καναλιού
- Deep learning
- Cloud masking
- Change detection
- Marine plastic litter
- Image-to-image translation
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Φυσικές Επιστήμες
- Τεχνητή νοημοσύνη
- Περιβαλλοντική Μηχανική
- Επιστήμες Μηχανικού και Τεχνολογία
- Τηλεπισκόπηση
- Computer and Information Sciences
- Natural Sciences
- Artificial Intelligence
- Environmental Engineering
- Engineering and Technology
- Remote Sensing
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/57114