Object detection is a fundamental task in computer vision, with significant applications in various domains. However, the reliance on large-scale annotated data and computational resource demands poses challenges in practical implementation. This thesis aims to address these complexities by exploring self-supervised training approaches for the detection transformer(DETR) family of object detectors. The project investigates the necessity of training the backbone under a semi-supervised setting and explores the benefits of initializing scene graph generation architectures with pretrained DETReg and DETR models for faster training convergence and reduced computational resource requirements. The significance of this research lies in the potential to mitigate the dependence on annotated data and make deep learning techniques more accessible to researchers and practitioners. By overcoming the limitations of data and computational resources, this thesis contributes to the accessibility of DETR and encourages a more sustainable and inclusive approach to deep learning research. / Objektigenkänning är en grundläggande uppgift inom datorseende, med betydande tillämpningar inom olika domäner. Dock skapar beroendet av storskaliga annoterade data och krav på datorkraft utmaningar i praktisk implementering. Denna avhandling syftar till att ta itu med dessa komplexiteter genom att utforska självövervakade utbildningsmetoder för detektions transformer (DETR) familjen av objektdetektorer. Projektet undersöker nödvändigheten av att träna ryggraden under en semi-övervakad inställning och utforskar fördelarna med att initiera scenegrafgenereringsarkitekturer med förtränade DETReg-modeller för snabbare konvergens av träning och minskade krav på datorkraft. Betydelsen av denna forskning ligger i potentialen att mildra beroendet av annoterade data och göra djupinlärningstekniker mer tillgängliga för forskare och utövare. Genom att övervinna begränsningarna av data och datorkraft, bidrar denna avhandling till tillgängligheten av DETR och uppmuntrar till en mer hållbar och inkluderande inställning till djupinlärning forskning.

[pt] O desenvolvimento de técnicas de aprendizado auto-supervisionado vem ganhando muita visibilidade na área de Visão Computacional pois possibilita o pré-treinamento de redes neurais profundas sem a necessidade de dados anotados. Em alguns domínios, as anotações são custosas, pois demandam muito trabalho especializado para a rotulação dos dados. Esse problema é muito comum no setor de Óleo e Gás, onde existe um vasto volume de dados não interpretados. O presente trabalho visa aplicar a técnica de aprendizado auto-supervisionado denominada Masked Autoencoders para pré-treinar modelos Vision Transformers com dados sísmicos. Para avaliar o pré-treino, foi aplicada a técnica de transfer learning para o problema de segmentação de fácies sísmicas. Na fase de pré-treinamento foram empregados quatro volumes sísmicos distintos. Já para a segmentação foi utilizado o dataset Facies-Mark e escolhido o modelo da literatura Segmentation Transformers. Para avaliação e comparação da performance da metodologia foram empregadas as métricas de segmentação utilizadas pelo trabalho de benchmarking de ALAUDAH (2019). As métricas obtidas no presente trabalho mostraram um resultado superior. Para a métrica frequency weighted intersection over union, por exemplo, obtivemos um ganho de 7.45 por cento em relação ao trabalho de referência. Os resultados indicam que a metodologia é promissora para melhorias de problemas de visão computacional em dados sísmicos. / [en] The development of self-supervised learning techniques has gained a lot of visibility in the field of Computer Vision as it allows the pre-training of deep neural networks without the need for annotated data. In some domains, annotations are costly, as they require a lot of specialized work to label the data. This problem is very common in the Oil and Gas sector, where there is a vast amount of uninterpreted data. The present work aims to apply the self-supervised learning technique called Masked Autoencoders to pre-train Vision Transformers models with seismic data. To evaluate the pre-training, transfer learning was applied to the seismic facies segmentation problem. In the pre-training phase, four different seismic volumes were used. For the segmentation, the Facies-Mark dataset was used and the Segmentation Transformers model was chosen from the literature. To evaluate and compare the performance of the methodology, the segmentation metrics used by the benchmarking work of ALAUDAH (2019) were used. The metrics obtained in the present work showed a superior result. For the frequency weighted intersection over union (FWIU) metric, for example, we obtained a gain of 7.45 percent in relation to the reference work. The results indicate that the methodology is promising for improving computer vision problems in seismic data.

DeepPaint is a semi-automated segmentation tool that utilises a U-net architecture to performbinary segmentation. To maximise the model’s performance and minimise user time, it isadvisable to apply Transfer Learning (TL) and reuse a model trained on a similar segmentationtask. However, due to the sensitivity of medical data and the unique properties of certainsegmentation tasks, TL is not feasible for some applications. In such circumstances, SelfSupervised Learning (SSL) emerges as the most viable option to minimise the time spent inDeepPaint by a user. Various pretext tasks, exploring both corruption segmentation and corruption restoration, usingsuperpixels and square patches, were designed and evaluated. With a limited number ofiterations in both the pretext and downstream tasks, significant improvements across fourdifferent datasets were observed. The results reveal that SSL models, particularly those pretrained on corruption segmentation tasks where square patches were corrupted, consistentlyoutperformed models without pre-training, with regards to a cumulative Dice SimilarityCoefficient (DSC). To examine whether a model could learn relevant features from a pretext task, Centred KernelAlignment (CKA) was used to measure the similarity of feature spaces across a model's layersbefore and after fine-tuning on the downstream task. Surprisingly, no significant positivecorrelation between downstream DSC and CKA was observed in the encoder, likely due to thelimited fine-tuning allowed. Furthermore, it was examined whether pre-training on the entiredataset, as opposed to only the training subset, yielded different downstream results. Asexpected, significantly higher DSC in the downstream task is more likely if the model hadaccess to all data during the pretext task. The differences in downstream segmentationperformance between models that accessed different data subsets during pre-training variedacross datasets.

Longitudinal velocity control, or adaptive cruise control (ACC), is a common advanced driving feature aimed at assisting the driver and reducing fatigue. It maintains the velocity of a vehicle and ensures a safe distance from the preceding vehicle. Many models for ACC are available, such as Proportional, Integral, and Derivative (PID) and Model Predictive Control (MPC). However, conventional models have some limitations as they are designed for simplified driving scenarios. Artificial intelligence (AI) and machine learning (ML) have made robust navigation and decision-making possible in complex environments. Recent approaches, such as reinforcement learning (RL), have demonstrated remarkable performance in terms of faster processing and effective navigation through unknown environments. This dissertation explores an RL approach, deep deterministic policy gradient (DDPG), for longitudinal velocity control. The baseline DDPG model has been modified in two different ways. In the first method, an attention mechanism has been applied to the neural network (NN) of the DDPG model. Integrating the attention mechanism into the DDPG model helps in decreasing focus on less important features and enhances overall model effectiveness. In the second method, the inputs of the actor and critic networks of DDPG are replaced with outputs of the self-supervised network. The self-supervised learning process allows the model to accurately predict future states from current states and actions. A custom reward function has been designed for the RL algorithm considering overall safety, efficiency, and comfort. The proposed models have been trained with human car-following data, and evaluated on multiple datasets, including publicly available data, simulated data, and sensor data collected from real-world environments. The analyses demonstrate that the new architectures can maintain strong robustness across various datasets and outperform the current state-of-the-art models.

Hyperspectral imaging is an expanding topic within the field of computer vision, that uses images of high spectral granularity. Contrastive learning is a discrim- inative approach to self-supervised learning, a form of unsupervised learning where the network is trained using self-created pseudo-labels. This work com- bines these two research areas and investigates how a pretrained network based on contrastive learning can be used for hyperspectral images. The hyperspectral images used in this work are generated from simulated RGB images and spec- tra from a spectral library. The network is trained with a pretext task based on data augmentations, and is evaluated through transfer learning and fine-tuning for a downstream task. The goal is to determine the impact of the pretext task on the downstream task and to determine the required amount of labelled data. The results show that the downstream task (a classifier) based on the pretrained network barely performs better than a classifier without a pretrained network. In the end, more research needs to be done to confirm or reject the benefit of a pretrained network based on contrastive learning for hyperspectral images. Also, the pretrained network should be tested on real-world hyperspectral data and trained with a pretext task designed for hyperspectral images.

Deep learning works best with vast andd well-distributed data collections. However, collecting and annotating large data sets can be very time-consuming and expensive. Moreover, deep learning is specific to domain knowledge, even with data and computation. E.g., models trained to classify animals would probably underperform when they classify vehicles. Although techniques such as domain adaptation and transfer learning have been popularised recently, tasks in cross-domain knowledge transfer have also taken off. However, most of these works are limited to computer vision. In the domain of time series, this is relatively underexplored. This thesis explores methods to use time series data from one domain to classify data generated from another domain via transfer learning. It focuses on using accelerometer data from running recordings to improve the classification performance on jumping data based on the apparent similarity of individual recordings. Thus, transfer learning and domain adaptation techniques were used to use the learning acquired through deep model training on running sequences. This thesis has performed four experiments to test this domain similarity. The first one consists of transforming time series with the continuous wavelet transform to get both time and frequency information. The model is then pre-trained within a contrastive learning framework. However, the continuous wavelet transformation (CWT) did not improve the classification results. The following two experiments consisted of pre-training the models with self-supervised learning. The first one with a contrastive pretext-task improved the classification results, and the resilience to data decrease. The second one with a forward forecasting pretext-task improved the results when all the data was available but was very sensitive to data decrease. Finally, the domain adaptation was tested and showed interesting performances on the classification task. Although some of the employed techniques did not show improvement, pre-training using contrastive learning on the running dataset has shown great improvement to classify the jumping dataset. / Djupinlärning fungerar bäst med stora och väl distribuerade datasamlingar. Det kan dock vara mycket tidskrävande och dyrt att samla in och kommentera stora datamängder. Även med alla data och beräkningar är djupinlärning specifik för domänkunskap. Exempelvis skulle modeller som tränats för att klassificera djur förmodligen underprestera när de klassificerar fordon. Även om tekniker som domänanpassning och överföringsinlärning har populariserats på senare tid, har även uppgifter inom kunskapsöverföring mellan olika domäner tagit fart. De flesta av dessa arbeten är dock begränsade till datorseende. Inom tidsseriernas område är detta relativt outforskat. I den här avhandlingen undersöks metoder för att använda tidsseriedata från en domän för att klassificera data från en annan domän med hjälp av djupinlärning. Fokus ligger på att använda accelerometerdata från löpning för att förbättra klassificeringen av hoppdata, baserat på den uppenbara likheten mellan löpning och hoppning. Således användes tekniker för överföringsinlärning och domänanpassning för att använda den inlärning som förvärvats genom träning av djupa modeller på löpsekvenser. I den här avhandlingen har fyra experiment utförts för att testa denna domänlikhet. Det första består av att omvandla tidsserier med den kontinuerliga wavelettransformen för att få fram både tids- och frekvensinformation. Modellen förtränas sedan inom en ram för kontrastiv inlärning. Användningen av CWT förbättrade dock inte klassificeringsresultaten. De följande två experimenten bestod av att förträna modellerna med självövervakad inlärning. Det första försöket med en kontrasterande förtextuppgift förbättrade klassificeringsresultaten och motståndskraften mot dataförlust. Det andra försöket med en prognostiserande förtextuppgift förbättrade resultaten när alla data var tillgängliga, men var mycket känslig för dataförlust. Slutligen testades domänanpassningen och visade intressanta resultat i klassificeringsuppgiften. Även om några av de använda teknikerna inte visade någon förbättring, har förträning med hjälp av kontrastinlärning på löpande dataset visat sig ge stora förbättringar när det gäller klassificering av hoppdata.

<p>Deep neural networks have made significant advances in computer vision. However, several challenges limit their real-world applications. For example, domain shifts in vision data degrade model performance; visual appearance variances affect model robustness; it is also non-trivial to extend a model trained on one task to novel tasks; and in many applications, large-scale labeled data are not even available for learning powerful deep models from scratch. This research focuses on improving the transferability of deep features and the efficiency of deep vision model adaptation, leading to enhanced generalization and new capabilities on computer vision tasks. Specifically, we approach these problems from the following two directions: architectural adaptation and label-efficient transferable feature learning. From an architectural perspective, we investigate various schemes that permit network adaptation to be parametrized by multiple copies of sub-structures, distributions of parameter subspaces, or functions that infer parameters from data. We also explore how model adaptation can bring new capabilities, such as continuous and stochastic image modeling, fast transfer to new tasks, and dynamic computation allocation based on sample complexity. From the perspective of feature learning, we show how transferable features emerge from generative modeling with massive unlabeled or weakly labeled data. Such features enable both image generation under complex conditions and downstream applications like image recognition and segmentation. By combining both perspectives, we achieve improved performance on computer vision tasks with limited labeled data, enhanced transferability of deep features, and novel capabilities beyond standard deep learning models.</p>

<p>Graph representation learning (GRL) has been increasing used to model and understand data from a wide variety of complex systems spanning social, technological, bio-chemical and physical domains. GRL consists of two main components (1) a parametrized encoder that provides representations of graph data and (2) a learning process to train the encoder parameters. Designing flexible encoders that capture the underlying invariances and characteristics of graph data are crucial to the success of GRL. On the other hand, the learning process drives the quality of the encoder representations and developing principled learning mechanisms are vital for a number of growing applications in self-supervised, transfer and federated learning settings. To this end, we propose a suite of models and learning algorithms for GRL which form the two main thrusts of this dissertation.</p> <p><br></p> <p>In Thrust I, we propose two novel encoders which build upon on a widely popular GRL encoder class called graph neural networks (GNNs). First, we empirically study the prediction performance of current GNN based encoders when applied to graphs with heterogeneous node mixing patterns using our proposed notion of local assortativity. We find that GNN performance in node prediction tasks strongly correlates with our local assortativity metric---thereby introducing a limit. We propose to transform the input graph into a computation graph with proximity and structural information as distinct types of edges. We then propose a novel GNN based encoder that operates on this computation graph and adaptively chooses between structure and proximity information. Empirically, adopting our transformation and encoder framework leads to improved node classification performance compared to baselines in real-world graphs that exhibit diverse mixing.</p> <p>Secondly, we study the trade-off between expressivity and efficiency of GNNs when applied to temporal graphs for the task of link ranking. We develop an encoder that incorporates a labeling approach designed to allow for efficient inference over the candidate set jointly, while provably boosting expressivity. We also propose to optimize a list-wise loss for improved ranking. With extensive evaluation on real-world temporal graphs, we demonstrate its improved performance and efficiency compared to baselines.</p> <p><br></p> <p>In Thrust II, we propose two principled encoder learning mechanisms for challenging and realistic graph data settings. First, we consider a scenario where only limited or even no labelled data is available for GRL. Recent research has converged on graph contrastive learning (GCL), where GNNs are trained to maximize the correspondence between representations of the same graph in its different augmented forms. However, we find that GNNs trained by traditional GCL often risk capturing redundant graph features and thus may be brittle and provide sub-par performance in downstream tasks. We then propose a novel principle, termed adversarial-GCL (AD-GCL), which enables GNNs to avoid capturing redundant information during the training by optimizing adversarial graph augmentation strategies used in GCL. We pair AD-GCL with theoretical explanations and design a practical instantiation based on trainable edge-dropping graph augmentation. We experimentally validate AD-GCL by comparing with state-of-the-art GCL methods and achieve performance gains in semi-supervised, unsupervised and transfer learning settings using benchmark chemical and biological molecule datasets. </p> <p>Secondly, we consider a scenario where graph data is silo-ed across clients for GRL. We focus on two unique challenges encountered when applying distributed training to GRL: (i) client task heterogeneity and (ii) label scarcity. We propose a novel learning framework called federated self-supervised graph learning (FedSGL), which first utilizes a self-supervised objective to train GNNs in a federated fashion across clients and then, each client fine-tunes the obtained GNNs based on its local task and available labels. Our framework enables the federated GNN model to extract patterns from the common feature (attribute and graph topology) space without the need of labels or being biased by heterogeneous local tasks. Extensive empirical study of FedSGL on both node and graph classification tasks yields fruitful insights into how the level of feature / task heterogeneity, the adopted federated algorithm and the level of label scarcity affects the clients’ performance in their tasks.</p>

Eye-tracking is a method for monitoring and measuring eye movements. The technology has had a significant impact so far and new application areas are emerging. Today, the technology is used in the gaming industry, health industry, self-driving cars, and not least in medicine. In the latter, large research resources are invested to investigate the extent to which eye-tracking can help with disease diagnostics. One disease of interest is Parkinson’s disease, a neuro-degenerative disease in which the dopamine production in nerve cells is destroyed. This leads to detoriating nerve signal transmission, which in turn affects the motor skills. One of the affected motor functions associated with PD is the oculomotor function, affecting the eye function. The declination can be observed clinically by physicians, however eye-tracking technology has a high potential here, but it remains to investigate which methodology and which test protocols are relevant to study and to what extent the technology can be used as a diagnostic tool. A novel class of algorithms for finding representations of data is called self-supervised learning (SSL). The class of algorithms seems to have a high potential in terms of categorizing biomarkers. This thesis examines to which extent an SSL network can learn representations of eye-tracking data on Parkinson’s patients, in order to distinguish between healthy and sick, patients on and off medication. The result suggests that the network does not succeed in learning distinct differences between groups. Furthermore, no difference is observed in the result when we in the model take into account the task-specific target information that the subjects are following. Today in the UK approximately 26 percent of Parkinson’s patients are misdiagnosed. In the initial state of the disease, the misdiagnosis is even higher. Potentially, the method can be used as a complement to regular diagnosis in different stages of the disease. This would provide better conditions for the patient as well as for medical and pharmaceutical research. The method also has the potential to reduce physicians’ workload. / Eye-tracking eller ögonrörelsemätning som är den svenska termen, är en metod för att följa och mäta ögats rörelser. Tekniken har fått en betydande genomslagskraft hittills och nya applikationsområden dyker upp titt som tätt. Idag används tekniken inom spelindustrin, hälsa, i självkörande bilar och inte minst inom medicin. Inom det senare läggs idag stora forskningsresurser för att undersöka i vilken utsträckning eye-tracking kan hjälpa till att diagnosticera sjukdomar. En sjukdom av intresse är Parkinson’s sjukdom, vilket är en neurodegenerativ sjukdom där dopaminproduktionen i nervceller förstörs. Det leder till att transmissionen av nervsignaler försämras som i sin tur gör att motoriken påverkas vilket bland annat leder till en nedsättning i ögats motorik. Det är något som man idag kan observera kliniskt, eye-tracking teknik har här en hög potential men det återstår att undersöka vilken metodik och vilka testprotokoll som är relevanta att undersöka och i vilken grad tekniken kan användas som ett diagnostiskt verktyg. En ny typ av algoritmer för att hitta representationer av data kallas för self-supervised learning (SSL), dessa algoritmer verkar ha en hög potential vad gäller kategorisering av biomarkörer. I denna uppsats undersöks i vilken grad ett SSL-nätverk kan lära sig representationer av eye-tracking data på Parkinson’s patienter för att kunna särskilja mellan friska och sjuka, medicinerade och omedicinerade. Resultatet är att nätverket inte lyckas lära sig skiljaktigheter mellan dessa klasser. Vidare noteras ingen skillnad i resultatet då vi i modellen tar hänsyn till de specifika uppgifterna som försökspersonerna fått. Idag får 30 procent av parkinsonpatienterna fel diagnos. I ett initialt tillstånd av sjukdomen är feldiagnosticeringen ännu högre. Potentiellt kan metoden användas som komplement till diagnosticering i olika skeden av sjukdomen. Detta skulle ge bättre förutsättningar för såväl patienten som för den medicinska och farmaceutiska forskningen. Metoden har dessutom potential att minska läkares arbetsbörda.

Transformers have shown remarkable results in the domains of Natural Language Processing and Computer Vision. This naturally raises the question whether the success could be replicated in other domains. However, due to Transformers being inherently data hungry and sensitive to weight initialization, applying the Transformer to new domains is quite a challenging task. Previously, the data demands have been met using large scale supervised or self-supervised pre-training on a similar task before supervised fine-tuning on a target down stream task. We show that Transformers are applicable for the task of multi-label error classification of trace data, and that masked data modelling based self-supervised learning methods can be used to leverage unlabelled data to increase performance compared to a baseline supervised learning approach. / Transformers har visat upp anmärkningsvärda resultat inom områdena Natural Language Processing och Computer Vision. Detta väcker naturligtvis frågan om dessa framgångar kan upprepas inom andra områden. På grund av att transformatorer i sig är datahungriga och känsliga för initialisering av vikt är det dock en utmaning att tillämpa transformatorn på nya områden. Tidigare har datakraven tillgodosetts med hjälp av storskalig övervakad eller självövervakad förträning på en liknande uppgift före övervakad finjustering på en måluppgift i efterföljande led. Vi visar att transformatorer kan användas för klassificering av spårdata med flera etiketter och att metoder för självövervakad inlärning som bygger på modellering av maskerade data kan användas för att utnyttja omärkta data för att öka prestandan jämfört med en grundläggande övervakad inlärningsmetod.