In representation learning we are interested in how data is represented by different models. Representations from different models are often compared by training a new model on a downstream task using the representations and testing their performance. However, this method is not always applicable and it gives limited insight into the representations. In this thesis, we compare natural image representations from classification models and the generative model BigGAN using two other approaches. The first approach compares the geometric clustering of the representations and the second approach compares if the pairwise similarity between images is similar between different models. All models are large pre-trained models trained on ImageNet and the representations are taken as middle layers of the neural networks. A variety of experiments are performed using these approaches. One of the main results of this thesis shows that the representations of different classes are geometrically separated in all models. The experiments also show that there is no significant geometric difference between representations from training data and representations from validation data. Additionally, it was found that the similarity of representations between different models was approximately the same between the classification models AlexNet and ResNet as well as between the classification models and the BigGAN generator. They were also approximately equally similar to each other as they were to the class embedding of the BigGAN generator. Along with the experiment results, this thesis also provide several suggestions for future work in representation learning since a large number of research questions were explored. / Detta verk studerar representationer från artificiella neuronnät. Representationerna tas som värdena på ett lager i mittendelen av neuronnätet. Eftersom dessa representationer har flera olika användningsområden är syftet att jämföra dem från olika modeller. Ofta jämförs representationer genom att testa hur bra de är som input till en ny modell med ett nytt mål; alltså hur bra representationerna är att använda inom “transfer learning”. Denna metod ger begränsad information om representationerna och är inte alltid applicerbar. Detta verk använder därför två andra tillvägagångssätt för att jämföra representationer. Den första är att jämföra geometriska grupperingar hos olika representationer. Den andra använder ett mått av hur lika olika representationer är. Flera olika experiment utförs med hjälp av dessa tillvägagångssätt. Representationerna kommer frånmodeller som redan tränats på ImageNet. Både klassifikationsmodeller och en generativa modell används med syfte att också jämföra dem med varandra. Det första huvudresultatet från experimenten är att det finns en tydlig geometrisk separation av representationer från olika klasser i modellerna. Experimenten visar också att det inte fanns en tydlig geometrisk separation av representationer från träningsdata och valideringsdata. Ett annat resultat är att representationerna från de olika klassifikationsmodellerna AlexNet och ResNet är ungefär lika lika varandra som mellan klassifikationsmodellerna och generatorn hos den generativa modellen BigGAN. Resultaten visar också att de har en liknande likhet till BigGANs “class embedding”. Fler forskningsfrågor undersöks i andra experiment. Utöver experimenten kommer detta verk med många idéer till framtida forskning.

No description available.

Spatiotemporal Predictive Learning (SPL) is an essential research topic involving many practical and real-world applications, e.g., motion detection, video generation, precipitation forecasting, and traffic flow prediction. The problems and challenges of this field come from numerous data characteristics in both time and space domains, and they vary depending on the specific task. For instance, spatial analysis refers to the study of spatial features, such as spatial location, latitude, elevation, longitude, the shape of objects, and other patterns. From the time domain perspective, the temporal analysis generally illustrates the time steps and time intervals of data points in the sequence, also known as interval recording or time sampling. Typically, there are two types of time sampling in temporal analysis: regular time sampling (i.e., the time interval is assumed to be fixed) and the irregular time sampling (i.e., the time interval is considered arbitrary) related closely to the continuous-time prediction task when data are in continuous space. Therefore, an efficient spatiotemporal predictive method has to model spatial features properly at the given time sampling types. In this thesis, by taking advantage of Machine Learning (ML) and Deep Learning (DL) methods, which have achieved promising performance in many complicated computational tasks, we propose three DL-based models used for Spatiotemporal Sequence Prediction (SSP) with several types of time sampling. First, we design the Trajectory Gated Recurrent Unit Attention (TrajGRU-Attention) with novel attention mechanisms, namely Motion-based Attention (MA), to improve the performance of the standard Convolutional Recurrent Neural Networks (ConvRNNs) in the SSP tasks. In particular, the TrajGRU-Attention model can alleviate the impact of the vanishing gradient, which leads to the blurry effect in the long-term predictions and handle both regularly sampled and irregularly sampled time series. Consequently, this model can work effectively with different scenarios of spatiotemporal sequential data, especially in the case of time series with missing time steps. Second, by taking the idea of Neural Ordinary Differential Equations (NODEs), we propose Trajectory Gated Recurrent Unit integrating Ordinary Differential Equation techniques (TrajGRU-ODE) as a continuous time-series model. With Ordinary Differential Equation (ODE) techniques and the TrajGRU neural network, this model can perform continuous-time spatiotemporal prediction tasks and generate resulting output with high accuracy. Compared to TrajGRU-Attention, TrajGRU-ODE benefits from the development of efficient and accurate ODE solvers. Ultimately, we attempt to combine those two models to create TrajGRU-Attention-ODE. NODEs are still in their early stage of research, and recent ODE-based models were designed for many relatively simple tasks. In this thesis, we will train the models with several video datasets to verify the ability of the proposed models in practical applications. To evaluate the performance of the proposed models, we select four available spatiotemporal datasets based on the complexity level, including the MovingMNIST, MovingMNIST++, and two real-life datasets: the weather radar HKO-7 and KTH Action. With each dataset, we train, validate, and test with distinct types of time sampling to justify the prediction ability of our models. In summary, the experimental results on the four datasets indicate the proposed models can generate predictions properly with high accuracy and sharpness. Significantly, the proposed models outperform state-of-the-art ODE-based approaches under SSP tasks with different circumstances of interval recording.

Over the past few decades, the demand for energy-efficient treatment processes to reduce carbon emissions and the need for high performance materials in advanced engineering applications have posed significant challenges for materials scientists. This research first investigates the influence of high magnetic fields during heat treatment an energy efficient alternative to conventional processes on the microstructural evolution and mechanical properties of hypoeutectoid steels. The study demonstrates how magnetic fields affect phase transformations, microstructural features, and mechanical behavior. To establish a robust structure-property relationship and enable microstructural tailoring for targeted mechanical properties, an end-to-end computational framework integrating experimental characterization, physics based finite element simulations, and deep learning techniques is developed. Additionally, a mesoscale finite element model is constructed for fully pearlitic steels to simulate plastic deformation and damage, calibrated and validated using experimental data. A deep learning-based approach is then applied to analyze the structure-property relationships and design pearlite lamellae for optimized mechanical performance. Furthermore, the study extends to bio-inspired materials, investigating Nacre like structures for topology optimization aimed at enhancing mechanical properties and wave filtering capabilities. The dynamic behavior of these metamaterials is examined, revealing how hierarchical design influences their multifunctional properties. The findings of this research contribute to advancing the understanding of magnetic field assisted heat treatment for ferrous alloys, providing a computational framework for mesoscale plastic deformation and damage modeling in metallic systems, and developing methodologies for forward and inverse structural design targeting specific engineering applications. These insights pave the way for optimizing materials to achieve superior performance while promoting sustainable and efficient manufacturing processes. / Doctor of Philosophy / In recent years, the demand for stronger, more durable materials and energy efficient manufacturing processes has grown significantly. This research explores how applying a magnetic field during heat treatment can influence the microstructure and mechanical properties of hypoeutectoid steels, a widely used class of structural materials. The study shows that magnetic fields can alter phase transformations, leading to improved material performance while offering a more energy efficient alternative to traditional heat treatment methods. To better understand and design materials with specific properties, a computational approach combining experiments, simulations, and artificial intelligence is developed. This framework helps analyze the relationship between a material's structure and its mechanical properties, allowing for the design of optimized microstructures with enhanced strength and durability. Additionally, the study investigates Nacre like bioinspired composites that mimic natural structures found in seashells using machine learning techniques to improve their mechanical properties and ability to filter vibrations. By integrating advanced computational tools with experimental data, this research provides new ways to develop high performance materials more efficiently, with potential applications in industries such as aerospace, automotive, and infrastructure.

Cardinality estimation is a fundamental task in database query processing and optimization. Unfortunately, the accuracy of traditional estimation techniques is poor resulting in non-optimal query execution plans. With the recent expansion of machine learning into the field of data management, there is the general notion that data analysis, especially neural networks, can lead to better estimation accuracy. Up to now, all proposed neural network approaches for the cardinality estimation follow a global approach considering the whole database schema at once. These global models are prone to sparse data at training leading to misestimates for queries which were not represented in the sample space used for generating training queries. To overcome this issue, we introduce a novel local-oriented approach in this paper, therefore the local context is a specific sub-part of the schema. As we will show, this leads to better representation of data correlation and thus better estimation accuracy. Compared to global approaches, our novel approach achieves an improvement by two orders of magnitude in accuracy and by a factor of four in training time performance for local models.

<p>Spiking Neural Networks (SNNs) have recently attracted significant research interest as the third generation of artificial neural networks that can enable low-power event-driven data analytics. We propose ANN-SNN conversion using “soft re-set” spiking neuron model, referred to as Residual Membrane Potential (RMP) spiking neuron, which retains the “resid- ual” membrane potential above threshold at the firing instants. In addition, we propose a time-based coding scheme, named Temporal-Switch-Coding (TSC), and a corresponding TSC spiking neuron model. Each input image pixel is presented using two spikes with opposite polarity and the timing between the two spiking instants is proportional to the pixel intensity. We demonstrate near loss-less ANN-SNN conversion using RMP neurons for VGG-16, ResNet-20, and ResNet-34 SNNs on challenging datasets including CIFAR-10, CIFAR-100, and ImageNet. With the help of TSC coding, it achieves 7-14.5× less inference latency, and 30-60× fewer addition operations and memory accesses per inference across datasets compared to the state of the art (SOTA) SNN models. In the second part of the thesis, we propose a new type of recurrent neural network (RNN) architecture, named Os- cillatory Fourier Neural Network (O-FNN). We demonstrate that O-FNN is mathematically equivalent to a simplified form of Discrete Fourier Transform applied onto periodical activa- tion. In particular, the computationally intensive back-propagation through time in training is eliminated, leading to faster training while achieving the SOTA inference accuracy in a diverse group of sequential tasks. For instance, applying the proposed model to sentiment analysis on IMDB review dataset reaches 89.4% test accuracy within 5 epochs, accompanied by over 35x reduction in the model size compared to Long Short-Term Memory (LSTM). The proposed novel RNN architecture is well poised for intelligent sequential processing in resource constrained hardware.</p>

Данная работа посвящена применению моделей глубокого обучения для решения этой проблемы в процессе разработки программного обеспечения. Разработан стенд для имитации процесса разработки ПО, на котором были сгенерированы синтетические данные логов из различных сервисов. Объединение разнородных логов позволило создать реалистичный набор данных со скрытыми зависимостями для более сложной задачи поиска аномалий. На созданном наборе данных были применены модели глубокого обучения DeepLog, LogAnomaly и LogBERT. Для каждой модели выполнено обучение и оценка эффективности обнаружения аномалий на тестовой выборке. Разработанный стенд может усложняться и использоваться для дальнейших исследований в области применения глубокого обучения к задаче поиска аномалий в логах в процессе разработки ПО. / This paper focuses on the application of deep learning models to address this problem in the software development. A simulation framework was developed to imitate the software development by generating synthetic log data from different services. Combining heterogeneous logs allowed the creation of a realistic dataset with hidden dependencies for a more complex anomaly search task. DeepLog, LogAnomaly and LogBERT deep learning models were applied on the created dataset. For each model, training and evaluation of anomaly detection performance on a test sample was performed. The developed framework can be extended and used for further research in the application of deep learning to the task of searching for anomalies in logs during software development.

The growing urgency of climate change has led to growth in the electrification technology field, where batteries have emerged as an essential role in the renewable energy transition, supporting the implementation of environmentally friendly technologies such as smart grids, energy storage systems, and electric vehicles. Battery cell degradation is a common occurrence indicating battery usage. Optimizing lithium-ion battery degradation during operation benefits the prediction of future degradation, minimizing the degradation mechanisms that result in power fade and capacity fade. This degree project aims to investigate battery degradation prediction based on capacity using deep learning methods. Through analysis of battery degradation and health prediction for lithium-ion cells using non-destructive techniques. Such as electrochemical impedance spectroscopy obtaining ECM and three different deep learning models using multi-channel data. Additionally, the AI models were designed and developed using multi-channel data and evaluated performance within MATLAB. The results reveal an increased resistance from EIS measurements as an indicator of ongoing battery aging processes such as loss o active materials, solid-electrolyte interphase thickening, and lithium plating. The AI models demonstrate accurate capacity estimation, with the LSTM model revealing exceptional performance based on the model evaluation with RMSE. These findings highlight the importance of carefully managing battery charging processes and considering factors contributing to degradation. Understanding degradation mechanisms enables the development of strategies to mitigate aging processes and extend battery lifespan, ultimately leading to improved performance.

Image Super-Resolution (ISR) is a technology that aims to increase image resolution while preserving as much content and detail as possible. In this study, we evaluate four different Deep Learning models (EDSR, LapSRN, ESPCN, and FSRCNN) to determine their effectiveness in increasing the resolution of lowresolution images. The study builds on previous research in the field as well as the results of the comparison between the different deep learning models. The problem statement for this study is: “Which of the four Deep Learning-based models, EDSR, LapSRN, ESPCN, and FSRCNN, generates an upscaled image with the best quality from a low-resolution image on a dataset of Abyssinian cats, with a factor of four, based on quantitative results?” The study utilizes a dataset consisting of pictures of Abyssinian cats to evaluate the performance and results of these different models. Based on the quantitative results obtained from RMSE, PSNR, and Structural Similarity (SSIM) measurements, our study concludes that EDSR is the most effective Deep Learning-based model. / Bildsuperupplösning (ISR) är en teknik som syftar till att öka bildupplösningen samtidigt som så mycket innehåll och detaljer som möjligt bevaras. I denna studie utvärderar vi fyra olika Deep Learning modeller (EDSR, LapSRN, ESPCN och FSRCNN) för att bestämma deras effektivitet när det gäller att öka upplösningen på lågupplösta bilder. Studien bygger på tidigare forskning inom området samt resultatjämförelser mellan olika djupinlärningsmodeller. Problemet som studien tar upp är: “Vilken av de fyra Deep Learning-baserade modellerna, EDSR, LapSRN, ESPCN och FSRCNN generarar en uppskalad bild med bäst kvalité, från en lågupplöst bild på ett dataset med abessinierkatter, med skalningsfaktor fyra, baserat på kvantitativa resultat?” Studien använder en dataset av bilder på abyssinierkatter för att utvärdera prestandan och resultaten för dessa olika modeller. Baserat på de kvantitativa resultaten som erhölls från RMSE, PSNR och Structural Similarity (SSIM) mätningar, drar vår studie slutsatsen att EDSR är den mest effektiva djupinlärningsmodellen.

Time series anomaly detection has been a longstanding area of research with applications across various domains. In recent years, there has been a surge of interest in applying deep learning models to this problem domain. This thesis presents a critical examination of the efficacy of deep learning models in comparison to classical approaches for time series anomaly detection. Contrary to the widespread belief in the superiority of deep learning models, our research findings suggest that their performance may be misleading and the progress illusory. Through rigorous experimentation and evaluation, we reveal that classical models outperform deep learning counterparts in various scenarios, challenging the prevailing assumptions. In addition to model performance, our study delves into the intricacies of evaluation metrics commonly employed in time series anomaly detection. We uncover how it inadvertently inflates the performance scores of models, potentially leading to misleading conclusions. By identifying and addressing these issues, our research contributes to providing valuable insights for researchers, practitioners, and decision-makers in the field of time series anomaly detection, encouraging a critical reevaluation of the role of deep learning models and the metrics used to assess their performance. / Tidsperiods avvikelsedetektering har varit ett långvarigt forskningsområde med tillämpningar inom olika områden. Under de senaste åren har det uppstått ett ökat intresse för att tillämpa djupinlärningsmodeller på detta problemområde. Denna avhandling presenterar en kritisk granskning av djupinlärningsmodellers effektivitet jämfört med klassiska metoder för tidsperiods avvikelsedetektering. I motsats till den allmänna övertygelsen om överlägsenheten hos djupinlärningsmodeller tyder våra forskningsresultat på att deras prestanda kan vara vilseledande och framsteg illusoriskt. Genom rigorös experimentell utvärdering avslöjar vi att klassiska modeller överträffar djupinlärningsalternativ i olika scenarier och därmed utmanar de rådande antagandena. Utöver modellprestanda går vår studie in på detaljerna kring utvärderings-metoder som oftast används inom tidsperiods avvikelsedetektering. Vi avslöjar hur dessa oavsiktligt överdriver modellernas prestandapoäng och kan därmed leda till vilseledande slutsatser. Genom att identifiera och åtgärda dessa problem bidrar vår forskning till att erbjuda värdefulla insikter för forskare, praktiker och beslutsfattare inom området tidsperiods avvikelsedetektering, och uppmanar till en kritisk omvärdering av djupinlärningsmodellers roll och de metoder som används för att bedöma deras prestanda.