<div>Neuromorphic systems promise a novel alternative to the standard von-Neumann architectures that are computationally expensive for analyzing big data, and are not efficient for learning and inference. This novel generation of computing aims at ``mimicking" the human brain based on deploying neural networks on event-driven hardware architectures. A key bottleneck in designing such brain-inspired architectures is the complexity of co-optimizing the algorithm’s speed and accuracy along with the hardware’s performance and energy efficiency. This complexity stems from numerous intrinsic hyperparameters in both software and hardware that need to be optimized for an optimum design.</div><div><br></div><div>In this work, we present a versatile hierarchical pseudo agent-based multi-objective hyperparameter optimization approach for automatically tuning the hyperparameters of several training algorithms (such as traditional artificial neural networks (ANN), and evolutionary-based, binary, back-propagation-based, and conversion-based techniques in spiking neural networks (SNNs)) on digital and mixed-signal neural accelerators. By utilizing the proposed hyperparameter optimization approach we achieve improved performance over the previous state-of-the-art on those training algorithms and close some of the performance gaps that exist between SNNs and standard deep learning architectures.</div><div><br></div><div>We demonstrate >2% improvement in accuracy and more than 5X reduction in the training/inference time for a back-propagation-based SNN algorithm on the dynamic vision sensor (DVS) gesture dataset. In the case of ANN-SNN conversion-based techniques, we demonstrate 30% reduction in time-steps while surpassing the accuracy of state-of-the-art networks on an image classification dataset (CIFAR10) on a simpler and shallower architecture. Further, our analysis shows that in some cases even a seemingly minor change in hyperparameters may change the accuracy of these networks by 5‑6X. From the application perspective, we show that the optimum set of hyperparameters might drastically improve the performance (52% to 71% for Pole-Balance control application). In addition, we demonstrate resiliency of different input/output encoding, training neural network, or the underlying accelerator modules in a neuromorphic system to the changes of the hyperparameters.</div>

In this thesis we consider the application of Fenchel's duality theory and gradient-based methods for the training and hyperparameter optimization of Support Vector Machines. We show that the dualization of convex training problems is possible theoretically in a rather general formulation. For training problems following a special structure (for instance, standard training problems) we find that the resulting optimality conditions can be interpreted concretely. This approach immediately leads to the well-known notion of support vectors and a formulation of the Representer Theorem. The proposed theory is applied to several examples such that dual formulations of training problems and associated optimality conditions can be derived straightforwardly. Furthermore, we consider different formulations of the primal training problem which are equivalent under certain conditions. We also argue that the relation of the corresponding solutions to the solution of the dual training problem is not always intuitive. Based on the previous findings, we consider the application of customized optimization methods to the primal and dual training problems. A particular realization of Newton's method is derived which could be used to solve the primal training problem accurately. Moreover, we introduce a general convergence framework covering different types of decomposition methods for the solution of the dual training problem. In doing so, we are able to generalize well-known convergence results for the SMO method. Additionally, a discussion of the complexity of the SMO method and a motivation for a shrinking strategy reducing the computational effort is provided. In a last theoretical part, we consider the problem of hyperparameter optimization. We argue that this problem can be handled efficiently by means of gradient-based methods if the training problems are formulated appropriately. Finally, we evaluate the theoretical results concerning the training and hyperparameter optimization approaches practically by means of several example training problems.

The focus of this study was deep learning. A small, autonomous robot car was used for obstacle avoidance experiments. The robot car used a camera for taking images of its surroundings. A convolutional neural network used the images for obstacle detection. The available dataset of 31 022 images was trained with the Xception model. We compared two different implementations for making the robot car avoid obstacles. Mapping image classes to steering commands was used as a reference implementation. The main implementation of this study was to separate obstacle detection and steering logic in different modules. The former reached an obstacle avoidance ratio of 80 %, the latter reached 88 %. Different hyperparameters were looked at during training. We found that frozen layers and number of epochs were important to optimize. Weights were loaded from ImageNet before training. Frozen layers decided how many layers that were trainable after that. Training all layers (no frozen layers) was proven to work best. Number of epochs decided how many epochs a model trained. We found that it was important to train between 10-25 epochs. The best model used no frozen layers and trained for 21 epochs. It reached a test accuracy of 85.2 %.

Optimization is omnipresent in our world. Its numerous applications spread from industrial cases, such as logistics, construction management or production planning; to the private sphere, filled with problems of selecting daycare or vacation planning. In this thesis, we concentrate on expensive black-box optimization (EBBO) problems, a subset of optimization problems (OPs), which are characterized by an expensive cost of evaluating an objective function. Such OPs are reoccurring in various domains, being known as: hyperpameter optimization in machine learning, performance configuration optimization or parameter tuning in search-based software engineering, simulation optimization in operations research, meta-optimization or parameter tuning in the optimization domain itself. High diversity of domains introduces a plethora of solving approaches, which adhere to a similar structure and workflow, but differ in details. The software frameworks stemming from different areas possess only partially intersecting manageability points, i.e., lack manageability. In this thesis, we argue that the lack of manageability in EBBO is a major problem, which leads to underachieving optimization quality. The goal of this thesis is to study the role of manageability in EBBO and to investigate whether improving the manageability of EBBO frameworks increases optimization quality. To reach this goal, we appeal to software product line engineering (SPLE), a methodology for developing highly-manageable software systems. Based on the foundations of SPLE, we introduce a novel framework for EBBO called BRISE. It offers: 1) a loosely-coupled software architecture, separating concerns of the experiment designer and the developer of EBBO strategies; 2) a full coverage of all EBBO problem types; and 3) a context-aware variability model, which captures the experiment-designer-defined OP with the content model; and manageability points including their variants and constraints with the cardinality-based feature model. High manageability of the introduced BRISE framework enables us: 1) to extend the framework with novel efficient strategies, such as adaptive repetition management; and 2) to introduce novel EBBO mechanisms, such as multi-objective compositional surrogate modeling, dynamic sampling and hierarchical surrogate modeling. The evaluation of the novel approaches with a set of case studies, including: the WFG benchmark for multi-objective optimization, combined selection and parameter control of meta-heuristics, and energy optimization; demonstrated their superiority over the state-of-the-art competitors. Thus, it supports the research hypothesis of this thesis: Improving manageability of an EBBO framework enables to increase optimization quality.

No description available.

Environment perception is crucial to autonomous driving. Panoptic segmentation and objects tracking are two challenging tasks, and the combination of both, namely 4D panoptic segmentation draws researchers’ attention recently. In this work, we implement 4D panoptic LiDAR segmentation (4D-PLS) on Volvo datasets and provide a pipeline of data preparation, model building and model optimization. The main contributions of this work include: (1) building the Volvo datasets; (2) adopting an 4D-PLS model improved by Hyperparameter Optimization (HPO). We annotate point cloud data collected from Volvo CE, and take a supervised learning approach by employing a Deep Neural Network (DNN) to extract features from point cloud data. On the basis of the 4D-PLS model, we employ Bayesian Optimization to find the best hyperparameters for our data, and improve the model performance within a small training budget. / Miljöuppfattning är avgörande för autonom körning. Panoptisk segmentering och objektspårning är två utmanande uppgifter, och kombinationen av båda, nämligen 4D panoptisk segmentering, har nyligen uppmärksammat forskarna. I detta arbete implementerar vi 4D-PLS på Volvos datauppsättningar och tillhandahåller en pipeline av dataförberedelse, modellbyggande och modelloptimering. De huvudsakliga bidragen från detta arbete inkluderar: (1) bygga upp Volvos datauppsättningar; (2) anta en 4D-PLS-modell förbättrad av HPO. Vi kommenterar punktmolndata som samlats in från Volvo CE och använder ett övervakat lärande genom att använda en DNN för att extrahera funktioner från punktmolnsdata. På basis av 4D-PLS-modellen använder vi Bayesian Optimization för att hitta de bästa hyperparametrarna för vår data och förbättra modellens prestanda inom en liten utbildningsbudget.

No description available.

Artificial neural networks are some of the most powerful machine learning models and have gained interest in the telecommunications domain as well as other fields and applications due to their strong performance and flexibility. Creating these models typically requires manually choosing their architecture along with other hyperparameters that are crucial for their performance. Neural Architecture Search (NAS) seeks to automate architecture choice and has gained increasing interest in recent years. In this thesis, we propose a new NAS method based on differential architecture search (DARTS) to find architectures of fully-connected feed forward networks on tabular datasets. We train a gating mechanism on a validation dataset and compare four candidate gate functions as a tool to determine the number of hidden units per hidden layer in our neural networks for different tasks. Our findings show that our new method can reliably find architectures that are more compact and outperform manually chosen architectures. Interestingly, we also found that extracting weights learned during the search process could generate models that achieve significantly higher and more stable performance than identical architectures retrained from scratch. Our method achieved equal in performance to that of another NAS-method, while only requiring half an hour of training compared to 280 hours. The trained models also demonstrated a competitive performance when benchmarked to other state-of-the-art machine learning models. The primary benefit of our method, stems from the extraction and fine-tuning of certain weights. Our results indicate that improvements from extracted weights could relate to the lottery ticket hypothesis of neural networks, which invites further study for a fuller understanding.

The object of the study is automated machine learning packages for forecasting time series. The subject of the study is hyperparameter optimization algorithms used in a number of selected packages. The purpose of the work is to compare automated machine learning packages in the context of the problem of forecasting time series and to identify the features of approaches to optimizing hyperparameters used in each package. Research methods: conducting a theoretical analysis of the available literature on the topic of the study, studying the documentation for the automatic machine learning packages involved in the work, conducting experiments, comparing and evaluating the forecasting results using the constructed pipelines, generalizing and interpreting the results. Results of the work: features in the implementation of hyperparameter optimization algorithms for the libraries under consideration are highlighted. / Объект исследования — пакеты автоматизированного машинного обучения для прогнозирования временных рядов. Предмет исследования — алгоритмы оптимизации гиперпараметров применяемые в ряде выбранных пакетов. Цель работы — проведение сравнения пакетов автоматизированного машинного обучения в контексте задачи прогнозирования временных рядов и выявление особенностей подходов к оптимизации гиперпараметров используемых в каждом пакете. Методы исследования: проведение теоретического анализа доступной литературы по теме исследования, изучение документации к задействованным в работе пакетам автоматического машинного обучения, проведение экспериментов, сравнение и оценка результатов прогнозирования с помощью построенных конвейеров, обобщение и интерпретация полученных результатов. Результаты работы: выделены особенности в реализации алгоритмов оптимизации гиперпараметров для рассматриваемых библиотек.

Notwithstanding the recent successes of deep convolutional neural networks for classification tasks, they are sensitive to the selection of their hyperparameters, which impose an exponentially large search space on modern convolutional models. Traditional hyperparameter selection methods include manual, grid, or random search, but these require expert knowledge or are computationally burdensome. Divergently, Bayesian optimization and evolutionary inspired techniques have surfaced as viable alternatives to the hyperparameter problem. Thus, an alternative hybrid approach that combines the advantages of these techniques is proposed. Specifically, the search space is partitioned into discrete-architectural, and continuous and categorical hyperparameter subspaces, which are respectively traversed by a stochastic genetic search, followed by a genetic-Bayesian search. Simulations on a prominent image classification task reveal that the proposed method results in an overall classification accuracy improvement of 0.87% over unoptimized baselines, and a greater than 97% reduction in computational costs compared to a commonly employed brute force approach. / Electrical and Mining Engineering / M. Tech. (Electrical Engineering)