abstract: The rapid advancement of Deep Neural Networks (DNNs), computing, and sensing technology has enabled many new applications, such as the self-driving vehicle, the surveillance drone, and the robotic system. Compared to conventional edge devices (e.g. cell phone or smart home devices), these emerging devices are required to deal with much more complicated and dynamic situations in real-time with bounded computation resources. However, there are several challenges, including but not limited to efficiency, real-time adaptation, model stability, and automation of architecture design. To tackle the challenges mentioned above, model plasticity and stability are leveraged to achieve efficient and online deep learning, especially in the scenario of learning streaming data at the edge: First, a dynamic training scheme named Continuous Growth and Pruning (CGaP) is proposed to compress the DNNs through growing important parameters and pruning unimportant ones, achieving up to 98.1% reduction in the number of parameters. Second, this dissertation presents Progressive Segmented Training (PST), which targets catastrophic forgetting problems in continual learning through importance sampling, model segmentation, and memory-assisted balancing. PST achieves state-of-the-art accuracy with 1.5X FLOPs reduction in the complete inference path. Third, to facilitate online learning in real applications, acquisitive learning (AL) is further proposed to emphasize both knowledge inheritance and acquisition: the majority of the knowledge is first pre-trained in the inherited model and then adapted to acquire new knowledge. The inherited model's stability is monitored by noise injection and the landscape of the loss function, while the acquisition is realized by importance sampling and model segmentation. Compared to a conventional scheme, AL reduces accuracy drop by >10X on CIFAR-100 dataset, with 5X reduction in latency per training image and 150X reduction in training FLOPs. Finally, this dissertation presents evolutionary neural architecture search in light of model stability (ENAS-S). ENAS-S uses a novel fitness score, which addresses not only the accuracy but also the model stability, to search for an optimal inherited model for the application of continual learning. ENAS-S outperforms hand-designed DNNs when learning from a data stream at the edge. In summary, in this dissertation, several algorithms exploiting model plasticity and model stability are presented to improve the efficiency and accuracy of deep neural networks, especially for the scenario of continual learning. / Dissertation/Thesis / Doctoral Dissertation Electrical Engineering 2020

付記する学位プログラム名: 京都大学卓越大学院プログラム「先端光・電子デバイス創成学」 / 京都大学 / 新制・課程博士 / 博士(情報学) / 甲第23325号 / 情博第761号 / 新制||情||130(附属図書館) / 京都大学大学院情報学研究科通信情報システム専攻 / (主査)教授 小野寺 秀俊, 教授 大木 英司, 教授 佐藤 高史 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM

Deep Neural Networks are state-of the art algorithms for various image and natural language processing tasks. These networks are composed of billions of operations working on an input to produce the desired result. Along with this computational complexity, these workloads are also massively parallel in nature. These inherent properties make deep neural networks an excellent target for custom acceleration. The main challenge faced by such accelerators is achieving a compromise between power consumption, software programmability, and resource utilization for the varied compute and data access patterns presented by DNN workloads. In this work, I present Inference Engine, a scalable and efficient DNN accelerator designed to be agnostic to the type of DNN workload. Inference Engine was designed to provide near peak hardware resource utilization, minimize data transfer, and offer a programmer friendly instruction set. Inference engine scales at the level of individually programmable clusters, each of which contains several hundred compute resources. It provides an instruction set designed to exploit parallelism within the workload while also allowing freedom for compiler based exploration of data access patterns.

In recent years, there has been a significant push to implement Deep Neural Networks (DNNs) on edge devices, which requires power and hardware efficient circuits to carry out the intensive matrix-vector multiplication (MVM) operations. This work presents hardware efficient MVM implementation techniques using bit-serial arithmetic and a novel MSB first computation circuit. The proposed designs take advantage of the pre-trained network weight parameters, which are already known in the design stage. Thus, the partial computation results can be pre-computed and stored into look-up tables. Then the MVM results can be computed in a bit-serial manner without using multipliers. The proposed novel circuit implementation for convolution filters and rectified linear activation function used in deep neural networks conducts computation in an MSB-first bit-serial manner. It can predict earlier if the outcomes of filter computations will be negative and subsequently terminate the remaining computations to save power. The benefits of using the proposed MVM implementations techniques are demonstrated by comparing the proposed design with conventional implementation. The proposed circuit is implemented on an FPGA. It shows significant power and performance improvements compared to the conventional designs implemented on the same FPGA.

Networking and distributed computing system have provided computational resources for machine learning (ML) application for a long time. Network system itself also can benefit from ML technologies. For example high performance packet classification is a key component to support scalable network applications like firewalls, intrusion detection, and differentiated services. With ever increasing demand in the line rate for core networks, a great challenge is to use hand-tuned heuristic approaches to design a scalable and high performance packet classification solution. By exploiting the sparsity present in a ruleset, in this thesis an algorithm is proposed to use few effective bits (EBs) to extract a large number of candidate rules with just a few number of memory access. These effective bits are learned with deep reinforcement learning and they are used to create a bitmap to filter out the majority of rules which do not need to be fully matched to improve the online system performance. Utilizing reinforcement learning allows the proposed solution to be learning based rather than heuristic based algorithms. So proposed learning-based selection method is independent of the ruleset, which can be applied to different rulesets without relying on the heuristics. Proposed multibit tries classification engine outperforms lookup time both in worst and average case by 55% and reduce memory footprint, compared to traditional decision tree without EBs. Furthermore, many field packet classification are required for openFlow supported switches. With the proliferation of fields in the packet header, a traditional 5-field classification technique isn’t applicable for an efficient classification engine for those openFlow supported switches. Although the algorithmic insights obtained from 5-field classification techniques could still be applied for many field classification engine. To decompose given fields of a ruleset, different grouping metrics like standard deviation of individual fields and a novel metric called Diversity Index (DI) is considered for such many field scenarios. A detailed discussion and evaluation of how to decompose rule fields/dimension into subgroup, how a decision tree construction can be considered as reinforcement learning problem, and how to encode state and action space, reward calculation to effectively build trees for each subgroup with a global optimization objective is introduced in this work. Finally, to identify benign or malicious heterogeneous type of traffic present in a modern home network, a deep neural network based approach is introduced. A split architecture of such traffic classifier, in application of home network intrusion detection system consists of multiple machine learning (ML) models. These models trained on two separate dataset for heterogeneous traffic types. An analysis of run-time implementation performance of the proposed IDS models is also discussed.

An initiative to standardize the nomenclature in Sweden started in 2016 along with the creation of the local database Medical Information Quality Archive (MIQA) and a national radiotherapy register on Information Network for CAncercare (INCA). A problem of identifying the clinical tumor volume (CTV) structures and prescribed dose arose when the consecutive number, which is added to the CTV-name, was made inconsistently in MIQA and INCA. Deep neural networks (DNN) were promising tools to solve the multi-label classification task of the CTV to enable automatic labeling in the database. Prostate cancer patients that often have more than one type of organ in the same CTV structure were chosen for proof of concept. The DNN used supervised training in a 2D fashion where the radiation therapy (RT) structures along with the CT image were fed, slice by slice, to AlexNet and VGGNet to label the CTV structures in the local database system MIQA and INCA. The study also includes three methods to classify a final label for the CTV structure since the model makes the predictions on each slice. The three methods were maximum method by taking the maximum prediction for each class, minimum method by taking the minimum prediction for each class and occurrence method. The occurrence method chooses the maximum prediction if the network has predicted the class over 0.5 at least two times and the minimum prediction if not. The DNN and volume classification methods performed well where the maximum and occurrence method performed the best and can be used to interpret RT volumes in MIQA and INCA for prostate cancer patients. This novel study gives promising results for the future development of deep neural networks classifying RT structures for more than one type of cancer patient. / Ett initiativ för att standardisera nomenklaturen i Sverige startade 2016 tillsammans med skapandet av den lokala databasen Medical Information Quality Archive (MIQA) och ett nationellt radioterapikvalitetsregister på plattformen Information Network for CAncercare (INCA). Ett problem med att identifiera kliniska tumörvolymstrukturer (CTV-strukturer) och ordinerad dos uppstod när de på varandra följande siffrorna, som adderas till CTV-namnet för att skilja de olika CTV:erna från varandra, gjordes inkonsekvent i MIQA och INCA. Djupa neurala nätverk (DNN) är lovande verktyg för att lösa klassificeringen av CTV för att möjliggöra automatisk annotering för multippla etiketter i databasen. Prostatacancerpatienter vars radioterapistrukturer (RT-strukturer) ofta innehåller fler än ett organ användes därför för att bevisa konceptet för fleretikettsklassificering. DNN:et använde övervakad inlärning av 2D-bilder där RT-strukturerna tillsammans med CT-bilderna matades in, snitt för snitt, till AlexNet och VGGNet för att namnge CTV-strukturerna i det lokala databassystemet MIQA och sedan i INCA. Studien inkluderar även tre metoder för en slutlig strukturetikett eftersom modellen gör sina förutsägelser på varje snitt. Metoderna var maximum där den högsta förutsägelsen noteras för varje klass, minimum där den lägsta förutsägelsen noteras för varje klass och förekomst där den högsta förutsägelsen noteras om klassen har fått minst två förutsägelser över 0.5 annars noteras den lägsta förutsägelsen. DNN:en och volymetikettmetoderna gav bra resultat där maximum- och förekomstmetoden gav bäst resultat och kan användas för att tolka RT-volymer i MIQA och INCA för prostatacancerpatienter. Denna nya studie ger lovande resultat för framtida utveckling av djupa neurala nätverk som klassificerar strukturer från mer än en typ av cancerpatient.

Malignant melanoma (MM) is a type of skin cancer that is associated with a very poor prognosis and can often lead to death. Early detection is crucial in order to administer the right treatment successfully but currently requires the expertise of a dermatologist. In the past years, studies have shown that automatic detection of MM is possible through computer vision and machine learning methods. Skin lesion segmentation and classification are the key methods in supporting automatic detection of different skin lesions. Compared with traditional computer vision as well as other machine learning methods, deep neural networks currently show the greatest promise both in segmentation and classification. In our work, we have implemented several deep neural networks to achieve the goals of skin lesion segmentation and classification. We have also applied different training schemes. Our best segmentation model achieves pixel-wise accuracy of \textbf{0.940}, Dice index of \textbf{0.867} and Jaccard index of \textbf{0.765} on the ISIC 2017 challenge dataset. This surpassed the official state of the art model whose pixel-wise accuracy was 0.934, Dice index 0.849 and Jaccard Index 0.765. We have also trained a segmentation model with the help of adversarial loss which improved the baseline model slightly. Our experiments with several neural network models for skin lesion classification achieved varying results. We also combined both segmentation and classification in one pipeline meaning that we were able to train the most promising classification model on pre-segmented images. This resulted in improved classification performance. The binary (melanoma or not) classification from this single model trained without extra data and clinical information reaches an area under the curve (AUC) of 0.684 on the official ISIC test dataset. Our results suggest that automatic detection of skin cancers through image analysis shows significant promise in early detection of malignant melanoma.

The present study focuses on developing an efficient and stable unmanned aerial system traffic management (UTM) system that utilizes a data-driven target tracking method and a distributed path planning algorithm for multiple Unmanned Aerial Vehicle (UAV) operations with local dynamic networks, which can provide flexible scalability, enabling autonomous operation of a large number of UAVs in dynamically changing environment. Traditional dynamic motion-based target tracking methods often encounter limitations due to their reliance on a finite number of dynamic motion models. To address this, data-driven target tracking methods were developed based on the statistical model of the Gaussian mixture model (GMM) and deep neural networks of long-short term memory (LSTM) model, to estimate instant and future states of UAV for local path planning problems. The estimation accuracy of the data-driven target tracking methods were analyzed and compared with dynamic model-based target tracking methods. A hybrid dynamic path planning algorithm was proposed, which selectively employs grid-free and -based path search methods depending on the spatio-temporal characteristics of the environments. In static environment, the artificial potential field (APF) method was utilized, while the $A^*$ algorithm was applied in the dynamic state environment. Furthermore, the data-driven target tracking method was integrated with the hybrid path planning algorithm to enhance deconfliction. To ensure smooth trajectories, a minimum snap trajectory method was applied to the planned paths, enabling controller tracking that remains dynamically feasible throughout the entire operation of UAVs. The methods were validated in the Software-in-the-loop (SITL) demonstration with the simple PID controller of the UAVs implemented in the software program. / Ph.D. / This dissertation focuses on developing data-driven models for tracking and path planning of Unmanned Aerial Vehicle (UAV) in dynamic environments with multiple operations. The goal is to improve the accuracy and efficiency of Unmanned Aircraft System traffic management (UTM) under such conditions. The data-driven models are based on Gaussian mixture model (GMM) and long-short term memory (LSTM) and are used to estimate the instant and consecutive future states of UAV for local planning problems. These models are compared to traditional target tracking models, which use dynamic motion models like constant velocity or acceleration. A hybrid dynamic path planning approach is also proposed to solve dynamic path planning problems for multiple UAV operations at an efficient computation cost. The algorithm selectively employs a path planning method between grid-free and grid-based methods depending on the characteristics of the environment. In static state conditions, the system uses the artificial potential field method (APF). When the environment is time-variant, local path planning problems are solved by activating the $A^*$ algorithm. Also, the planned paths are refined by minimum snap trajectory to ensure that the path is dynamically feasible throughout a full operation of the UAV along with controller tracking. The methods were validated in the Software-in-the-loop (SITL) demonstration with the simple PID controller of the UAVs implemented in the software program.

Recently, lithium-ion batteries (LIBs) have achieved wide acceptance for various energy storage applications, such as electric vehicles (EVs) and smart grids. As a vital component in EVs, the performance of lithium-ion batteries in the last few decades has made significant progress. The development of a robust battery management system (BMS) has become a necessity to ensure the reliability and safety of battery packs. In addition, state of charge (SOC) estimation and thermal models with high-fidelity are essential to ensure efficient BMS performance. The SOC of a LIB is an essential factor that should be reported to the vehicle’s electronic control unit and the driver. Inaccurate reported SOC impacts the reliability and safety of the lithium-ion battery packs (LIBP) and the vehicle. Different algorithms are used to estimate the SOC of a LIBP, including measurement-based, adaptive filters and observers, and data-driven; however, there is a gap in feasibility studies of running these algorithms for multi-cell LIBP on BMS microprocessors. On the other hand, temperature sensors are utilized to monitor the temperature of the cells in LIBPs. Using a temperature sensor for every cell is often impractical due to cost and wiring complexity. Robust temperature estimation models can replace physical sensors and help the fault detection algorithms by providing a redundant monitoring system. In this thesis, an accurate SOC estimation and thermal modeling for lithium-ion batteries (LIBs) are presented using deep neural networks (DNNs). Firstly, two DNN-based SOC estimation algorithms, including a feedforward neural network (FNN) enhanced with external filters and a recurrent neural network with a long short-term memory layer (LSTM), are developed and benchmarked versus an extended Kalman filter (EKF) and EKF with recursive least squares filter (EKF-RLS) SOC estimation algorithms. The execution time of EKF, EKF-RLS, FNN, and LSTM SOC estimation algorithms with similar accuracy was found to be 0.24 ms, 0.25 ms, 0.14 ms, and 0.71 ms, respectively. The DNN SOC estimation algorithms were also demonstrated to have lower RAM use than the EKFs, with less than 1 kB RAM required to run one estimator. The proposed FNN and LSTM models are also used to predict the surface temperature of different lithium-ion cells. These DNN models are shown to be capable of estimating temperature with less than 2 ⁰C root mean square error for challenging low ambient temperature drive cycles and just 0.3 ⁰C for 4C rate fast charging conditions. In addition, a DNN model which is trained to estimate the temperature of a new battery cell, is found to still have a very low error of just 0.8 ⁰C when tested on an aged cell. Finally, an integrated physics, and neural network-based battery pack thermal model (LP+FNN) is developed and used to detect and identify different thermal faults of a LIBP. The proposed fault detection and identification method is validated using various thermal faults, including fan system failure, airflow lower and higher than setpoint, airflow blockage of submodule and temperature sensor reading faults. The proposed method is able to detect different cooling system faults within 10 to 35 minutes after fault occurrence. In addition, the proposed method demonstrated being capable of detecting temperature sensor reading offset and scale faults of ±3 ⁰C and ±0.15% or more, respectively with 100% accuracy. / Thesis / Doctor of Philosophy (PhD)

Artificial Neural Networks make up the core of mostMachine Learning algorithms. In the past decade Machine learninghave successfully taken on fields such as image recognition,Data analytics and medical technologies. As the area of usebecome less prone to mistakes, it raises the responsibility lookinto the black box of code and understand it to a deeper level. Inthis project, I built a Deep Neural Network from scratch, withouthigh level libraries, and trained it for a supervised classificationtask. The finished algorithm is flexible and can be adapted toany classification problem. The training method is based onBackpropagation and Gradient Descent. At last, the algorithmwas trained on the Modified National Institute of Standardsand Technology (MNIST) database, and performed with a 77%prediction acccuracy. There are a few optimization methods yetto be tested to further increase the performance. / Artificiella neurala nätverk utgör kärnan i de flesta maskininlärningsalgoritmer idag. Under det senaste decenniet har maskininlärning framgångsrikt tagit an områden som bildigenkänning, dataanalys och medicinsk teknik. När användningsområdena blir mindre benägna till misstag, ökar ansvaret av att titta under huven och förstå den djupare nivåkoden. I denna studie var syftet att bygga ett djupt neuralt nätverk från grunden, utan högnivåbibliotek, och träna det för en övervakad klassificeringsuppgift. Den färdiga algoritmen är flexibel och kan designas för flera klassificeringsproblem. Nätverkets träningsmetod är baserad på Backpropagation och Gradient Descent. Valideringsdatan kunde till slut köras med 77% korrekt noggrannhet, och det finns finns ytterligare optimeringsmetoder att testa för att höja prestationen. / Kandidatexjobb i elektroteknik 2022, KTH, Stockholm