Global ETD Search
Search theses and dissertations gathered from participating repositories worldwide. Every result links back to the library that holds it. No account is needed.
Results
Showing 1 to 20 of 246 for “"Knowledge Distillation"”.
-
Continual Domain Adaptation through Knowledge Distillation
… called Continual Domain Adaptation through Knowledge Distillation (CAKE), that uses knowledge distillation (KD) to transfer to a CNN the more complex Vision Transformer's knowledge. By doing so, CNN-based adaptation obtains a similar performance to transformer-based adaptation while reducing …
-
Enhanced knowledge distillation by auxiliary classifiers
… proposed approaches to mitigate this limitation, knowledge distillation has gained much attention due to its generalizability and simplicity in implementation. This thesis introduces the enhanced knowledge distillation (EKD), a simple yet effective approach to outperform the canonical knowledge …
-
DEEPFAKE DETECTION GENERALIZATION VIA KNOWLEDGE DISTILLATION
… to address these problems, through the use of knowledge distillation (KD) using heterogeneous teachers. In a typical class setting, a student learns from several teachers, each an expert in their domain. This process is analogous to the use of KD with heterogeneous teachers. A contribution of …
-
KNOWLEDGE DISTILLATION FOR MOLECULAR PROPERTY PREDICTION
Knowledge distillation (KD) is a powerful model compression technique that transfers knowledge from complex teacher models to compact student models, reducing computational costs while preserving predictive accuracy. This study investigated KD's efficacy in molecular property prediction across …
-
StarGAN-v2 compression using knowledge distillation
… on the task of compressing StarGAN-v2 using knowledge distillation. Using depthwise separable convolutional layers and reduced channels for intermediate layers, we develop efficient architectures for different StarGAN-v2 modules. In a GAN mini-max optimization environment, the efficient …
-
Compressing Deep Neural Networks via Knowledge Distillation
There has been a continuous evolution in deep neural network architectures since Alex Krizhevsky proposed AlexNet in 2012. Part of this has been due to increased complexity of the data and easier availability of datasets and part of it has been due to increased complexity of applications. These two …
-
Minifying Deep Denoising Networks with Knowledge Distillation
… investigating and measuring the effectiveness of knowledge distillation in creating a smaller and faster model. In using knowledge distillation, we extract the capabilities of a larger, more powerful teacher model into a smaller, more efficient student model, optimizing its suitability for …
-
Multi-Teacher Knowledge Distillation Using Teacher's Domain Expertise
… with low computing power and storage capacity. Knowledge Distillation solves this problem by distilling knowledge into a smaller BERT model while retaining much of the teacher’s accuracy in student. A teacher expert in predicting one class should be chosen, by student, over others for that class …
-
Improving Deep Neural Network Training with Knowledge Distillation
Knowledge distillation, as a popular compression technique, has been widely used to reduce deep neural network (DNN) size for a variety of applications. However, in recent years, some research had found its potential for improving deep neural network performance. This dissertation focuses on …
-
Training Binary Deep Neural Networks Using Knowledge Distillation
… of a binary network can be improved by using knowledge distillation. Three different knowledge distillation methods were tested for various network types. Additionally, different architectures of a residual block in ResNet were suggested and tested. Test on CIFAR10 showed an 1.5% increase in …
-
Knowledge Distillation as a Solution for Domain Transfer
… in fact verification tasks. We present a data distillation technique for delexicalization, which we then combine with a model distillation method to prevent aggressive data distillation. We show that by using our solution, not only does the performance of an ex- isting state-of-the-art model …
-
Efficient transformer-based panoptic segmentation via knowledge distillation
Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-09-01 without embargo terms
-
Knowledge Distillation Circumvents Nonlinearity for Optical Convolutional Neural Networks
… (SCLC) network architecture and develop a Knowledge Distillation (KD) approach to circumvent the need for nonlinearity and successfully train such networks. While the KD approach is known in machine learning as an effective process for network pruning, we adapt the approach to transfer the …
-
Knowledge Distillation and Its Application to Network Traffic Classification
Knowledge distillation improves the performance of a low-complexity student model with the help of a more powerful teacher. In this dissertation, we study knowledge distillation and its application to network traffic. We posit that knowledge distillation is an essential ingredient of traffic …
-
Unsupervised Contrastive Representation Learning for Knowledge Distillation and Clustering
… learning methods and apply them to knowledge distillation and clustering. The knowledge distillation transfers knowledge from high-capacity teachers to small student models and then improves the performance of students. And the representational knowledge distillation methods try to …
-
Knowledge Distillation in Federated Magnetic Resonance Image (MRI) Reconstruction
… and data privacy issues. On the other hand, knowledge distillation is a student-teacher-based model compression technique that can address communication and data privacy issues. In this thesis, we explore the inclusion of knowledge distillation in a Federated MRI reconstruction framework to …
-
DistilBERT-EL-KLD: Knowledge Distillation and Greek Language Modeling
Σκοπός της εν λόγω διπλωματικής εργασίας είναι η ανάπτυξη, εκπαίδευση και αξιολόγηση ενός μοντέλου DistilBERT, εξειδικευμένου αποκλειστικά στην Ελληνική γλώσσα. Μετά από εμβριθή ανασκόπηση του θεωρητικού υποβάθρου (κλασική μηχανική μάθηση, βαθιά μάθηση και νευρωνικά δίκτυα) με σκοπό να καταστεί …
-
Domain-Aware Reasoning with Lightweight Models via Knowledge Distillation
… This thesis investigates the efficacy of knowledge distillation for transferring advanced reasoning from a large teacher model (DeepSeek-R1) to a lightweight student model (Meta-Llama-3.1-8B-Instruct) within a Retrieval-Augmented Generation (RAG) framework for cybersecurity intelligence. …
-
Temperature-centric investigation of speculative decoding with knowledge distillation
… speculative decoding’s efficacy. Beginning with knowledge distillation (KD), we first highlight the challenge of decoding at higher temperatures, and demonstrate KD in a consistent temperature setting could be a remedy. We also investigate the effects of out-of-domain testing sets with …
Page 1 of 13