Democritus University of Thrace (DUTH)
Learning from positive and unlabeled data for imbalanced datasets: theory and real-world applications
Abstract
dc:descriptionMachine Learning (ML) has revolutionized data analysis and decision-making by allow- ing systems to learn and improve from experience, in contrast to traditional program- ming’s rules-based approach. ML algorithms analyze large datasets to uncover patterns and make predictions, improving with more data. This adaptive learning process en- ables applications in image and speech recognition, natural language processing, and predictive analytics, handling tasks that are impractical for humans.ML is a subset of artificial intelligence (AI) focused on learning from data. It in- cludes supervised learning, which uses labeled data to predict outputs for new data, and unsupervised learning, which identifies patterns in unlabeled data. Semi-supervised learning uses a mix of labeled and unlabeled data, while reinforcement learning in- volves learning through feedback in dynamic environments. Data science, a broader field, encompasses data collection, preprocessing, analysis, and visualization, with ML providing tools for building predictive models. The real-world application of ML faces challenges such as messy, incomplete data and the need for continuous model maintenance. Academic datasets are controlled and curated, while real-world data requires sophisticated preprocessing. ML models must integrate into larger systems, considering user experience, ethics, and compliance. Scalability is crucial for handling large data volumes and adapting to changing conditions, especially in high-stakes industries like healthcare and finance. The era of big data presents opportunities and challenges. Abundant data sources generate vast information, but labeling this data is expensive and time-consuming. Semi-supervised and unsupervised learning techniques aim to utilize unlabeled data effectively. Data understanding is critical for robust models, and addressing dataset imbalance ensures fair and effective systems. Positive and Unlabeled (PU) learning addresses data scarcity and imbalance by using small sets of positive examples and large pools of unlabeled data. Common in fields like fraud detection and medical diagnosis, PU learning leverages limited positive examples to build accurate predictive models. This thesis explores PU learning’s theoretical foundations and practical applications, proposing robust methodologies for real-world datasets. This PhD thesis explores the application and advancement of Positive-Unlabeled (PU) learning, a novel approach in machine learning that addresses the challenges posed by imbalanced and partially labeled datasets. Through five comprehensive chapters, the thesis systematically delves into the theoretical foundations, benchmarking methods, bias mitigation strategies, and real-world applications of PU learning, ultimately offering significant contributions to both academia and industry. The thesis is structured as follows: Chapter 1 introduces the core concept of learning from data, where machine learning (ML) is framed as a method of generalizing from specific instances to new, unseen data. The chapter compares ML with related fields such as statistics, data mining, and artificial intelligence (AI), emphasizing the unique challenges posed by real-world data, which is often messy, incomplete, and noisy. A critical issue discussed is data imbalance, where one class is significantly underrepresented. PU learning is presented as a promising solution to this challenge, leveraging a small set of known positive examples and a large pool of unlabeled data. The chapter sets the stage for the rest of the thesis by outlining the structure and goals of the research. Chapter 2 focuses on the benchmarking of PU learning models, emphasizing the need for specialized evaluation metrics and validation strategies due to the absence of labeled negative examples. Traditional metrics like accuracy and error rate are deemed inadequate for imbalanced datasets, and alternatives such as the F-score and Matthews Correlation Coefficient (MCC) are recommended. The chapter also provides an overview of various datasets used for benchmarking, ranging from general image recognition to specialized industrial applications. This chapter establishes the foundation for rigorous evaluation, which is essential for developing robust PU learning models. Chapter 3 addresses the inevitable bias in dataset creation and proposes methods to mitigate it. The chapter discusses how dataset bias can lead to poor generalization on unseen data, a common issue in supervised learning. To combat this, a novel data collection pipeline is introduced, designed to create more representative datasets by filtering out irrelevant and duplicate content. The chapter also explores the application of deep learning techniques and the Burkhard-Keller Tree (BK-tree) indexing in improving dataset quality and model performance. The methods proposed here significantly reduce the impact of dataset bias, enhancing the generalizability of AI models. Chapter 4 presents Dense-PU, an innovative method that improves the performance of PU learning by generating new samples through boundary-aware sampling and anomaly detection. This approach transforms the PU learning challenge into a binary classification problem, enabling more effective model training. Dense-PU was tested on benchmark datasets such as CIFAR-10 and Fashion-MNIST, where it demonstrated state-of-the-art performance. The chapter highlights the method’s applicability in real- world scenarios, particularly those involving imbalanced datasets, and suggests future research directions to further enhance PU learning. Chapter 5 explores the real-world applications of PU learning across various do- mains, emphasizing its effectiveness in managing imbalanced and partially labeled datasets. The chapter presents case studies, including the identification of accident black spots in Greek road networks and defect detection in industrial settings. These studies showcase the adaptability and robustness of PU learning, demonstrating its superiority over traditional supervised learning methods in handling real-world data challenges. The chapter concludes by outlining future research opportunities to optimize PU learning further. In summary, this thesis makes significant contributions to the field of machine learning by advancing the theoretical understanding of PU learning, demonstrating its practical applications, and providing a roadmap for future research. The findings offer a solid foundation for the continued development and application of PU learning in diverse domains, particularly where data imbalance and limited labeled data are prevalent. The contributions are also listed bellow in three categories: (a) theoretical contributions, (b) practical contributions, and (c) other contributions to the broader field of ML. Theoretical contributions of this dissertation:1. Introduction of the Dense-PU Method: This study presents a novel method for PU learning called Dense-PU [1]. The method innovatively uses boundary-aware sampling and anomaly detection to transform the PU learning problem into a conventional binary classification task. By interpolating between latent encodings of positive-labeled data, Dense-PU effectively approximates the boundary of the positive class, reducing reliance on negative data, which is often unavailable in PU learning scenarios.2. Identification and Mitigation of Dataset Bias: This study provides a thorough analysis of dataset bias, a significant challenge in machine learning, particularly in the generalization of models to real-world data [2]. It introduces innovative methods to detect and mitigate bias during the dataset creation process, ensuring that models trained on these datasets perform more effectively across diverse and unseen environments.3. Integration of Anomaly-Informed Training during DNN training: One of the key contributions is the development of an innovative method that enhances weakly supervised learning through anomaly-informed weighted training [3]. This approach refines the defect detection process in industrial applications, leading to substantial improvements in classification metrics across diverse datasets.4. Use of Deep Convolutional Generative Adversarial Networks (DCGAN) for approximating an unseen negative class: The study presents the integration of DCGANs for data approximation in the context of PU learning, particularly in the detection of defects in photovoltaic cells. This method achieves performance levels on par with fully supervised learning while requiring only a small subset of labeled data, highlighting the efficiency of PU learning in data-scarce environments [4]. Practical contributions with real-world applications:1. Adaptability to Real-World Scenarios: This study pioneers the application of PU learning to complex, real-world problems such as the identification of accident black spots within Greek road networks [5, 6] and defect detection in industrial settings [3, 4, 7]. By doing so, it demonstrates the versatility and effectiveness of PU learning in contexts where traditional supervised learning methods struggle, particularly in managing imbalanced datasets.2. Computational Efficiency and Simplicity: Despite incorporating advanced techniques like interpolation and anomaly detection, Dense-PU remains com- putationally efficient and straightforward to implement. This balance between complexity and practicality makes the method accessible for real-world applications without requiring extensive computational resources [3].3. Development of a Novel Dataset Construction Pipeline: A novel pipeline for constructing datasets from web-acquired data is introduced. This pipeline includes multiple stages, such as irrelevant content detection, duplicate image removal, and ranking of representative samples, all aimed at reducing noise and bias in the dataset [2, 8]. The pipeline is designed to be scalable, efficient, and applicable to a wide range of domains.4. Adaptation of BK-Tree for Efficient Deduplication and Place Recognition: The study adapts the Burkhard-Keller Tree (BK-tree) structure for two purposes: (a) efficient duplicate image detection during dataset construction [2, 8] and (b) enhancing visual loop-closure detection in Simultaneous Localization and Mapping (SLAM) systems. This adaptation significantly reduces computational complexity and improves accuracy, making it a valuable tool for both dataset construction and autonomous navigation systems [9].Other contributions related to the broader field of ML:1. State-of-the-Art Performance on Benchmark Datasets: The effectiveness of Dense-PU is demonstrated through extensive experimentation on widely recognized benchmark datasets such as CIFAR-10 and Fashion-MNIST. The method consistently outperforms existing state-of-the-art approaches in terms of F1-score, precision, recall, and overall accuracy, establishing a new benchmark for PU learning methods [1, 3].2. Paradigm change to pu-learning surpassed supervised learning performance: The study introduces a novel approach that applies PU learning to the classification of accidents at black spots [5, 6]. This methodology significantly improves the accuracy, precision, recall, and F1-score compared to existing supervised learning. It also provides a comprehensive comparative analysis, revealing the statistical significance of the improvements achieved with PU learning.3. Introduction of the GREFood dataset: The proposed pipeline is validated through detailed case studies in the domain of food recognition [2, 8]. These case studies demonstrate the pipeline’s effectiveness in creating balanced, representative datasets that significantly reduce the impact of dataset bias. The study introduces a novel dataset for food recognition comprised of Greek food dishes. The results show that models trained on these newly constructed datasets have better generalization capabilities compared to those trained on traditional datasets.
Degree
thesis:*- Grantor dc:publisher
- Democritus University of Thrace (DUTH)
- Year dc:date
- 2024
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Sevetlidis, Vasileios
- Σεβετλίδης, Βασίλειος
Subjects
dc:subject × 22- Μηχανική μάθηση και αναγνώριση προτύπων
- Μηχανική μάθηση
- Βιομηχανικές εφαρμογές
- Ανίχνευση ανωμαλιών
- Ημι-εποπτευόμενη μάθηση
- Μη ισσοροπημένα σύνολα δεδομένων
- Θετικά και μη σημασμένη μάθηση
- Machine learning and Pattern recognition
- Machine learning
- Industrial applications
- Anomaly detection
- Semi-supervised learning
- Imbalanced data
- Positive and unlabelled learning
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Φυσικές Επιστήμες
- Τεχνητή νοημοσύνη
- Επιστήμη ηλεκτρονικών υπολογιστών, θεωρία και μέθοδοι
- Computer and Information Sciences
- Natural Sciences
- Artificial Intelligence
- Computer science, theory and methods
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/57794