Dans cette thèse nous étudions différentes questions relatives à la mise en pratique de modèles d'apprentissage profond. En effet malgré les avancées prometteuses de ces algorithmes en vision par ordinateur, leur emploi dans certains cas d'usage réels reste difficile. Une première difficulté est, pour des tâches de classification d'images, de rassembler pour des milliers de catégories suffisamment de données d'entraînement pour chacune des classes. C'est pourquoi nous proposons deux nouvelles approches adaptées à ce scénario d'apprentissage, appelé <<classification zero-shot>>.L'utilisation d'information sémantique pour modéliser les classes permet de définir les modèles par description, par opposition à une modélisation à partir d'un ensemble d'exemples, et rend possible la modélisation sans donnée de référence. L'idée fondamentale du premier chapitre est d'obtenir une distribution d'attributs optimale grâce à l'apprentissage d'une métrique, capable à la fois de sélectionner et de transformer la distribution des données originales. Dans le chapitre suivant, contrairement aux approches standards de la littérature qui reposent sur l'apprentissage d'un espace d'intégration commun, nous proposons de générer des caractéristiques visuelles à partir d'un générateur conditionnel. Une fois générés ces exemples artificiels peuvent être utilisés conjointement avec des données réelles pour l'apprentissage d'un classifieur discriminant. Dans une seconde partie de ce manuscrit, nous abordons la question de l'intelligibilité des calculs pour les tâches de vision par ordinateur. En raison des nombreuses et complexes transformations des algorithmes profonds, il est difficile pour un utilisateur d'interpréter le résultat retourné. Notre proposition est d'introduire un <<goulot d'étranglement sémantique>> dans le processus de traitement. La représentation de l'image est exprimée entièrement en langage naturel, tout en conservant l'efficacité des représentations numériques. L'intelligibilité de la représentation permet à un utilisateur d'examiner sur quelle base l'inférence a été réalisée et ainsi d'accepter ou de rejeter la décision suivant sa connaissance et son expérience humaine. / In this thesis, we examine some practical difficulties of deep learning models.Indeed, despite the promising results in computer vision, implementing them in some situations raises some questions. For example, in classification tasks where thousands of categories have to be recognised, it is sometimes difficult to gather enough training data for each category.We propose two new approaches for this learning scenario, called <<zero-shot learning>>. We use semantic information to model classes which allows us to define models by description, as opposed to modelling from a set of examples.In the first chapter we propose to optimize a metric in order to transform the distribution of the original data and to obtain an optimal attribute distribution. In the following chapter, unlike the standard approaches of the literature that rely on the learning of a common integration space, we propose to generate visual features from a conditional generator. The artificial examples can be used in addition to real data for learning a discriminant classifier. In the second part of this thesis, we address the question of computational intelligibility for computer vision tasks. Due to the many and complex transformations of deep learning algorithms, it is difficult for a user to interpret the returned prediction. Our proposition is to introduce what we call a <<semantic bottleneck>> in the processing pipeline, which is a crossing point in which the representation of the image is entirely expressed with natural language, while retaining the efficiency of numerical representations. This semantic bottleneck allows to detect failure cases in the prediction process so as to accept or reject the decision.

This dissertation explores the application of zero-shot text classification, a technique for categorizing texts without annotated data in the target domain. A true zero-shot setting breaks with the conventions of the traditional supervised machine learning paradigm that relies on quantitative in-domain evaluation for optimization, performance measurement, and model selection. The dissertation summarizes existing research to build a theoretical foundation for zero-shot methods, emphasizing efficiency and transparency. It benchmarks selected approaches across various tasks and datasets to understand their general performance, strengths, and weaknesses, mirroring the model selection process. On this foundation, two case studies demonstrate the application of zero-shot text classification: The first engages with historical German stock market reports, utilizing zero-shot methods for aspect-based sentiment classification. The case study reveals that although there are qualitative differences between finetuned and zero-shot approaches, the aggregated results are not easily distinguishable, sparking a discussion about the practical implications. The second case study integrates zero-shot text classification into a civil engineering document management system, showcasing how the flexibility of zero-shot models and the omission of the training process can benefit the development of prototype software, at the cost of an unknown performance. These findings indicate that, although zero-shot text classification works for the exemplary cases, the results are not generalizable. Taking up the findings of these case studies, the dissertation discusses dilemmas and theoretical considerations that arise from omitting the in-domain evaluation of applying zero-shot text classification. It concludes by advocating a broader focus beyond traditional quantitative metrics in order to build trust in zero-shot text classification, highlighting their practical utility as well as the necessity for further exploration as these technologies evolve.:1 Introduction 1.1 Problem Context 1.2 Related Work 1.3 Research Questions & Contribution 1.4 Author’s Publications 1.5 Structure of This Work 2 Research Context 2.1 The Current State of Text Classification 2.2 Efficiency 2.3 Approaches to Addressing Data Scarcity in Machine Learning 2.4 Challenges of Recent Developments 2.5 Model Sizes and Hardware Resources 2.6 Conclusion 3 Zero-shot Text Classification 3.1 Text Classification 3.2 State-of-the-Art in Text Classification 3.3 Neural Network Approaches to Data-Efficient Text Classification 3.4 Zero-shot Text Classification 3.5 Application 3.6 Requirements for Zero-shot Models 3.7 Approaches to Transfer Zero-shot 3.7.1 Terminology 3.7.2 Similarity-based and Siamese Networks 3.7.3 Language Model Token Predictions 3.7.4 Sentence Pair Classification 3.7.5 Instruction-following Models or Dialog-based Systems 3.8 Class Name Encoding in Text Classification 3.9 Approach Selection 3.10 Conclusion 4 Model Performance Survey 4.1 Experiments 4.1.1 Datasets 4.1.2 Model Selection 4.1.3 Hypothesis Templates 4.2 Zero-shot Model Evaluation 4.3 Dataset Complexity 4.4 Conclusion 5 Case Study: Historic German Stock Market Reports 5.1 Project 5.2 Motivation 5.3 Related Work 5.4 The Corpus and Dataset - Berliner Börsenzeitung 5.4.1 Corpus 5.4.2 Sentiment Aspects 5.4.3 Annotations 5.5 Methodology 5.5.1 Evaluation Approach 5.5.2 Trained Pipeline 5.5.3 Zero-shot Pipeline 5.5.4 Dictionary Pipeline 5.5.5 Tradeoffs 5.5.6 Label Space Definitions 5.6 Evaluation - Comparison of the Pipelines on BBZ 5.6.1 Sentence-based Sentiment 5.6.2 Aspect-based Sentiment 5.6.3 Qualitative Evaluation 5.7 Discussion and Conclusion 6 Case Study: Document Management in Civil Engineering 6.1 Project 6.2 Motivation 6.3 Related Work 6.4 The Corpus and Knowledge Graph 6.4.1 Data 6.4.2 BauGraph – The Knowledge Graph 6.5 Methodology 6.5.1 Document Insertion Pipeline 6.5.2 Frontend Integration 6.6 Discussion and Conclusion 7 MLMC 7.1 How it works 7.2 Motivation 7.3 Extensions of the Framework 7.4 Other Projects 7.4.1 Product Classification 7.4.2 Democracy Monitor 7.4.3 Climate Change Adaptation Finance 7.5 Conclusion 8 Discussion: The Five Dilemmas of Zero-shot 8.1 On Evaluation 8.2 The Five Dilemmas of Zero-shot 8.2.1 Dilemma of Evaluation or Are You Working at All? 8.2.2 Dilemma of Comparison or How Do I Get the Best Model? 8.2.3 Dilemma of Annotation and Label Definition or Are We Talking about the Same Thing? 8.2.4 Dilemma of Interpretation or Am I Biased? 8.2.5 Dilemma of Unsupervised Text Classification or Do I Have to Trust You? 8.3 Trust in Zero-shot Capabilities 8.4 Conclusion 9 Conclusion 9.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 9.1.1 RQ1: Strengths and Weaknesses . . . . . . . . . . . . . . . . 139 9.1.2 RQ2: Application Studies . . . . . . . . . . . . . . . . . . . . 141 9.1.3 RQ3: Implications . . . . . . . . . . . . . . . . . . . . . . . . 143 9.2 Final Thoughts & Future Directions . . . . . . . . . . . . . . . . . . 144 References 147 A Appendix for Survey Chapter A.1 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 A.2 Task-specific Hypothesis Templates . . . . . . . . . . . . . . . . . . 180 A.3 Fractions of SotA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 181 B Uncertainty vs. Accuracy 182 C Declaration of Authorship 185 D Declaration: Use of AI-Tools 186 E Bibliographic Data 187 / In dieser Dissertation wird die Anwendung von Zero-Shot-Textklassifikation -- die Kategorisierung von Texten ohne annotierte Daten in der Anwendungsdomäne -- untersucht. Ein echter Zero-Shot-Ansatz bricht mit den Konventionen des traditionellen überwachten maschinellen Lernens, welches auf einer quantitativen Evaluierung in der Zieldomäne zur Optimierung, Performanzmessung und Modellauswahl (model selection) basiert. Eine Zusammenfassung bestehender Forschungsarbeiten bildet die theoretische Grundlage für die verwendeten Zero-Shot-Methoden, wobei Effizienz und Transparenz im Vordergrund stehen. Ein Vergleich ausgewählter Ansätze mit verschiedenen Tasks und Datensätzen soll allgemeine Stärken und Schwächen aufzeigen und den Prozess der Modellauswahl widerspiegeln. Auf dieser Grundlage wird die Anwendung der Zero-Shot-Textklassifikation anhand von zwei Fallstudien demonstriert: Die erste befasst sich mit historischen deutschen Börsenberichten, wobei Zero-Shot zur aspekt-basierten Sentiment-Klassifikation eingesetzt wird. Es zeigt sich, dass es zwar qualitative Unterschiede zwischen trainierten und Zero-Shot-Ansätzen gibt, dass die aggregierten Ergebnisse aber nicht leicht zu unterscheiden sind, was Überlegungen zu praktischen Implikationen anstößt. Die zweite Fallstudie integriert Zero-Shot-Textklassifikation in ein Dokumentenmanagementsystem für das Bauwesen und zeigt, wie die Flexibilität von Zero-Shot-Modellen und der Wegfall des Trainingsprozesses die Entwicklung von Prototypen vereinfachen können -- mit dem Nachteil, dass die Genauigkeit des Modells unbekannt bleibt. Die Ergebnisse zeigen, dass die Zero-Shot-Textklassifikation in den Beispielanwendungen zwar annähernd funktioniert, die Ergebnisse aber nicht leicht verallgemeinerbar sind. Im Anschluss werden Dilemmata und theoretische Überlegungen erörtert, die sich aus dem Wegfall der Evaluierung in der Zieldomäne von Zero-Shot-Textklassifikation ergeben. Abschließend wird ein breiterer Fokus über die traditionellen quantitativen Metriken hinaus vorgeschlagen, um Vertrauen in die Zero-Shot-Textklassifikation aufzubauen und den praktischen Nutzen zu verbessern. Die Überlegungen zeigen aber auch die Notwendigkeit weiterer Forschung im Zuge der Weiterentwicklung dieser Technologien.:1 Introduction 1.1 Problem Context 1.2 Related Work 1.3 Research Questions & Contribution 1.4 Author’s Publications 1.5 Structure of This Work 2 Research Context 2.1 The Current State of Text Classification 2.2 Efficiency 2.3 Approaches to Addressing Data Scarcity in Machine Learning 2.4 Challenges of Recent Developments 2.5 Model Sizes and Hardware Resources 2.6 Conclusion 3 Zero-shot Text Classification 3.1 Text Classification 3.2 State-of-the-Art in Text Classification 3.3 Neural Network Approaches to Data-Efficient Text Classification 3.4 Zero-shot Text Classification 3.5 Application 3.6 Requirements for Zero-shot Models 3.7 Approaches to Transfer Zero-shot 3.7.1 Terminology 3.7.2 Similarity-based and Siamese Networks 3.7.3 Language Model Token Predictions 3.7.4 Sentence Pair Classification 3.7.5 Instruction-following Models or Dialog-based Systems 3.8 Class Name Encoding in Text Classification 3.9 Approach Selection 3.10 Conclusion 4 Model Performance Survey 4.1 Experiments 4.1.1 Datasets 4.1.2 Model Selection 4.1.3 Hypothesis Templates 4.2 Zero-shot Model Evaluation 4.3 Dataset Complexity 4.4 Conclusion 5 Case Study: Historic German Stock Market Reports 5.1 Project 5.2 Motivation 5.3 Related Work 5.4 The Corpus and Dataset - Berliner Börsenzeitung 5.4.1 Corpus 5.4.2 Sentiment Aspects 5.4.3 Annotations 5.5 Methodology 5.5.1 Evaluation Approach 5.5.2 Trained Pipeline 5.5.3 Zero-shot Pipeline 5.5.4 Dictionary Pipeline 5.5.5 Tradeoffs 5.5.6 Label Space Definitions 5.6 Evaluation - Comparison of the Pipelines on BBZ 5.6.1 Sentence-based Sentiment 5.6.2 Aspect-based Sentiment 5.6.3 Qualitative Evaluation 5.7 Discussion and Conclusion 6 Case Study: Document Management in Civil Engineering 6.1 Project 6.2 Motivation 6.3 Related Work 6.4 The Corpus and Knowledge Graph 6.4.1 Data 6.4.2 BauGraph – The Knowledge Graph 6.5 Methodology 6.5.1 Document Insertion Pipeline 6.5.2 Frontend Integration 6.6 Discussion and Conclusion 7 MLMC 7.1 How it works 7.2 Motivation 7.3 Extensions of the Framework 7.4 Other Projects 7.4.1 Product Classification 7.4.2 Democracy Monitor 7.4.3 Climate Change Adaptation Finance 7.5 Conclusion 8 Discussion: The Five Dilemmas of Zero-shot 8.1 On Evaluation 8.2 The Five Dilemmas of Zero-shot 8.2.1 Dilemma of Evaluation or Are You Working at All? 8.2.2 Dilemma of Comparison or How Do I Get the Best Model? 8.2.3 Dilemma of Annotation and Label Definition or Are We Talking about the Same Thing? 8.2.4 Dilemma of Interpretation or Am I Biased? 8.2.5 Dilemma of Unsupervised Text Classification or Do I Have to Trust You? 8.3 Trust in Zero-shot Capabilities 8.4 Conclusion 9 Conclusion 9.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 9.1.1 RQ1: Strengths and Weaknesses . . . . . . . . . . . . . . . . 139 9.1.2 RQ2: Application Studies . . . . . . . . . . . . . . . . . . . . 141 9.1.3 RQ3: Implications . . . . . . . . . . . . . . . . . . . . . . . . 143 9.2 Final Thoughts & Future Directions . . . . . . . . . . . . . . . . . . 144 References 147 A Appendix for Survey Chapter A.1 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 A.2 Task-specific Hypothesis Templates . . . . . . . . . . . . . . . . . . 180 A.3 Fractions of SotA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 181 B Uncertainty vs. Accuracy 182 C Declaration of Authorship 185 D Declaration: Use of AI-Tools 186 E Bibliographic Data 187