It is no longer any secret that people worldwide are struggling with their mental health, in terms of diagnostic disorders as well as non-diagnostic measures like perceived stress. Barriers to receiving professional mental healthcare are significant, and even in locations where the availability of such care is increasing, our infrastructures are not equipped to find people the support they need. Meanwhile, in a highly-connected digital world, many people turn to outlets like social media to express themselves and their struggles and interact with like-minded others. This setting---where human experts are overwhelmed and human patients are acutely in need---is one in which we believe artificial intelligence (AI) and natural language processing (NLP) systems have great potential to do good. At the same time, we must acknowledge the limitations of our models and strive to deploy them responsibly alongside human experts, such that their logic and mistakes are transparent. We argue that models that make and explain their predictions in ways guided by domain-specific research will be more understandable to humans, who can benefit from the models' statistical knowledge but use their own judgment to mitigate the models' mistakes. In this thesis, we leverage domain expertise in the form of psychology research to develop models for two categories of emotional tasks: identifying emotional reactions in text and explaining the causes of emotional reactions. The first half of the thesis covers our work on detecting emotional reactions, where we focus on a particular, understudied type of emotional reaction: psychological distress. We present our original dataset, Dreaddit, gathered for this problem from the social media website Reddit, as well as some baseline analysis and benchmarking that shows psychological distress detection is a challenging problem. Drawing on literature that connects particular emotions to the experience of distress, we then develop several multitask models that incorporate basic emotion detection, and quantitatively change the way our distress models make their predictions to make them more readily understandable. Then, the second half of the thesis expands our scope to consider not only the emotional reaction being experienced, but also its cause. We treat this cause identification problem first as a span extraction problem in news headlines, where we employ multitask learning (jointly with basic emotion classification) and commonsense reasoning; and then as a free-form generation task in response to a long-form Reddit post, where we leverage the capabilities of large language models (LLMs) and their distilled student models. Here, as well, multitask learning with basic emotion detection is beneficial to cause identification in both settings. Our contributions in this thesis are fourfold. First, we produce a dataset for psychological distress detection, as well as emotion-infused models that incorporate emotion detection for this task. Second, we present multitask and commonsense-infused models for joint emotion detection and emotion cause extraction, showing increased performance on both tasks. Third, we produce a dataset for the new problem of emotion-focused explanation, as well as characterization of the abilities of distilled generation models for this problem. Finally, we take an overarching approach to these problems inspired by psychology theory that incorporates expert knowledge into our models where possible, enhancing explainability and performance.

In this short paper we present work in progress that tries to leverage crowdsourced music metadata and crowdsourced affective word norms to create a comprehensive dataset of music emotions, which can be used for sentiment analyses in the music domain. We combine a mixture of different data sources to create a new dataset of 90,408 songs with their associated embeddings in Russell’s model of affect, with the dimensions valence, dominance and arousal. In addition, we provide a Spotify ID for the songs, which can be used to add more metadata to the dataset via the Spotify API.

Работа предложила систему идентификации эмоций с использованием глубокого обучения. Исследование продвигает взаимодействие человека и компьютера, мониторинг психического здоровья, маркетинговые исследования, анализ настроений и подчеркивает необходимость нейронных сетей. Оно стремится построить модель, которая учится на основе сырой речи. Оно разработано с использованием модели CNN и LSTM, блок классификации использует блоки LSTM для захвата долгосрочных временных корреляций. Это происходит после того, как блок извлечения признаков использует одновременные CNN и MFCC. Эти методы гарантируют, что блок категоризации может точно отображать данные. Подготовка данных для обучения и тестирования модели прогнозирования эмоций на основе набора данных CREMA-D является значительной. Для оптимизации производительности нейронной сети метод включает разделение признаков и меток, кодирование, разделение набора данных, стандартизацию и изменение формы данных. Для упрощения и снижения сложности он исключает подходы к дополнению данных. Модель обучается и оценивается с использованием CREMA-D, набор данных содержит 7442 голосовых записи, представляющих различные эмоции. В исследовании принимают участие 84 мужчины и 43 женщины в возрасте от 20 до 74 лет. Средняя точность модели составляет 86,92% по результатам проверки. В будущем исследования могут быть сосредоточены на разработке приложений для идентификации эмоций в реальном времени и интеграции мультимодальных данных для повышения точности и надежности систем обнаружения эмоций. / The work proposed emotion identification system using deep learning. The research advances human-computer interaction, mental health monitoring, market research, sentiment analysis and emphasizes the necessity of neural networks. It’s seeking to construct a model that learns from raw speech audio. It’s developed using CNNs and LSTMs model, a classification block uses LSTM units to capture long-term temporal correlations. This happens after a feature extraction block uses imultaneous CNNs and MFCCs. These methods ensure the categorization block can accurately display data. Data preparation for training and testing a CREMA-D dataset-based emotion prediction model is considerable. To optimize neural network performance, the method includes feature-label separation, encoding, dataset splitting, standardization, and data reshaping. To simplify and reduce complexity, it excludes data augmentation approaches. The model is trained and evaluated using CREMA-D, dataset contains 7,442 voice recordings representing different emotions. There are 84 male and 43 female performers, with ages ranging from 20 to 74 years old. The model has an average accuracy of 86.92% across validation. In the future, research may focus on developing real-time emotion identification applications and integrating multimodal data to enhance the accuracy and robustness of emotion detection systems.

Children with autism spectrum disorder (ASD) show significant deficits in communication, emotion recognition, perspective taking, and social skills. One intervention gaining increased attention is the use of computer assisted instruction (CAI) to teach social, emotional and perspective-taking skills to individuals with ASD with the purpose of improving theory of mind skills. This study evaluated the effectiveness of CAI for improving theory of mind skills in four children with high functioning autism ages 5 to 12 years. A single-subject multiple baseline research design across participants was utilized to evaluate the effectiveness of CAI. The software contained 22 instructional scenarios that asked participants to identify emotions of characters based on situational cues displayed in line drawn pictures and audio feedback for correct and incorrect responses. Mind-reading skills were assessed using ten randomly selected scenarios for various emotions and no audio feedback. Visual analysis of the data revealed that all four participants increased mind-reading skills during the CAI condition. Additionally, this study evaluated levels of task engagement during experimental conditions. Three of the four participants showed an increase in task engagement during CAI compared to paper-based social stories used during baseline. Generalization of skills was assessed through the use of social scenarios acted out by family members of participants. All four participants were able to correctly identify emotions displayed in generalization scenarios. Results demonstrated that CAI was an effective and socially viable method for improving ToM skills in children with autism and they could generalize their skills to untrained settings.

Multimodal Emotion Recognition (MER) has gained increasing attention due to its exceptional performance. In this thesis, we evaluate feature-level fusion, decision-level fusion, and two proposed hierarchical fusion methods for MER systems using a dialogue-based dataset. The first hierarchical approach integrates abstract features across different temporal levels by employing RNN-based and transformer-based context modeling techniques to capture nearby and global context respectively. The second hierarchical strategy incorporates shared information between modalities by facilitating modality interactions through attention mechanisms. Results reveal that RNN-based hierarchical fusion surpasses the baseline by 2%, while transformer-based context modeling and modality interaction methods improve accuracy by 0.5% and 0.6%, respectively. These findings underscore the significance of capturing meaningful emotional cues in nearby context and emotional invariants in dialogue MER systems. We also emphasize the crucial role of text modality. Overall, our research highlights the potential of hierarchical fusion approaches for enhancing MER system performance, presenting systematic strategies supported by empirical evidence. / Multimodal Emotion Recognition (MER) har fått ökad uppmärksamhet på grund av dess exceptionella prestanda. I denna avhandling utvärderar vi feature-level fusion, decision-level fusion och två föreslagna hierarkiska fusion-metoder för MER-system med hjälp av en dialogbaserad dataset. Den första hierarkiska metoden integrerar abstrakta funktioner över olika tidsnivåer genom att använda RNN-baserade och transformer-baserade tekniker för kontextmodellering för att fånga närliggande och globala kontexter, respektive. Den andra hierarkiska strategin innefattar delad information mellan modaliteter genom att underlätta modalitetsinteraktioner genom uppmärksamhetsmekanismer. Resultaten visar att RNN-baserad hierarkisk fusion överträffar baslinjen med 2%, medan transformer-baserad kontextmodellering och modellering av modalitetsinteraktion ökar noggrannheten med 0.5% respektive 0.6%. Dessa resultat understryker betydelsen av att fånga meningsfulla känslomässiga ledtrådar i närliggande sammanhang och emotionella invarianter i dialog MER-system. Vi betonar också den avgörande rollen som textmodalitet spelar. Övergripande betonar vår forskning potentialen för hierarkiska fusion-metoder för att förbättra prestandan i MER-system, genom att presentera systematiska strategier som stöds av empirisk evidens.

Dans ce travail, nous explorons la faisabilité de doter les machines de la capacité de prédire, dans un contexte d'interaction homme-machine (IHM), l'émotion d'un utilisateur, ainsi que son intensité, de manière instantanée pour une grande variété de situations. Plus spécifiquement, une application a été développée, appelée machine émotionnelle, capable de «comprendre» la signification d'une situation en se basant sur le modèle théorique d'évaluation de l'émotion Ortony, Clore et Collins (OCC). Cette machine est apte, également, à prédire les réactions émotionnelles des utilisateurs, en combinant des versions améliorées des k plus proches voisins et des réseaux de neurones. Une procédure empirique a été réalisée pour l'acquisition des données. Ces dernières ont fourni une connaissance consistante aux algorithmes d'apprentissage choisis et ont permis de tester la performance de la machine. Les résultats obtenus montrent que la machine émotionnelle proposée est capable de produire de bonnes prédictions. Une telle réalisation pourrait encourager son utilisation future dans des domaines exploitant la reconnaissance automatique de l'émotion. / This work explores the feasibility of equipping computers with the ability to predict, in a context of a human computer interaction, the probable user’s emotion and its intensity for a wide variety of emotion-eliciting situations. More specifically, an online framework, the Emotional Machine, is developed enabling computers to «understand» situations using OCC model of emotion and to predict user’s reaction by combining refined versions of Artificial Neural Network and k Nearest Neighbours algorithms. An empirical procedure including a web-based anonymous questionnaire for data acquisition was designed to provide the chosen machine learning algorithms with a consistent knowledge and to test the application’s recognition performance. Results from the empirical investigation show that the proposed Emotional Machine is capable of producing accurate predictions. Such an achievement may encourage future using of our framework for automated emotion recognition in various application fields.

La reconnaissance d'émotions est un domaine jeune mais dont la maturité grandissante implique de nouveaux besoins en termes de modélisation et d'intégration dans des modèles existants. Ce travail de thèse expose un modèle conceptuel pour la conception d'applications interactives sensibles aux émotions de l'utilisateur. Notre approche se fonde sur les résultats conceptuels issus de l'interaction multimodale: nous redéfinissons les concepts de modalité et de multimodalité dans le cadre de la reconnaissance passive d'émotions. Nous décrivons ensuite un modèle conceptuel à base de composants logiciels s'appuyant sur cette redéfinition: la branche émotion, facilitant la conception, le développement et le maintien d'applications reconnaissant l'émotion de l'utilisateur. Une application multimodale de reconnaissance d'émotions par la gestuelle a été développée selon le modèle de la branche émotion et intégrée dans un système d'augmentation de spectacle de ballet sensible aux émotions d'un danseur. / Computer-based emotion recognition is a growing field which develops new needs in terms of software modeling and integration of existing models. This thesis describes a conceptual framework for designing emotionally-aware interactive software. Our approach is based upon conceptual results from the field of multimodal interaction: we redefine the concepts of modality and multimodality within the frame of passive emotion recognition. We then describe a component-based conceptual model relying on this redefinition. The emotion branch facilitates the design, development and maintenance of emotionally-aware systems. A multimodal, interactive, gesture-based emotion recognition software based on the emotion branch was developed. This system was integrated within an augmented reality system to augment a ballet dance show according to the dancer's expressed emotions.

[pt] As expressões faciais transmitem muita informação sobre um indivíduo, tornando a capacidade de interpretá-las uma tarefa muito importante, com aplicações em diversas áreas, tais como Interação Homem Máquina, Jogos Digitais, storytelling interativo e TV/Cinema digital. Esta dissertação discute o processo de reconhecimento de emoções em tempo real usando ASM (Active Shape Model) e SVM (Support Vector Machine) e apresenta uma comparação entre duas formas comumente utilizadas na etapa de extração de atributos: faces neutra e média. Como não existe tal comparação na literatura, os resultados apresentados são valiosos para o desenvolvimento de aplicações envolvendo expressões de emoção em tempo real. O presente trabalho considera seis tipos de emoções: felicidade, tristeza, raiva, medo, surpresa e desgosto. / [en] The facial expressions provide a high amount of information about a person, making the ability to interpret them a high valued task that can be used in several fields of Informatics such as Human Machine Interface, Digital Games, interactive storytelling and digital TV/Cinema. This dissertation discusses the process of recognizing emotions in real time using ASM (Active Shape Model) and SVM (Support Vector Machine) and presents a comparison between two commonly used ways when extracting the attributes: neutral face and average. As such comparison can not be found in the literature, the results presented are valuable to the development of applications that deal with emotion expression in real time. The current study considers six types of emotions: happiness, sadness, anger, fear, surprise and disgust.

Dans ce travail, nous explorons la faisabilité de doter les machines de la capacité de prédire, dans un contexte d'interaction homme-machine (IHM), l'émotion d'un utilisateur, ainsi que son intensité, de manière instantanée pour une grande variété de situations. Plus spécifiquement, une application a été développée, appelée machine émotionnelle, capable de «comprendre» la signification d'une situation en se basant sur le modèle théorique d'évaluation de l'émotion Ortony, Clore et Collins (OCC). Cette machine est apte, également, à prédire les réactions émotionnelles des utilisateurs, en combinant des versions améliorées des k plus proches voisins et des réseaux de neurones. Une procédure empirique a été réalisée pour l'acquisition des données. Ces dernières ont fourni une connaissance consistante aux algorithmes d'apprentissage choisis et ont permis de tester la performance de la machine. Les résultats obtenus montrent que la machine émotionnelle proposée est capable de produire de bonnes prédictions. Une telle réalisation pourrait encourager son utilisation future dans des domaines exploitant la reconnaissance automatique de l'émotion. / This work explores the feasibility of equipping computers with the ability to predict, in a context of a human computer interaction, the probable user’s emotion and its intensity for a wide variety of emotion-eliciting situations. More specifically, an online framework, the Emotional Machine, is developed enabling computers to «understand» situations using OCC model of emotion and to predict user’s reaction by combining refined versions of Artificial Neural Network and k Nearest Neighbours algorithms. An empirical procedure including a web-based anonymous questionnaire for data acquisition was designed to provide the chosen machine learning algorithms with a consistent knowledge and to test the application’s recognition performance. Results from the empirical investigation show that the proposed Emotional Machine is capable of producing accurate predictions. Such an achievement may encourage future using of our framework for automated emotion recognition in various application fields.

Η παρούσα διδακτορική διατριβή πραγματεύεται προβλήματα που αφορούν το χώρο της τεχνολογίας ομιλίας, με στόχο τη αναγνώριση συναισθημάτων από ομιλία με χρήση τεχνικών ψηφιακής επεξεργασίας σήματος και μηχανικής μάθησης. Πιο αναλυτικά, στα πλαίσια της διατριβής προτάθηκαν και μελετήθηκαν καινοτόμες μέθοδοι σε μια σειρά από εφαρμογές που αξιοποιούν σύστημα αναγνώρισης συναισθηματικών καταστάσεων από ομιλία. Ο βασικός στόχος των μεθόδων ήταν η αντιμετώπιση των προκλήσεων που παρουσιάζονται όταν ένα σύστημα αναγνώρισης συναισθηματικών καταστάσεων καλείται να λειτουργήσει σε πραγματικές συνθήκες, με αυθόρμητες αντιδράσεις, ανεξαρτήτως ομιλητή. Πιο συγκεκριμένα, στα πλαίσια της διατριβής, αξιολογήθηκε η συμπεριφορά ενός συστήματος αναγνώρισης συναισθημάτων σε προσποιητή ομιλία και σε διαφορετικές συνθήκες θορύβου, και συγκρίθηκε η απόδοση του συστήματος με την υποκειμενική αξιολόγηση των ακροατών. Επιπλέον, περιγράφηκε ο σχεδιασμός και η υλοποίηση βάση δεδομένων συναισθηματικής ομιλίας, όπως αυτή προκύπτει από την αλληλεπίδραση μη-έμπειρων χρηστών με ένα διαλογικό σύστημα και προτάθηκε ένα σύστημα το οποίο εντοπίζει αρνητικές συναισθηματικές καταστάσεις, στο ανεξάρτητου ομιλητή πρόβλημα, με χρήση μοντέλου Γκαουσιανών κατανομών. Η προτεινόμενη αρχιτεκτονική συνδυάζει παραμέτρους ομιλίας χαμηλού και υψηλού επιπέδου και εφαρμόζεται στα πραγματικά δεδομένα. Επίσης, αξιολογήθηκε και υλοποιήθηκε η πρακτική εφαρμογή ενός συστήματος αναγνώρισης συναισθημάτων βασισμένου σε οικουμενικό μοντέλο Γκαουσιανών κατανομών σε διαφορετικούς τύπους δεδομένων πραγματικής ζωής. Ακόμα, παρουσιάστηκε μια πρωτότυπη αρχιτεκτονική κατηγοριοποίησης για αναγνώριση συνυπαρχόντων συναισθημάτων από ομιλία προερχόμενη από αλληλεπίδραση σε πραγματικά περιβάλλοντα. Σε αντίθεση με γνωστές προσεγγίσεις, η προτεινόμενη αρχιτεκτονική μοντελοποιεί τις συνυπάρχουσες συναισθηματικές καταστάσεις μέσω της κατασκευής μιας πολυσταδιακής αρχιτεκτονικής κατηγοριοποίησης. Τα πειραματικά αποτελέσματα που διενεργήθηκαν υποδεικνύουν ότι η προτεινόμενη αρχιτεκτονική είναι πλεονεκτική για τις συναισθηματικές καταστάσεις που είναι πιο διαχωρίσιμες, γεγονός που οδηγεί σε βελτίωση της συνολικής απόδοσης του συστήματος. / In this doctoral dissertation a number of novel approaches were proposed and evaluated in different applications that utilize emotion awareness. The major target of the proposed methods was facing the difficulties existing, when an emotion recognition system is asked to operate in real-life conditions, where human speech is characterized by spontaneous and genuine formulations. In detail, within the present dissertation, the performance of an emotion recognition system was evaluated, initially, in acted speech, under different noise conditions, and this performance was compared to the one of human listeners. Further, the design and implementation of a real world emotional speech corpus is described, as this results from the interaction of naive users with a smart home dialogue system. Moreover, a system which utilizes low and high level descriptors was suggested. The suggested architecture leads to significantly better performance in some working points of the integrated system in the dialogue system. Furthermore, we propose a novel multistage classification scheme for affect recognition from real-life speech. In contrast with conventional approaches for affect/emotion recognition from speech, the proposed scheme models co-occurring affective states by constructing a multistage classification scheme. The empirical experiments performed indicate that the proposed classification scheme offers an advantage for those classes that are more separable, which contributes for improving the overall performance of the affect recognition system.