The automatic analysis of emotions is a well-established area in the natural language processing ( NLP ) research field. It has shown valuable and relevant applications in a wide array of domains such as health and well-being, empathetic conversational agents, author profiling, consumer analysis, and security. Most emotion analysis research till now has focused on sources such as news documents and product reviews. In these cases, the NLP task is the classification into predefined closed-set emotion categories (e.g. happy, sad), or alternatively labels (positive, negative). A deep and fine-grained emotion analysis would require explanations of the trigger events that may have led to a user state. This type of analysis is still in its infancy. In this work, we introduce the concept of Emotion Carriers (EC) as the speech or text segments that may include persons, objects, events, or actions that manifest and explain the emotions felt by the narrator during the recollection. In order to investigate this emotion concept, we analyze Personal Narratives (PN) - recollection of events, facts, or thoughts from one’s own experience, - which are rich in emotional information and are less explored in emotion analysis research. PNs are widely used in psychotherapy and thus also in mental well-being applications. The use of PNs in psychotherapy is rooted in the association between mood and recollection of episodic memories. We find that ECs capture implicit emotion information through entities and events whereas the valence prediction relies on explicit emotion words such as happy, cried, and angry. The cues for identifying the ECs and their valence are different and complementary. We propose fine-grained emotion analysis using valence and ECs. We collect and annotate spoken and written PNs, propose text-based and speech-based annotation schemes for valence and EC from PNs, conduct annotation experiments, and train systems for the automatic identification of ECs and their valence.