Humans can often report a subjective sense of confidence in a decision before knowing its outcome. Such confidence judgements are positively correlated to accuracy in perceptual and memory tasks, but the strength of this relationship (known as metacognitive accuracy) differs across people and contexts. Computationally, confidence judgements are believed to relate to the strength of evidence favouring each option, but it has been suggested that confidence also captures information from other sources, such as response time. This thesis explores the pragmatics of confidence: what factors influence confidence judgements, how accurate confidence judgements are, and how they might influence future behaviour. Our knowledge of the antecedents of confidence is extended by this work in two ways, by introducing novel predictors of confidence and by increasing our understanding of well-known ones. I find that bilinguals have worse metacognitive accuracy than monolinguals. This bilingual deficiency in metacognitive accuracy cannot be explained by response time and stimulus strength, suggesting that there is at least one important predictor of confidence that remains unaccounted for. I introduce such a predictor in a new eye tracking correlate of confidence: Gaze-shift-frequency, the number of saccades between options, negatively predicts subsequent confidence in perceptual and value-based decisions. In the value domain, the total value of the options is shown to positively relate to confidence despite being negatively related to accuracy, the first such dissociation to be recorded, as far as I am aware. The dissertation extends our understanding of response time as a predictor of confidence by showing that it influences confidence more for judgements that are made after a choice, relative to those made simultaneously with the choice. This differential influence of response time explains the higher metacognitive accuracy of sequential confidence reports. I explore the consequences of confidence judgements in the context of value-based choice. Lower levels of confidence are associated with changes of mind when the same options recur in subsequent trials. To test whether these changes of mind are rational, I approximate choice accuracy in the value domain. I propose a novel method based on the transitivity of the full choice set, so that choices that violate the most transitive ordering of the items can be treated as errors. I find that participants who were more metacognitively accurate showed a decrease in transitivity violations over time. These results extend prior work linking confidence judgements to error correction in the perceptual domain.

La dépendance à la nicotine est un problème sociétal majeur. De nombreuses personnes continuent à fumer, malgré des conséquences négatives pour la santé bien connues. La recherche sur la toxicomanie se concentre généralement sur la motivation à obtenir cette drogue, le sevrage, rendu difficile par les symptômes de manque et la persistance des comportements appris. Au cours de cette thèse, je me suis concentrée sur les interactions entre les systèmes dopaminergique et nicotinique dans les comportements de prise de décision. Je me suis particulièrement intéressée aux effets de la nicotine administrée passivement de manière chronique. Mettre en évidence les modifications dans la signalisation dopaminergique, et les altérations des comportements de choix qu'implique la consommation de drogue, permet un autre éclairage sur l'état de dépendance, mais ouvre aussi de nouvelles perspectives sur le traitement des addictions et sur les notions de vulnérabilité et de comorbidité avec d'autres pathologies. Les théories de la prise de décision suggèrent que les individus analysent les bénéfices et les coûts potentiels pour guider leurs actions. Ainsi, faire des choix appropriés nécessite d'apprendre la valeur des options disponibles à partir de l'expérience. Ce processus reposerait principalement sur l'activité des neurones dopaminergiques. Une question cruciale est de savoir si les processus computationnels sous-jacents aux choix peuvent être modifiés de manière durable par l'exposition à la nicotine et/ou par la manipulation des récepteurs nicotiniques. Nous avons développé un paradigme comportemental, de type bandit manchot avec des stimulations intra-crâniales comme récompenses, qui permet d'évaluer la prise de décision basée sur la valeur et l'exploration. Nous avons étudié le rôle de la transmission nicotinique dans l'équilibre comportemental entre exploration et exploitation à l'aide de souris knock-out pour la sous-unité beta2 du récepteur nicotinique. Nous démontrons que les souris beta2KO explorent moins que les souris de type sauvage suggérant un rôle des nAChR dans la traduction de l'incertitude attendue en motivation à explorer. Dans une deuxième étape, nous analysons l'effet d'une exposition chronique à la nicotine sur ces processus de prise de décision. Il a été proposé que la libération phasique de dopamine serait cruciale pour l\'apprentissage, alors que l'activité tonique serait plus impliquée dans l'expression d'un comportement précédemment acquis. Nous montrons ici que la nicotine, sur le long terme, peut altérer l'activité spontanée des cellules dopaminergiques et ainsi modifier la libération phasique et tonique de dopamine. Cette modification se traduit par une augmentation de la "sensibilité à la valeur", et donc une altération des choix. Un individu sous nicotine chronique se concentre davantage sur les récompenses plus importantes. Nous avons mimé cet effet par une activation optogénétique tonique des neurones dopaminergiques. Ces travaux mettent en lumière les mécanismes qui sous-tendent les changements dans la prise de décision lors d'une exposition aux drogues. / Nicotine addiction is a major societal dilemma: despite the well-known adverse consequences to health, more than one billion persist in the habit. Much addiction research focuses on the motivation to obtain and take drugs and the difficulties of abstinence due to withdrawal effects and learned behaviour. My research focuses on the interactions between the dopaminergic and nicotinic systems in value-based decision-making behaviour, as well as the effects of passively administered nicotine within this framework. Understanding the alterations in the dopamine (DA) system and choice behaviour occurring with drug consumption provides another angle on the addictive state, the general predispositions and vulnerabilities for co morbid disorders. This knowledge could unveil new perspectives towards addiction treatment. Theories of decision-making suggest that individuals analyse potential benefits and costs to guide their actions based on experience. A multitude of studies have drawn links between DA cell activity and such choice behaviour. Dopamine signals attribute to the evaluation of available options to select future actions. The addictive nature of nicotine is well established, but can a much more alarming link be made between its effect on the DA system and fundamental perception of our environment ? Could basic computations underlying choices be significantly modified by drug exposure and/or by the manipulation of nicotinic receptors ? We designed a behavioural paradigm that allows us to assess value-based decision-making in different mouse models and with different manipulations of the cholinergic-dopaminergic circuits. We investigated the role of nicotinic transmission in the exploration/exploitation tradeoff using a beta2 subunit-containing nicotinic acetylcholine receptor knockout mouse model. Using a spatial version of a three-armed bandit task and intra-cranial self-stimulation as reward, we demonstrate that beta2KO mice explore less than the WT. This finding suggests the role of specific nAChRs in the translation of expected uncertainty into motivational value and exploratory decision-making. Secondly, we analysed the effect of chronic nicotine exposure in this decision process. It has been proposed that phasic dopamine release could provide a teaching signal necessary for learning, while tonic dopamine levels could influence the implementation of previously learned behaviours. Here we show that long-term nicotine exposure can alter the spontaneous activity of VTA DA cells and therefore change phasic and tonic DA signalling. These modifications translate into an enhanced value-sensitivity; the ability to discriminate between different values. We replicated this effect through optical stimulation of VTA DA neurons on a high tonic frequency. This illustrates that organisms under chronic nicotine focus on higher rewards. This data sheds a new light on the mechanisms underlying drug-induced changes in decision-making.

Alcohol use disorder (AUD) is a widespread mental disease denoted by chronic alcohol use despite significant negative consequences for a person’s life. It affected more than 14 million persons in Europe alone and accounted for more than 5% of deaths worldwide in 2011-2012. Understanding the psychological and neurobiological mechanisms driving the development and maintenance of pathological alcohol use is key to conceptualizing new programs for prevention and therapy of AUD. There has been a variety of etiological models trying to describe and relate these mechanisms. Lately, the view of AUD as a disorder of learning and decision making has received much support proposing dual systems to be at work in AUD – one system being deliberate, forward-planning, and goal-directed and the other one reflexive, automatic, and habitual. Both systems supposedly work in parallel in a framework of value-based decision making and their balance can be flexibly adjusted in healthy agents, while a progressive imbalance favoring habitual over goal-directed choice strategies is assumed in AUD. This imbalance has been theoretically associated to neural adaptations to chronic alcohol use in corticostriatal pathways involved in reward processing, especially in ventral striatum. However, these theoretical models are grounded strongly on animal research while empirical research in the human domain remains rather sparse and inconclusive. Furthermore, alterations in value-based decision-making processes and their neural implementation might not only result from prolonged alcohol misuse but may also represent premorbid interindividual differences posing a risk factor for the development of AUD. Therefore, I here present three studies investigating the relation of alcohol use with the balance between goal-directed and habitual decision systems and with parameters modulating option valuation processes of these systems, namely delay, risk, and valence of option outcomes. To separate the investigation of these decision processes as predisposing risk for or consequence of alcohol use, two samples were examined: one sample of 201 eighteen-year-old men being neither abstinent from nor dependent on alcohol as well as one sample of 114 AUD patients in detoxification treatment and 98 control participants matched for age, sex, educational background, and smoking status. Both samples had a baseline assessment of several behavioral tasks, questionnaires, and neuropsychological testing and were followed-up over one year to examine drinking trajectories in the sample of young men and relapse in detoxified patients. The behavioral tasks included a sequential choice task using model-free and model-based reinforcement learning as operationalization of habitual and goal-directed decision making, respectively, during functional magnetic resonance imaging and four tasks probing participants’ delay discounting, probability discounting for gains and losses, and loss aversion. Study 1 presents the cross-sectional analysis of the sequential choice task in relation to baseline drinking behavior of the young-adult sample. These analyses did not reveal an association between non-pathological alcohol use and habitual and goal-directed control on neither a behavioral nor neural level except for one exploratory finding of increased BOLD responses to model-free habitual learning signals in participants with earlier onset of drinking. Study 2 examined the same task in AUD patients compared to control participants showing no difference in behavioral control or neural correlates between those groups. However, prospectively relapsing AUD patients showed lower BOLD responses associated to model-based goal-directed control than abstaining patients and control participants. Additionally, the interaction of goal-directed control and positive expectancies of alcohol effects discriminated subsequently relapsing and abstaining patients revealing an increased risk of relapse for those patients who showed higher levels of goal-directed control and low alcohol expectancies or low levels of goal-directedness and high expectancies. Study 3 examined modulating features of goal-directed and habitual option valuation – delay, risk, and valence of options – in association to alcohol use in the young-adult sample and AUD status in the sample of patients and matched control participants on a cross-sectional as well as longitudinal level. This study revealed no relation of delay, risk, and loss aversion with current alcohol use and consumption one year later in the young men. In contrast, AUD patients showed systematically more impulsive choice behavior than control participants in all four tasks: a higher preference for immediate rewards, more risky choices when facing gains and less when facing losses, and lower loss aversion. Furthermore, a general tendency to overestimate the probability of uncertain losses could predict relapse risk over the following year in AUD patients. Taken together, these results do not support the hypothesis that mechanisms of value-based decision making might be predisposing risk factors for alcohol consumption. The findings for patients already suffering from AUD are mixed: while choice biases regarding delays, risks, and valence of option outcomes seem to be altered systematically in AUD, there was no indication of an imbalance of habitual and goal-directed control. These findings challenge the assumption of a generalized outcome-unspecific shift of behavioral control from goal-directed to habitual strategies during the development of AUD and point towards several possible future avenues of research to modify or extend the theoretical model.

Alcohol use disorder (AUD) is a widespread mental disease denoted by chronic alcohol use despite significant negative consequences for a person’s life. It affected more than 14 million persons in Europe alone and accounted for more than 5% of deaths worldwide in 2011-2012. Understanding the psychological and neurobiological mechanisms driving the development and maintenance of pathological alcohol use is key to conceptualizing new programs for prevention and therapy of AUD. There has been a variety of etiological models trying to describe and relate these mechanisms. Lately, the view of AUD as a disorder of learning and decision making has received much support proposing dual systems to be at work in AUD – one system being deliberate, forward-planning, and goal-directed and the other one reflexive, automatic, and habitual. Both systems supposedly work in parallel in a framework of value-based decision making and their balance can be flexibly adjusted in healthy agents, while a progressive imbalance favoring habitual over goal-directed choice strategies is assumed in AUD. This imbalance has been theoretically associated to neural adaptations to chronic alcohol use in corticostriatal pathways involved in reward processing, especially in ventral striatum. However, these theoretical models are grounded strongly on animal research while empirical research in the human domain remains rather sparse and inconclusive. Furthermore, alterations in value-based decision-making processes and their neural implementation might not only result from prolonged alcohol misuse but may also represent premorbid interindividual differences posing a risk factor for the development of AUD. Therefore, I here present three studies investigating the relation of alcohol use with the balance between goal-directed and habitual decision systems and with parameters modulating option valuation processes of these systems, namely delay, risk, and valence of option outcomes. To separate the investigation of these decision processes as predisposing risk for or consequence of alcohol use, two samples were examined: one sample of 201 eighteen-year-old men being neither abstinent from nor dependent on alcohol as well as one sample of 114 AUD patients in detoxification treatment and 98 control participants matched for age, sex, educational background, and smoking status. Both samples had a baseline assessment of several behavioral tasks, questionnaires, and neuropsychological testing and were followed-up over one year to examine drinking trajectories in the sample of young men and relapse in detoxified patients. The behavioral tasks included a sequential choice task using model-free and model-based reinforcement learning as operationalization of habitual and goal-directed decision making, respectively, during functional magnetic resonance imaging and four tasks probing participants’ delay discounting, probability discounting for gains and losses, and loss aversion. Study 1 presents the cross-sectional analysis of the sequential choice task in relation to baseline drinking behavior of the young-adult sample. These analyses did not reveal an association between non-pathological alcohol use and habitual and goal-directed control on neither a behavioral nor neural level except for one exploratory finding of increased BOLD responses to model-free habitual learning signals in participants with earlier onset of drinking. Study 2 examined the same task in AUD patients compared to control participants showing no difference in behavioral control or neural correlates between those groups. However, prospectively relapsing AUD patients showed lower BOLD responses associated to model-based goal-directed control than abstaining patients and control participants. Additionally, the interaction of goal-directed control and positive expectancies of alcohol effects discriminated subsequently relapsing and abstaining patients revealing an increased risk of relapse for those patients who showed higher levels of goal-directed control and low alcohol expectancies or low levels of goal-directedness and high expectancies. Study 3 examined modulating features of goal-directed and habitual option valuation – delay, risk, and valence of options – in association to alcohol use in the young-adult sample and AUD status in the sample of patients and matched control participants on a cross-sectional as well as longitudinal level. This study revealed no relation of delay, risk, and loss aversion with current alcohol use and consumption one year later in the young men. In contrast, AUD patients showed systematically more impulsive choice behavior than control participants in all four tasks: a higher preference for immediate rewards, more risky choices when facing gains and less when facing losses, and lower loss aversion. Furthermore, a general tendency to overestimate the probability of uncertain losses could predict relapse risk over the following year in AUD patients. Taken together, these results do not support the hypothesis that mechanisms of value-based decision making might be predisposing risk factors for alcohol consumption. The findings for patients already suffering from AUD are mixed: while choice biases regarding delays, risks, and valence of option outcomes seem to be altered systematically in AUD, there was no indication of an imbalance of habitual and goal-directed control. These findings challenge the assumption of a generalized outcome-unspecific shift of behavioral control from goal-directed to habitual strategies during the development of AUD and point towards several possible future avenues of research to modify or extend the theoretical model.:Table of Contents List of Figures List of Tables List of Abbreviations Abstract Chapter 1. Perspectives on alcohol use disorder 1.1 The size of alcohol use disorder 1.1.1 Terminology of alcohol-use related disorders 1.1.2 Size and burden of alcohol consumption and alcohol use disorders 1.2 Cognitive psychological perspectives on alcohol use disorder 1.2.1 A unified framework for addiction 1.2.2 Value-based decision making 1.2.3 Goal-directed and habitual systems 1.3 Neurobiological perspectives on alcohol use disorders 1.3.1 Neural underpinnings of the reward circuit 1.3.2 Neural underpinning of goal-directed and habitual decision making 1.3.3 Striatal adaptations associated with chronic alcohol consumption 1.4 Synopsis and research questions Chapter 2. Study 1 2.1 Abstract 2.2 Introduction 2.3 Material and methods 2.3.1 Participants and procedure 2.3.2 Measures of goal-directed and habitual behavioral control 2.3.3 Measure of alcohol consumption 2.3.4 Behavioral statistical analyses 2.3.5 Functional magnetic resonance imaging data acquisition and analysis 2.4 Results 2.4.1 Sample characteristics 2.4.2 Behavioral results 2.4.3 Functional magnetic resonance imaging results 2.5 Discussion Chapter 3. Study 2 3.1 Abstract 3.2 Introduction 3.3 Methods and materials 3.3.1 Participants 3.3.2 Procedure 3.3.3 Alcohol Expectancy Questionnaire 3.3.4 Task 3.3.5 Magnetic Resonance Imaging 3.3.6 Follow-up procedure 3.3.7 Data analysis 3.3.8 fMRI analysis 3.4 Results 3.4.1 Sample characteristics 3.4.2 Task-related group differences 3.4.3 Interaction between alcohol expectancies and model-based control 3.4.4 fMRI results 3.5 Discussion Chapter 4. Study 3 4.1 Abstract 4.2 Introduction 4.3 Study 3.1 4.3.1 Material and methods 4.3.2 Results 4.4 Study 3.2 4.4.1 Material and methods 4.4.2 Results 4.5 Discussion Chapter 5. General discussion 5.1 Summary of findings and discussion 5.1.1 Goal-directed and habitual decision making and alcohol use (disorder) 5.1.2 Neuroimaging correlates of goal-directed and habitual control 5.1.3 Modulators of the valuation systems and alcohol use (disorders) 5.1.4 Integration of findings 5.2 Limitations 5.2.1 Methodological critique of the Two-Step task 5.3 Outlook for future studies 5.3.1 Tentative framework for future studies 5.4 Conclusions References Appendix A Supplementary Information of Study 1 A.1 Supplementary Methods 1 - behavioral A.2 Supplementary Methods 2 - fMRI A.3 Supplementary Results - behavioral A.4 Supplementary results - fMRI B Supplementary Information of Study 2 B.1 Computational fits B.2 Preprocessing of the functional imaging data B.3 Exclusion criteria for different analyses B.4 First level analysis of the functional imaging analysis B.5 Voxel-based morphometry B.6 Drinking Motives Questionnaire B.7 Model-free comparisons B.8 Association with time to relapse B.9 Number of detoxifications and model-based control: behavioral and neuroimaging analyses C Supplementary Information of Study 3 C.1 Differences between VBDM version used in this study compared to the VBDM version reported in Pooseh et al. (under review) C.2 Additional correlational analyses D Supplementary Information for additional analyses

Fluctuating ovarian hormones have been shown to affect decision-making processes in women. While emerging evidence suggests effects of endogenous ovarian hormones such as estradiol and progesterone on value-based decision-making in women, the impact of exogenous synthetic hormones, as in most oral contraceptives, is not clear. In a between-subjects design, we assessed measures of value-based decision-making in three groups of women aged 18 to 29 years, during (1) active oral contraceptive intake (N = 22), (2) the early follicular phase of the natural menstrual cycle (N = 20), and (3) the periovulatory phase of the natural menstrual cycle (N = 20). Estradiol, progesterone, testosterone, and sex-hormone binding globulin levels were assessed in all groups via blood samples. We used a test battery which measured different facets of value-based decision-making: delay discounting, risk-aversion, risk-seeking, and loss aversion. While hormonal levels did show the expected patterns for the three groups, there were no differences in value-based decision-making parameters. Consequently, Bayes factors showed conclusive evidence in support of the null hypothesis. We conclude that women on oral contraceptives show no differences in value-based decision-making compared to the early follicular and periovulatory natural menstrual cycle phases.

Cette thèse repose sur l’idée que les individus prennent des décisions en assignant des valeurs aux conséquences appétitives (récompenses) et aversives (punitions) associées aux différentes options proposées, puis comparent ces valeurs pour sélectionner l’une d’elles. Dans la plupart des cas, ces valeurs sont apprises par essais et erreurs. En neuroscience et neuro-économie, il a été proposé que le cerveau représente ces valeurs de manière abstraite dans une échelle de valeur commune indépendante de la nature des conséquences attendues. Cette hypothèse est soutenue par un grand nombre d’études qui se sont intéressées à l’évaluation et à l’apprentissage des récompenses. Ces études mettent en évidence une implication du mésencéphale, du striatum et du cortex préfrontal dans le traitement de récompense primaires, monétaires, morales ou sociales. Beaucoup moins de travaux se sont intéressés au traitement des punitions. L’objectif de cette thèse est donc de comparer les corrélats cérébraux des processus d’évaluation des stimuli appétitifs et aversifs. Dans un premier temps nous avons utilisé un paradigme d’apprentissage par conditionnement pavlovien pour comparer les régions cérébrales impliquées dans la représentation de l’erreur de prédiction au cours de l’apprentissage avec des récompenses et des punitions gustatives, visuelles et monétaires. Ensuite nous nous sommes intéressés aux régions cérébrales impliquées dans traitement de récompenses et punitions plus abstraites tel que l’approbation sociale ou les considérations morales, dans un contexte de prise de décision pro-sociale. Ces études nous ont permis de démontrer que l’erreur de prédiction et l’évaluation liées à des stimuli appétitifs et aversifs étaient, en partie, sous-tendus par des réseaux cérébraux communs. / There is a growing consensus in behavioral neuroscience and neuroeconomic that individuals make decisions by assigning values to different options and comparing them to make a choice. Most often, these values are acquired on the basis of trial and error learning. A long-held view is that the brain assigns values to the different goods using an abstract signal that is encoded in a common currency. Multiple studies have found evidence for such value signals in midbrain, striatum and prefrontal cortex during learning or decision making involving primary or secondary rewards. An important open question is whether aversive outcomes expectation and learning engage the same or different valuation networks. The goal of this thesis is thus to compare the brain network involved in appetitive and aversive stimuli valuation. First we used a pavlovian conditioning paradigm to compare the cerebral correlates of prediction error during learning with gustative, visual and monetary rewards and punishments. Second, we investigated the brain regions involved in moral and social rewards and punishments in prosocial decision making. Overall, we found that prediction error and valuation related to appetitive and aversive stimuli are processed in part by common brain networks.

Le sentiment d’agentivité est défini comme le sentiment de contrôler nos actions, et à travers elles, les évènements du monde extérieur. Cet ensemble phénoménologique dépend de notre capacité d’apprendre les contingences entre nos actions et leurs résultats, et un algorithme classique pour modéliser cela vient du domaine de l’apprentissage par renforcement. Dans cette thèse, nous avons utilisé l’approche de modélisation cognitive pour étudier l’interaction entre agentivité et apprentissage par renforcement. Tout d’abord, les participants réalisant une tâche d’apprentissage par renforcement tendent à avoir plus d’agentivité. Cet effet est logique, étant donné que l’apprentissage par renforcement consiste à associer une action volontaire et sa conséquence. Mais nous avons aussi découvert que l’agentivité influence l’apprentissage de deux manières. Le mode par défaut pour apprendre des contingences action-conséquence est que nos actions ont toujours un pouvoir causal. De plus, simplement choisir une action change l’apprentissage de sa conséquence. En conclusion, l’agentivité et l’apprentissage par renforcement, deux piliers de la psychologie humaine, sont fortement liés. Contrairement à des ordinateurs, les humains veulent être en contrôle, et faire les bons choix, ce qui biaise notre aquisition d’information. / Sense of agency or subjective control can be defined by the feeling that we control our actions, and through them effects in the outside world. This cluster of experiences depend on the ability to learn action-outcome contingencies and a more classical algorithm to model this originates in the field of human reinforcementlearning. In this PhD thesis, we used the cognitive modeling approach to investigate further the interaction between perceived control and reinforcement learning. First, we saw that participants undergoing a reinforcement-learning task experienced higher agency; this influence of reinforcement learning on agency comes as no surprise, because reinforcement learning relies on linking a voluntary action and its outcome. But our results also suggest that agency influences reinforcement learning in two ways. We found that people learn actionoutcome contingencies based on a default assumption: their actions make a difference to the world. Finally, we also found that the mere fact of choosing freely shapes the learning processes following that decision. Our general conclusion is that agency and reinforcement learning, two fundamental fields of human psychology, are deeply intertwined. Contrary to machines, humans do care about being in control, or about making the right choice, and this results in integrating information in a one-sided way.