生活中許多決策情境是「不確定下的決策（decisions under uncertainty）」，只瞭解選項的結果，不知道結果發生的機率。人們會累積經驗，以學習到適當的決策。許多證據支持，自身會透過增強學習（reinforcement learning）機制學習，根據每次獲得的經驗，調整對於選項的期望，之後選擇期望最大的選項，幫助做出適當的決策。經驗可以透過自身決策或觀察他人決策所獲得，然而，過去較少研究探討觀察學習。因此，本研究欲探討決策中的觀察學習，並釐清同理心對於觀察學習的調節作用。實驗一中，改善過去了研究限制，量測膚電反應、學習速率與行為表現，讓參與者在自身學習、觀察他人與觀察電腦情境進行作業，並透過同理心問卷測量參與者的同理心特質。結果顯示，觀察學習在正向學習與負向學習不同，正向學習為趨向優勢選項，負向學習為避開劣勢選項。正向學習在三種學習情境中無任何差異，負向學習在觀察他人學習時，會受到同理心的調節作用。同理心分數越高，觀察他人的負向行為表現越好，觀察他人負向回饋的膚電反應越大。實驗一只透過問卷測量同理心，無法推論因果關係，因此實驗二直接操弄了不同的同理程度。回饋呈現的同時，呈現他人的情緒或中性臉孔圖片，以引發參與者的同理程度高或低。實驗中，量測回饋相關負波（Feedback-Related Negativity，FRN）、學習速率與行為表現。如同實驗一，只有負向學習受到同理程度不同的影響。同理程度高時，負向學習表現較好。FRN則顯示了同理程度與預期性的交互作用，同理程度低時，與過去研究一致，非預期FRN比預期FRN更加負向；同理程度高時，則無此預期性效果。雖然FRN無預期性差異，但依然能學習到符號機率，行為表現不受影響，推測可能有其他系統參與決策學習。綜上所述，本研究顯示，只有負向學習中，觀察學習會受到同理心的調節，同理心越高，行為表現越好。 / In daily life, we made many decisions under uncertainty. In each decision, we know only the outcome but no probabilities of the outcome. We have to accumulate the experience to learn adaptive decisions. Bunches of studies have shown that people may learn adaptive decisions by reinforcement learning. People modified the expectation for each option according to decision feedbacks, and, in the next time, chose the option with the maximum expectation. People can receive feedback from decisions making by self or others. However, fewer studies examined observational learning in decision making. Therefore, present research would clarify observational learning in decision making, and examine how empathy modulated observational learning. In experiment 1, skin conductance response, learning rate and behavioral performance were recorded and analyzed. Participants would learning decisions in different situations of self learning, observing others and observing computer. The questionnaire of empathy was also measured to examine its modulation in observational learning. The results showed that there were difference in positive learning and negative learning. Positive learning is to approach to the advantageous option, while negative learning is to avoid from the disadvantageous option. In positive learning, there were no difference among the three learning situations, but, in negative learning, empathy would modulate learning by observing others. The higher the empathy score was, the better the behavioral performance of negative learning was. Moreover, the skin conductance response when participants observing others’ negative feedback positively correlated with the empathy score. In experiment 2, the empathy level was manipulated by display pictures of others faces with feedback. Displaying the emotional faces or neutral faces would induce high or low empathy level for others, respectively. The feedback-related negativity (FRN), learning rate and behavioral performance were recorded and analyzed. Similar to experiment 1, only the negative learning was modulated by the empathy level. When participants were induced high empathy level, the behavioral performance was better. The results of FRN showed the interaction between empathy levels and expectancy of feedback. When participant’s empathy level was low, unexpected FRN was more negative than expected FRN. This result was consistent with previous studies. Nevertheless, when participant’s empathy level was high, there was no difference between unexpected FRN and expected FRN. Although FRN didn't show the effect of expectancy, participants could still learn the probabilities of each signs and made adaptive decisions. This result may result from other systems involved in observational learning. From the results of experiment 1 and 2, present research showed that, only in negative learning, observational learning was modulated by empathy, and the higher the empathy level was, the better the behavioral performance was.

本篇文章使用了增強學習與捲積深度學習結合的DQCN模型製作交易系統，希望藉由此交易系統能自行判斷是否買賣ETF，由於ETF屬於穩定性高且手續費高的衍生性金融商品，所以該系統不即時性的做買賣，採用每二十個開盤日進行一次買賣，並由這20個開盤日進行買賣的預測，希望該系統能最大化我們未來的報酬。 DQN是一種增強學習的模型，並在其中使用深度學習進行動作價值的預測，利用增強學習的自我更新動作價值的機制，再用深度學習強大的學習能力成就了人工智慧，並在其取得良好的成效。 / In this paper, we used DCQN model, which is combined with reinforcement learning and CNN to train a trading system and hope the trading system could judge whether buy or sell ETFs. Since ETFs is a derivative financial good with high stability and related fee, the system does not perform real-time trading and it performs every 20 trading day. The system predicts value of action based on data in the last 20 opening days to maximize our future rewards. DQN is a reinforcement learning model, using deep learning to predict value of actions in model. Combined with the RL's mechanism, which updates value of actions, and deep learning, which has a strong ability of learning, to finish an artificial intelligence. We got a perfect effect.

人們對於錯誤訊息處理經由自我覺察或外在回饋之管道，可藉由事件關聯電位分別測得ERN (error-related negativity) 及FRN (feedback-related negativity)。過去研究曾指出雙側作業(Flanker task)中錯誤所引發的ERN會受到以圖片呈現的短期情緒所調節，然而對於回饋誘發的FRN與個體情緒調節的關係則未曾被探討過。過去FRN的研究認為唯有當受試者所進行的作業為增強學習作業時，受試者對於回饋結果的預期狀態才能反映於FRN的反應強度。本研究利用兩個實驗分別採用非增強學習作業及增強學習作業，其中並以IAPS情緒圖片進行短期情緒的引發，在受試者於實驗中對其反應結果的不同預期狀態，探測受試者FRN受短期情緒調節的效果。 實驗一利用非增強學習作業，結果顯示FRN的強度可以反映受試者對於回饋結果的預期狀態，其中以非預期時FRN的強度為最大，預期時FRN的強度為最小；另外，正向情緒圖片對於FRN具有調節效果，正向情緒下FRN反應強度小於中性以及負向情緒下FRN反應強度。實驗二利用增強學習作業，前述的FRN強度反映受試者對回饋結果的預期效果，只有在實驗前半段的嘗試次中被觀測到，此效果未見於全部嘗試次納入分析；另外，實驗二中沒有觀察到情緒對於FRN的調節效果。 綜合而言，本研究發現受試者唯有持續處於學習的情形下，FRN才能反映受試者對於回饋結果的預期狀態，情緒對FRN的調節效果也僅於此情況下才能被觀測到。 / Error-related information in human can be processed via self-awareness and/or feedback given externally, which are measurable by the use of event-related potential (ERP) and termed error-related negativity (ERN) and feedback-related negativity (FRN) respectively. Previous studies showed that short-term affective stimuli would modulate the magnitude of ERN elicited by Flanker task. However, such modulation effect has not been tested on FRN. Furthermore, the magnitude of FRN is indicated to be related to the expectancy states toward feedback when the subject is undergoing a reinforcement learning task. Present study, thus, was designed to test the affective modulation effect on FRN in two separate tasks. In which, emotional pictures adopted from IAPS were used as the short-term affective stimuli, and different expectancy states in both non-reinforcement learning task (Experiment1) and reinforcement learning task (Experiment 2) were manipulated. In the results of Experiment 1, the magnitude of FRN was larger under the unexpected condition in comparing to the expected one. Modulation effect of short-term affective stimuli on FRN was obtained when positive emotion pictures were presented in non-reinforcement learning task, which FRN amplitude was significantly smaller in comparing to those measured after the presentation of neutral and negative pictures. In the results of Experiment 2, FRN elicited in the unexpected condition was only obtained from analyzing the dada collected in the first half of trails. Such effect was not confirmed when the data from all trials were analyzed. A lack of modulation effect of short-term affective stimuli on FRN appeared in Experiment 2. In conclusion, it is indicated that the expectancy depended FRN is most apparent when the subject is undergoing a continuous learning-demanded process. Meanwhile, short-term affective stimuli can modulate such FRN.

增強式學習（Reinforcement Learning）透過與環境不斷的互動來學習，以達到極大化每一期報酬的總和的目標，廣泛被運用於多期的決策過程。基於這些特性，增強式學習可以應用於建立需不斷動態調整投資組合配置比例的動態資產配置策略。 本研究應用Deep Q-Learning演算法建立動態資產配置策略，研究如何在每期不同的環境狀態之下，找出最佳的配置權重。採用2007年7月2日至2017年6月30日的美國中大型股的股票ETF及投資等級的債券ETF建立投資組合，以其日報酬率資料進行訓練，並與買進持有策略及固定比例投資策略比較績效，檢視深度增強式學習在動態資產配置適用性。 / Reinforcement learning learns by interacting with the environment continuously, in order to achieve the target of maximizing the sum of each return. It has been used to solve multi-period decision making problem broadly. Because of these characteristics, reinforcement learning can be applied to build the strategies of dynamic asset allocation which keep reallocating the mix of portfolio consistently. In this study, we apply deep Q-Learning algorithm to build the strategies of dynamic asset allocation. Studying how to find the optimal weights in the different environment. We use Large-Cap, Mid-Cap ETFs and investment-grade bond ETFs in the U.S. to build up the portfolio. We train the model with the data of daily return, and then we measure its performance by comparing with buy-and-hold and constant-mix strategy to check the fitness of deep Q-Learning.

現今解釋錯誤偵測系統及前扣帶皮質（ACC）的關係之理論主要為增強學習理論。增強學習理論認為個體會在行為後對於行為結果產生預期，並將該預期與實際結果進行比較，若實際結果較預期結果差則會活化ACC進而觀察到較大的FN（Feedback Negativity）振幅。近年來有學者提出奠基於增強學習理論的錯誤可能性理論，錯誤可能性理論則認為當個體在學習到行為與結果之間的關聯後，當接收到可能犯錯的訊息時便會活化ACC而引起較大的FN。本研究主要的目的為探討增強學習理論及錯誤可能性理論的適用性，其次為探討風險之因素是否能反映於FN上。由於兩理論對於風險情境中是否會觀察到FN有不同的預測，錯誤可能性理論預測會在高風險的情況下觀察到較大的FN；而增強學習理論則預測由於風險畫面並非回饋畫面，故風險不會影響FN。實驗一藉由探討風險與FN間之關係企圖提供兩理論初步之區分並提供風險研究的實驗證據，實驗一結果顯示FN確實會反映風險之因素。也點出增強學習理論用以解釋錯誤偵測系統之不完備之處。而實驗二則利用操弄回饋結果好壞及高低錯誤可能性以檢驗錯誤可能性對於錯誤偵測系統之必要性，實驗二結果顯示錯誤可能性為事件評估之因素之一。除此之外，實驗二亦提供FN反映懊悔之支持性證據。