Given the increasing competition in the software industry and the critical consequences of software errors, it has become important for companies to achieve high levels of software quality. While cost reduction and timeliness of projects continue to be important measures, software companies are placing increasing attention on identifying the user needs and better defining software quality from a customer perspective. Software quality goes beyond just correcting the defects that arise from any deviations from the functional requirements. System engineers also have to focus on a large number of quality requirements such as security, availability, reliability, maintainability, performance and temporal correctness requirements. The fulfillment of these run-time observable quality requirements is important for customer satisfaction and project success. Generating early forecasts of potential quality problems can have significant benefits to quality improvement. One approach to better software quality is to improve the overall development cycle in order to prevent the introduction of defects and improve run-time quality factors. Many methods and techniques are available which can be used to forecast quality of an ongoing project such as statistical models, opinion polls, survey methods etc. These methods have known strengths and weaknesses and accurate forecasting is still a major issue. This research utilized a novel approach using prediction markets, which has proved useful in a variety of situations. In a prediction market for software quality, individual estimates from diverse project stakeholders such as project managers, developers, testers, and users were collected at various points in time during the project. Analogous to the financial futures markets, a security (or contract) was defined that represents the quality requirements and various stakeholders traded the securities using the prevailing market price and their private information. The equilibrium market price represents the best aggregate of diverse opinions. Among many software quality factors, this research focused on predicting the software correctness. The goal of the study was to evaluate if a suitably designed prediction market would generate a more accurate estimate of software quality than a survey method which polls subjects. Data were collected using a live software project in three stages: viz., the requirements phase, an early release phase and a final release phase. The efficacy of the market was tested with results from prediction markets by (i) comparing the market outcomes to final project outcome, and (ii) by comparing market outcomes to results of opinion poll. Analysis of data suggests that predictions generated using the prediction market are significantly different from those generated using polls at early release and final release stages. The prediction market estimates were also closer to the actual probability estimates for quality compared to the polls. Overall, the results suggest that suitably designed prediction markets provide better forecasts of potential quality problems than polls.

對於現有文獻中討論的預測市場模型，嘗試加入風險與聲譽變數，觀察與分析其成效，並參考文獻中的代理人系統實驗方法，對論文中相關部分進行修正、設計並模擬之預測市場模型。 / In this research, we proposed two variables that could be incorporated with prediction markets: Reputation and Risk. Instead of attracting new players, The reputation system could stop losing bankrupted player, Player willing to help bankrupted player will gain reputation, and bankrupted player will lose reputation. Previous works suggest longshot bias is related to the risk-neutrality of players. Our approach is to experiment dierent risk distribution. We observe the impact of these variables in an agent-based model of prediction markets. We use zero-intelligence agents, where human qualities such as maximizing prot, learning or obeserving are missing. We further discuss the result, and the impact of risk and reputation.

In the modern world, making informed decisions requires obtaining and aggregating relevant information about events of interest. For many political, business, and entertainment events, the information of interest only exists as opinions, beliefs, and judgments of dispersed individuals, and we can only get a complete picture by putting the separate pieces of information together. Thus, an important first step towards decision making is motivating the individuals to reveal their private information and coalescing the separate pieces of information together. In this dissertation, I study three information elicitation and aggregation methods, prediction markets, peer prediction mechanisms, and adaptive polling, using both theoretical and applied approaches. These methods mainly differ by their assumptions on the participants' behavior, namely whether the participants possess noisy or perfect information and whether they strategically decide on what information to reveal. The first two methods, prediction markets and peer prediction mechanisms, assume that the participants are strategic and have perfect information. Their primary goal is to use carefully designed monetary rewards to incentivize the participants to truthfully reveal their private information. As a result, my studies of these methods focus on understanding to what extent are these methods incentive compatible in theory and in practice. The last method, adaptive polling, assumes that the participants are not strategic and have noisy information. In this case, our goal is to accurately and efficiently estimate the latent ground truth given the noisy information, and we aim to evaluate whether this goal can be achieved by using this method experimentally. I make four main contributions in this dissertation. First, I theoretically analyze how the participants' knowledge of one another's private information affects their strategic behavior when trading in a prediction market with a finite number of participants. Each participant may trade multiple times in the market, and hence may have an incentive to withhold or misreport his information in order to mislead other participants and capitalize on their mistakes. When the participants' private information is unconditionally independent, we show that the participants reveal their information as late as possible at any equilibrium, which is arguably the worse outcome for the purpose of information aggregation. We also provide insights on the equilibria of such prediction markets when the participants' private information is both conditionally and unconditionally dependent given the outcome of the event. Second, I theoretically analyze the participants' strategic behavior in a prediction market when a participant has outside incentives to manipulate the market probability. The presence of such outside incentives would seem to damage the information aggregation in the market. Surprisingly, when the existence of such incentives is certain and common knowledge, we show that there exist separating equilibria where all the participants' private information is revealed and fully aggregated into the market probability. Although there also exist pooling equilibria with information loss, we prove that certain separating equilibria are more desirable than many pooling equilibria because the separating equilibria satisfy domination based belief refinements, maximize the social welfare of the setting, or maximize either participant's total expected payoff. When the existence of the outside incentives is uncertain, trust cannot be established and the separating equilibria no longer exist. Third, I experimentally investigate participants' behavior towards the peer prediction mechanisms, which were proposed to elicit information without observable ground truth. While peer prediction mechanisms promise to elicit truthful information by rewarding participants with carefully constructed payments, they also admit uninformative equilibria where coordinating participants provide no useful information. We conduct the first controlled online experiment of the Jurca and Faltings peer prediction mechanism, engaging the participants in a multiplayer, real-time and repeated game. Using a hidden Markov model to capture players' strategies from their actions, our results show that participants successfully coordinate on uninformative equilibria and the truthful equilibrium is not focal, even when some uninformative equilibria do not exist or result in lower payoffs. In contrast, most players are consistently truthful in the absence of peer prediction, suggesting that these mechanisms may be harmful when truthful reporting has similar cost to strategic behavior. Finally, I design and experimentally evaluate an adaptive polling method for aggregating small pieces of imprecise information together to produce an accurate estimate of a latent ground truth. In designing this method, we make two main contributions: (1) Our method aggregates the participants' noisy information by using a theoretical model to account for the noise in the participants' contributed information. (2) Our method uses an active learning inspired approach to adaptively choose the query for each participant. We apply this method to the problem of ranking a set of alternatives, each of which is characterized by a latent strength parameter. At each step, adaptive polling collects the result of a pairwise comparison, estimates the strength parameters from the pairwise comparison data, and adaptively chooses the next pairwise comparison question to maximize expected information gain. Our MTurk experiment shows that our adaptive polling method can effectively incorporate noisy information and improve the estimate accuracy over time. Compared to a baseline method, which chooses a random pairwise comparison question at each step, our adaptive method can generate more accurate estimates with less cost. / Engineering and Applied Sciences

A prediction market is a special type of market which offers trades for securities associated with future states that are observable at a certain time in the future. Recently, prediction markets have shown the promise of being an abstract framework for designing distributed, scalable and self-incentivized machine learning systems which could then apply to large scale problems. However, existing designs of prediction markets are far from achieving such machine learning goal, due to (1) the limited belief modelling power and also (2) an inadequate understanding of the market dynamics. This work is thus motivated by improving and extending current prediction market design in both aspects. This research is focused on potential based prediction markets, that is, prediction markets that are administered by potential (or cost function) based market makers (PMM). To improve the market’s modelling power, we first propose the partially-observable potential based market maker (PoPMM), which generalizes the standard PMM such that it allows securities to be defined and evaluated on future states that are only partially-observable, while also maintaining the key properties of the standard PMM. Next, we complete and extend the theory of generalized exponential families (GEFs), and use GEFs to free the belief models encoded in the PMM/PoPMM from always being in exponential families. To have a better understanding of the market dynamics and its link to model learning, we discuss the market equilibrium and convergence in two main settings: convergence driven by traders, and convergence driven by the market maker. In the former case, we show that a market-wise objective will emerge from the traders’ personal objectives and will be optimized through traders’ selfish behaviours in trading. We then draw intimate links between the convergence result to popular algorithms in convex optimization and machine learning. In the latter case, we augment the PMM with an extra belief model and a bid-ask spread, and model the market dynamics as an optimal control problem. This convergence result requires no specific models on traders, and is suitable for understanding the markets involving less controllable traders.

Prediction markets provide a promising approach for future event prediction. Most existing prediction market approaches are based on auction mechanisms. Despite their theoretical appeal and success in various application settings, these mechanisms suffer from several major drawbacks. First, opinions from experts and amateurs are treated equally. Second, continuous attention from participants is assumed. Third, such mechanisms are subject to various forms of market manipulation. To alleviate these limitations, we propose to employ the classic fixed odds betting as an alternative prediction market mechanism. We build a structural model based on a belief-decision framework as the event probability estimator. This belief-decision framework models bettors' beliefs with mixed beta distributions and bettors' decisions with prospect theory. A maximum likelihood approach is applied to estimate the model parameters. We conducted experiments on three real-world betting datasets to evaluate our proposed approach. Experimental results show that fixed odds betting based prediction outperforms the reduced form models based on odds and betting results, and achieves a comparable performance with auction-based prediction markets. The results suggest the possibility of employing fixed odds betting as a prediction market in a variety of application contexts where the assumptions made by auction-based approaches do not hold.

Leveraging the knowledge of an organization is an ongoing challenge that has given rise to the field of knowledge management. Yet, despite spending enormous sums of organizational resources on Information Technology (IT) systems, executives recognize there is much more knowledge to harvest. Prediction markets are emerging as one tool to help extract this tacit knowledge and make it operational. Yet, prediction markets, like other markets, are subject to pathologies (e.g., bubbles and crashes) which compromise their accuracy and may discourage organizational use. The techniques of experimental economics were used to study the characteristics of prediction markets. Empirical data was gathered from an on-line asynchronous prediction market. Participants allocated tickets based on private information and, depending on the market type, public information indicative of how prior participants had allocated their tickets. The experimental design featured three levels of feedback (no-feedback, percentages of total allocated tickets and frequency of total allocated tickets) presented to the participants. The research supported the hypothesis that information assimilation in feedback markets is composed of two mechanisms - information collection and aggregation. These are defined as: Collection - The compilation of dispersed information - individuals using their own private information make judgments and act accordingly in the market. Aggregation - The market's judgment on the implications of this gathered information - an inductive process. This effect comes from participants integrating public information with their private information in their decision process. Information collection was studied in isolation in no feedback markets and the hypothesis that markets outperform the average of their participants was supported. The hypothesis that with the addition of feedback, the process of aggregation would be present was also supported. Aggregation was shown to create agreement in markets (as measured by entropy) and drive market results closer to correct values (the known probabilities). However, the research also supported the hypothesis that aggregation can lead to information mirages, creating a market bubble. The research showed that the presence and type of feedback can be used to modulate market performance. Adding feedback, or more informative feedback, increased the market's precision at the expense of accuracy. The research supported the hypotheses that these changes were due to the inductive aggregation process which creates agreement (increasing precision), but also occasionally generates information mirages (which reduces accuracy). The way individual participants use information to make allocations was characterized. In feedback markets the fit of participants' responses to various decision models demonstrated great variety. The decision models ranged from little use of information (e.g., MaxiMin), use of only private information (e.g., allocation in proportion to probabilities), use of only public information (e.g., allocating in proportion to public distributions) and integration of public and private information. Analysis of all feedback market responses using multivariate regression also supported the hypothesis that public and private information were being integrated by some participants. The subtle information integration results are in contrast to the distinct differences seen in markets with varying levels of feedback. This illustrates that the differences in market performance with feedback are an emergent phenomenon (i.e., one that could not be predicted by analyzing the behavior of individuals in different market situations). The results of this study have increased our collective knowledge of market operation and have revealed methods that organizations can use in the construction and analysis of prediction markets. In some situations markets without feedback may be a preferred option. The research supports the hypothesis that information aggregation in feedback markets can be simultaneously responsible for beneficial information processing as well as harmful information mirage induced bubbles. In fact, a market subject to mirage prone data resembles a Prisoner's Dilemma where individual rationality results in collective irrationality.

「預測市場」以未來事件為交易標的，透過網路平台彙整即時資訊，運用價格來判斷未來事件的發展，此研究方法同時具有「適當獎懲」與「連續修正」兩項特性。 本研究以文獻分析途徑探究預測市場在不同交易機制下的運作方式與市場價格決定過程，並據此分析不同交易機制之間的差異與影響預測市場運作的因素；除此之外，本研究另以個案研究途徑來探討「連續雙向拍賣」與「市場計分法則」兩個交易機制在價格準確度、市場流動性、價格炒作、參與誘因與莊家風險之間的差異。 本研究發現：若交易機制是連續雙向拍賣，則「0-100型」合約價格的預測準確度較高；若交易機制是市場計分法則，則「落點預測型」合約價格的預測準確度較高。連續雙向拍賣機制具有市場流動性不足的問題；市場計分法則機制面臨莊家風險的危機且不適用於市場競爭度高的環境；而上述兩種交易機制皆會出現價格炒作的現象。 / “Prediction market” is a research method based on immediate information collecting and organizing on the internet platform. With future events as the object of transaction, variations of the price of each transaction thus immediately provide the prediction of the development of future events. Therefore, this method has two properties including “appropriate incentives” and “continuous correction”. In this study, document analysis is first conducted to review the operation modes of different trading mechanisms for prediction markets and the process of price making. Accordingly, differences between trading mechanisms and the factors that affect the operation of prediction market will also be analyzed. Furthermore, comparisons of the price accuracy, market liquidity, price speculation, incentives and maker risks between "continuous double auction" and "market scoring rule" are discussed in case study. The findings of this study: if the trading mechanism is “continuous double auction”, the price accuracy of “winner-take-all” contract is higher; if the trading mechanism is “market scoring rules”, the price accuracy of “index” contract is higher. There exists insufficient market liquidity in “continuous double auction;” while in “market scoring rule,” there exists maker risks and it is hard to be applied in highly competitive market. The phenomenon of price speculation exits in both trading mechanisms.

金融市場常常受到政治、經濟與社會環境等因素所影響，所得到價格為眾多變數交互作用的結果，包含了許多雜訊。本文引進一套數據處理方法「整體經驗模態分解」(Ensemble Empirical Mode Decomposition，EEMD)來分析「期貨市場」以及「預測市場」。第一個實證利用EEMD處理台股期貨，分析對台股指數的解釋能力，並同時與原始台股期貨預測台股指數，比較預測結果；第二個實證利用EEMD來分析預測市場，判別是否能有效的消除雜訊，準確預測選舉結果。 第一個實證結果發現，EEMD能有效地過濾期貨市場的雜訊，另外，在最後到期日前十二天或者是前九天，以週期為6.5日經EEMD處理的台股期貨對台股指數的預測較原始台股期貨預測準確；第二個實證結果指出，直接利用EEMD處理預測市場得到的長期趨勢「剩餘訊號」(Residue)來預測選舉並無優於原始預測市場，主因為預測市場參與者不只在乎長期趨勢，亦在乎短期事件的衝擊，故直接利用剩餘訊號預測選舉結果會有所失真，而將剩餘訊號由低頻率之「本質模態函數」(Intrinsic Modes Function，IMF)合併至週期為6日與12日的IMF，得到了EEMD週趨勢價格，分成選前一天和選前十天的資料並與原始預測市場以及民調預測做比較，從不同的準則來看，發現以EEMD週趨勢價格來做選舉預測，準確度較原始預測市場與民調預測的結果更好。根據中選會2012年初選前對選罷法做成的解釋，未來事件交易所在選前十日亦須停止交易，我們可將EEMD運用在日後的選舉預測，把預測市場的合約價格以EEMD處理，應可提高選舉預測的準確度。 / The financial markets are usually affected by political, economic and social environment factors, and thus the volatilities of asset prices in these markets are subject to a lot of noises and shocks. To filter out noises and quantify shocks, this paper applies a data processing method, Ensemble Empirical Mode Decomposition (EEMD), and demonstrates its improved prediction to the futures and election prediction markets. While the first empirical application shows that the EEMD effectively filters out the noises in the futures market, the second one indicates that the Taiwanese election prediction using EEMD “residue” is not as accurate as that by original data from the prediction market. The reason why the residue cannot serve as a good predictor is that the market participants consider not only the long-term trend, but also shocks, especially those right before the elections. We then attempt to predict the election outcomes by the week trend series processed by EEMD. The prediction by the week EEMD trend series turns out to be more accurate than that by the poll and original prediction market. Based on this study, we can apply the EEMD to the next election prediction and improve its accuracy.