In many applications one is faced with the problem of inferring some functional relation between input and output variables from given data. Consider, for instance, the task of email spam filtering where one seeks to find a model which automatically assigns new, previously unseen emails to class spam or non-spam. Building such a predictive model based on observed training inputs (e.g., emails) with corresponding outputs (e.g., spam labels) is a major goal of machine learning. Many learning methods assume that these training data are governed by the same distribution as the test data which the predictive model will be exposed to at application time. That assumption is violated when the test data are generated in response to the presence of a predictive model. This becomes apparent, for instance, in the above example of email spam filtering. Here, email service providers employ spam filters and spam senders engineer campaign templates such as to achieve a high rate of successful deliveries despite any filters. Most of the existing work casts such situations as learning robust models which are unsusceptible against small changes of the data generation process. The models are constructed under the worst-case assumption that these changes are performed such to produce the highest possible adverse effect on the performance of the predictive model. However, this approach is not capable to realistically model the true dependency between the model-building process and the process of generating future data. We therefore establish the concept of prediction games: We model the interaction between a learner, who builds the predictive model, and a data generator, who controls the process of data generation, as an one-shot game. The game-theoretic framework enables us to explicitly model the players' interests, their possible actions, their level of knowledge about each other, and the order at which they decide for an action. We model the players' interests as minimizing their own cost function which both depend on both players' actions. The learner's action is to choose the model parameters and the data generator's action is to perturbate the training data which reflects the modification of the data generation process with respect to the past data. We extensively study three instances of prediction games which differ regarding the order in which the players decide for their action. We first assume that both player choose their actions simultaneously, that is, without the knowledge of their opponent's decision. We identify conditions under which this Nash prediction game has a meaningful solution, that is, a unique Nash equilibrium, and derive algorithms that find the equilibrial prediction model. As a second case, we consider a data generator who is potentially fully informed about the move of the learner. This setting establishes a Stackelberg competition. We derive a relaxed optimization criterion to determine the solution of this game and show that this Stackelberg prediction game generalizes existing prediction models. Finally, we study the setting where the learner observes the data generator's action, that is, the (unlabeled) test data, before building the predictive model. As the test data and the training data may be governed by differing probability distributions, this scenario reduces to learning under covariate shift. We derive a new integrated as well as a two-stage method to account for this data set shift. In case studies on email spam filtering we empirically explore properties of all derived models as well as several existing baseline methods. We show that spam filters resulting from the Nash prediction game as well as the Stackelberg prediction game in the majority of cases outperform other existing baseline methods. / Eine der Aufgabenstellungen des Maschinellen Lernens ist die Konstruktion von Vorhersagemodellen basierend auf gegebenen Trainingsdaten. Ein solches Modell beschreibt den Zusammenhang zwischen einem Eingabedatum, wie beispielsweise einer E-Mail, und einer Zielgröße; zum Beispiel, ob die E-Mail durch den Empfänger als erwünscht oder unerwünscht empfunden wird. Dabei ist entscheidend, dass ein gelerntes Vorhersagemodell auch die Zielgrößen zuvor unbeobachteter Testdaten korrekt vorhersagt. Die Mehrzahl existierender Lernverfahren wurde unter der Annahme entwickelt, dass Trainings- und Testdaten derselben Wahrscheinlichkeitsverteilung unterliegen. Insbesondere in Fällen in welchen zukünftige Daten von der Wahl des Vorhersagemodells abhängen, ist diese Annahme jedoch verletzt. Ein Beispiel hierfür ist das automatische Filtern von Spam-E-Mails durch E-Mail-Anbieter. Diese konstruieren Spam-Filter basierend auf zuvor empfangenen E-Mails. Die Spam-Sender verändern daraufhin den Inhalt und die Gestaltung der zukünftigen Spam-E-Mails mit dem Ziel, dass diese durch die Filter möglichst nicht erkannt werden. Bisherige Arbeiten zu diesem Thema beschränken sich auf das Lernen robuster Vorhersagemodelle welche unempfindlich gegenüber geringen Veränderungen des datengenerierenden Prozesses sind. Die Modelle werden dabei unter der Worst-Case-Annahme konstruiert, dass diese Veränderungen einen maximal negativen Effekt auf die Vorhersagequalität des Modells haben. Diese Modellierung beschreibt die tatsächliche Wechselwirkung zwischen der Modellbildung und der Generierung zukünftiger Daten nur ungenügend. Aus diesem Grund führen wir in dieser Arbeit das Konzept der Prädiktionsspiele ein. Die Modellbildung wird dabei als mathematisches Spiel zwischen einer lernenden und einer datengenerierenden Instanz beschrieben. Die spieltheoretische Modellierung ermöglicht es uns, die Interaktion der beiden Parteien exakt zu beschreiben. Dies umfasst die jeweils verfolgten Ziele, ihre Handlungsmöglichkeiten, ihr Wissen übereinander und die zeitliche Reihenfolge, in der sie agieren. Insbesondere die Reihenfolge der Spielzüge hat einen entscheidenden Einfluss auf die spieltheoretisch optimale Lösung. Wir betrachten zunächst den Fall gleichzeitig agierender Spieler, in welchem sowohl der Lerner als auch der Datengenerierer keine Kenntnis über die Aktion des jeweils anderen Spielers haben. Wir leiten hinreichende Bedingungen her, unter welchen dieses Spiel eine Lösung in Form eines eindeutigen Nash-Gleichgewichts besitzt. Im Anschluss diskutieren wir zwei verschiedene Verfahren zur effizienten Berechnung dieses Gleichgewichts. Als zweites betrachten wir den Fall eines Stackelberg-Duopols. In diesem Prädiktionsspiel wählt der Lerner zunächst das Vorhersagemodell, woraufhin der Datengenerierer in voller Kenntnis des Modells reagiert. Wir leiten ein relaxiertes Optimierungsproblem zur Bestimmung des Stackelberg-Gleichgewichts her und stellen ein mögliches Lösungsverfahren vor. Darüber hinaus diskutieren wir, inwieweit das Stackelberg-Modell bestehende robuste Lernverfahren verallgemeinert. Abschließend untersuchen wir einen Lerner, der auf die Aktion des Datengenerierers, d.h. der Wahl der Testdaten, reagiert. In diesem Fall sind die Testdaten dem Lerner zum Zeitpunkt der Modellbildung bekannt und können in den Lernprozess einfließen. Allerdings unterliegen die Trainings- und Testdaten nicht notwendigerweise der gleichen Verteilung. Wir leiten daher ein neues integriertes sowie ein zweistufiges Lernverfahren her, welche diese Verteilungsverschiebung bei der Modellbildung berücksichtigen. In mehreren Fallstudien zur Klassifikation von Spam-E-Mails untersuchen wir alle hergeleiteten, sowie existierende Verfahren empirisch. Wir zeigen, dass die hergeleiteten spieltheoretisch-motivierten Lernverfahren in Summe signifikant bessere Spam-Filter erzeugen als alle betrachteten Referenzverfahren.

<p dir="ltr">The rise of digital media has allowed for unprecedented access to information. In particular, people are able to form beliefs based on information sources that span the full spectrum of reputation, information quality, and motivated biases. Such access is a double-edged sword because “with great power, comes great responsibility” (“Spider-Man”, 2002). Heterogeneity in information quality may be due to a variety of factors, and it is often up to the consumer to consider quality signals when evaluating the quality of information. My research explores this complicated process, and contributes to the understanding of how people demand and utilize information in different environments. I do so over three chapters. The first studies how people respond to signals of information quality in a sequential prediction game. In the second chapter, biased incentives are introduced in a prediction game experiment to test how intrinsic and extrinsic biases affect demand and utilization of information. The third chapter contains a survey in which subjects report their valuations of an X account that varies on political affiliation, occupation credentials, and number of followers.</p><p dir="ltr">My first chapter focuses on how subjects respond to signals of information quality. In it, subjects predict which of two urns was randomly chosen in each of 30 rounds. They observe a private ball drawn from the selected urn each round to help them make their prediction. The color of the ball signals the urn it came from. The subjects then sequentially broadcast their belief about which urn was selected for the session without revealing the color of the observed ball. Future subjects can use the previous broadcasts to infer additional information that may help them accurately predict the urn.</p><p dir="ltr">In the control, subjects exhibit very low utilization of previous predictions when informing their own behavior. While consistent with prior research, behaving in such a manner is suboptimal. To experiment on the malleability of subjects’ beliefs about the rationality of others, I implement two novel treatments. In the first, the subjects’ prediction order in the last 15 rounds is determined by their accrued earnings in the first 15 rounds, with highest earners predicting first. The prediction order is similarly determined in the second treatment, except a quiz on conditional updating ability is used. Subjects who score the highest on the quiz predict first. In both cases, the sorting mechanism is explained to the subjects.</p><p dir="ltr">Sorting on earnings yields a modest increase in valuations of previous subjects’ predictions. A much more significant increase is observed when sorting on ability. Additionally, the subjects who make the fewest irrational predictions (ones against the color of the ball when they do not have additional information to suggest otherwise) are the ones who score the best in the ability sort. Placing them at the beginning of rounds increases the entire round’s average earnings.</p><p dir="ltr">My second chapter uses a similar environment to study the role that bias plays in demanding and utilizing information. In it, participants predict which of two states (red or blue) each of 30 rounds was assigned. To aid them, participants observe two predictions from ‘experts,’ who are informed by a private signal with a known precision. Participants can bid to receive additional information about the state from two sources: a private signal and another independent expert’s prediction. Both sources’ precision is known. This method is the first of its kind, and allows for direct comparison between information types. The bid results are revealed once this process is complete. Participants then predict the state.</p><p dir="ltr">Two innovative treatments are implemented to implement bias into the basic environment exogenously. In the first, participants receive a small bonus each time they predict the state is blue. In the second, experts receive the same bonus each time they predict the state is blue instead of the participants. Surprisingly, participants value the private signal and additional expert’s prediction similarly, except when the experts are biased. This is a departure from most research using similar environments, which assume that some sub-optimal behavior can be attributed to mistrust in others’ ability to understand the environment. That assumption may warrant further and more careful evaluation. The most striking valuation behavior is when participants are biased. Their bids are higher when their existing information set already favors their bias, relative to when it is against it. Doing so is antithetical to the rational equilibrium and inconsistent with prior research on confirmation bias.</p><p dir="ltr">Participants generally utilize information obtained from a successful bid at a lower rate when it is against the initial experts than with it. No difference is detected between information sources. This is expected, albeit inconsistent with rational decision-making. One exception is noted. When participants are biased, they use the newly obtained information at a much higher rate when it is consistent with their bias than against it. Doing so is at odds with bidding behavior, as it implies participants bid more to receive information that they utilize less. Participants generally do a much better job of rationalizing and responding to the experts’ bias than their own in the experiment.</p><p dir="ltr">My third chapter is motivated partly by the findings in my first two chapters, using a more contextualized setting. In it, subjects are presented with a series of X account versions. The versions vary on political affiliation, occupation credentials, and number of followers. Subjects are asked to rate how much they would value information from each account version. Subjects value account versions with an unrevealed political party affiliation more than their analogs which report a party affiliation, regardless of the party or the subject’s beliefs.</p><p dir="ltr">A partisan penalty is uniformly implemented. Additionally, credentials are insufficient to overcome bias concerns. The penalty assessed to an account version aligning with a party is similar when the version has high credentials versus when it does not. Followers are also a valuable resource, regardless of political affiliation or credential levels. The marginal value that followers provide is similar for all account versions, meaning that even relative experts in a field should seek validation if they want to be valued by others.</p><p dir="ltr">Previous research would expect subjects to value versions more when they are congruent with their own beliefs, so these findings are surprising. Two groups are identified as the most likely to deviate and value same-typed account versions more: subjects who believe echo chambers are good and subjects who are concerned they have believed fake news in the past. The former group does not require a significant number of followers to highly value a politically congruent account version. The latter value politically unaffiliated accounts even more, but are more skeptical of opposition account versions and are even more sensitive to the number of followers they have.</p><p dir="ltr">These three chapters explore new avenues for researching how biases and expertise are evaluated and responded to. People are generally much better at considering the potential biases that others have than rationalizing their own biases. I also find good news in an era of heightened concern about eroding trust in experts. In each case, subjects respond to signals of expertise, and demonstrate efforts to exploit the information that experts provide.</p>