Research in interactive information retrieval (IR) usually relies on lab user studies or online ones. A key concern of these studies is the generalizability and reproducibility of the results, especially when the studies involved only a limited number of participants. The interactive IR community, however, does not have a commonly agreed guideline regarding how many participants should recruit. We study this fundamental research protocol issue by examining the generalizability and reproducibility of results with respect to a different number of participants using simulation-based approaches. Specifically, we collect a relatively large number of participants' observations for a representative interactive IR experiment setting from online user studies using crowdsourcing. We sample smaller numbers of participants' results from the collected observations to simulate the results of online user studies with a smaller scale. We empirically analyze the patterns of generalizability and reproducibility regarding different dependent variables and draw conclusions related to the optimal number of participants. Our study contributes to interactive information retrieval research by 1) establishing a methodology for evaluating the generalizability and reproducibility of results, and 2) providing guidelines regarding the optimal number of participants for search engine user studies. / Master of Science / In the domain of Information Retrieval, researchers or scientists usually require human participants to interact, test and evaluate a novel system, which is usually called user studies. However, researchers usually perform these studies with small sample size, some of them recruited fewer than 20 participants, which casts doubt on the generalizability and reproducibility of these studies. Generalizability means how reliable the results of relatively small sample size in an experimental setting can be generalized to the outcomes of a larger population. Reproducibility means whether the results from two groups with the same amount of sample size are consistent with each other. In order to examine the generalizability and reproducibility of online user studies in interactive information retrieval systems, we conducted an online user study with large sample size. We reproduced a well-recognized lab user study from Kelly et al. (2015) in an online environment. We established a simulation-based methodology for evaluating the generalizability and reproducibility of the results and then provided guidelines regarding the optimal number of participants for search engine user studies.

This item is not the definitive copy. Please use the following citation when referencing this material: Dillon and Vaughan (1997) It's the journey and the destination: Shape and the emergent property of genre in digital documents. New Review of Multimedia and Hypermedia, 3, 91-106. Introduction: To anyone versed in the literature on hypermedia, it is clear that the last 10 yearsâ worth of research on usability since Conklinâ s (1987) seminal article has largely been ignored by web designers. Surfing web sites even casually will likely expose a user to screens of badly formatted text, superfluous graphics, mixed fonts, unreadable color combinations, and dangling or dead links. While the issue of knowledge transfer between research disciplines and design practice is fraught with problems and is a fascinating topic in and of itself (see e.g., Klein and Eason, 1993), this is not the focus of the present paper. Instead we wish to extend work that started with the birth of hypertext systems and continues to demand attention in these days of free-for-all web design: the evaluation of user behaviour in electronic space. Specifically, this paper will extend the analysis of â user navigationâ to the evaluation of user behaviour in web environments.

This study reports an experiment that examines whether librarians provide equitable virtual reference services to diverse user groups. The relative absence of social cues in the virtual environment may mean greater equality of services though at the same time greater inequalities may arise as librarians can become less self-aware online. Findings indicate that the quality of service librarians provide to African Americans and Arabs is lower than the quality of service they provide to Caucasian, Hispanic, Asian, and Jewish students. This study adds to the knowledge of subjective bias in the virtual environment by specifying those that are discriminated against online, identifying the kinds of discriminatory actions of virtual reference librarians, and identifying the type of queries that more frequently result in unbiased service.

Many disciplines spend considerable resources studying behavior. Tools range from pen-and-paper observation to biometric sensing. A tool's appropriateness depends on the goal and justification of the study, the observable context and feature set of target behaviors, the observers' resources, and the subjects' tolerance to intrusiveness. We present two systems: Viz-A-Vis and Tableau Machine. Viz-A-Vis is an analytical tool appropriate for onsite, continuous, wide-coverage and long-term capture, and for objective, contextual, and detailed analysis of the physical actions of subjects who consent to overhead video observation. Tableau Machine is a creative artifact for the home. It is a long-lasting, continuous, interactive, and abstract Art installation that captures overhead video and visualizes activity to open opportunities for creative interpretation. We focus on overhead video observation because it affords a near one-to-one correspondence between pixels and floor plan locations, naturally framing the activity in its spatial context. Viz-A-Vis is an information visualization interface that renders and manipulates computer vision abstractions. It visualizes the hidden structure of behavior in its spatiotemporal context. We demonstrate the practicality of this approach through two user studies. In the first user study, we show an important search performance boost when compared against standard video playback and against the video cube. Furthermore, we determine a unanimous user choice for overviewing and searching with Viz-A-Vis. In the second study, a domain expert evaluation, we validate a number of real discoveries of insightful environmental behavior patterns by a group of senior architects using Viz-A-Vis. Furthermore, we determine clear influences of Viz-A-Vis over the resulting architectural designs in the study. Tableau Machine is a sensing, interpreting, and painting artificial intelligence. It is an Art installation with a model of perception and personality that continuously and enduringly engages its co-occupants in the home, creating an aura of presence. It perceives the environment through overhead cameras, interprets its perceptions with computational models of behavior, maps its interpretations to generative abstract visual compositions, and renders its compositions through paintings. We validate the goal of opening a space for creative interpretation through a study that included three long-term deployments in real family homes. / <p>QC 20160405</p>

"We report the results of a series of crowd-sourced user studies in the formal-methods domain. Specifically, we explore the efficacy of the notion of "minimal counterexample" -- or more colloquially, "minimal bug report" -- when reasoning about logical specifications. Our results here suggest that minimal counterexamples are beneficial some specific cases, and harmful in others. Furthermore, our analysis leads to refined hypotheses about the role of minimal counterexamples that can be further evaluated in future studies. User-based evaluation has little precedent in the formal methods community. Therefore, as a further contribution, we discuss and analyze our research methodology, and offer guidelines for future user studies in formal methods research. "

This study explores human-robot interaction during a 16-hour high-fidelity Urban Search and Rescue (USAR) disaster response drill with teleoperated robots. Situation awareness and team interaction were examined using communication analysis. Operators (n=5) sought assistance from team members to compensate for difficulties building or maintaining situation awareness. Operator-team member communication focused on relating what was seen through the robot's eye view with prior knowledge and planning search strategies. Results suggest operators need a new cognitive mental model to filter and comprehend data provided by the robot, and that robot-assisted search is a team task rather than an individual one. USAR technical search teams need a new shared mental model of robot-assisted search in order to coordinate activities effectively.

<p>The main purpose of this essay is to investigate professional artists experience of the need and use of information in their creative work. I have collected the material through qualitative interviews with four professional artist working with different materials and techniques. One of the results of the study is that the artists first of all used information as a source of inspiration in their creative work. They gathered their information from various scources, for example; pictures, exhibitions, visual arts, books and novells, depending on what project they are working with. Other important information scources were colleges, especially for the technical information need. The artists also searched for information in a wide range of subjects and showed interest in areas that are not considerable art-related.</p>

<p>This thesis addresses some important characteristics of human visual perception and their implications for three-dimensional (3D) information visualization. The effort can be divided into two parts. First, findings from vision science are explored and validated. As a starting point, the compilation of perceptually motivated evidence about what constitutes an effective and efficient method for mapping of data is undertaken. Second, the knowledge obtained is used to create candidate visualizations and to demonstrate the predictive power of the findings.</p><p>Results indicate a general difficulty to convey metric, i.e. quantitative, information in 3D visualizations. Structure as defined by Euclidean geometry is not perceived with accuracy and information encoded by such distinctions are misunderstood or overlooked. On the other hand, qualitative properties as defined by affine geometry are salient are perceived with accuracy (paper I). These findings are also characteristic of two-dimensional (2D) visualizations when these need to be rapidly examined (paper II). </p><p>A novel method (3D surface glyphs) for abstract multivariate data sets was developed to investigate the possible merit of encoding information by qualitative distinctions, (paper III). Evaluations showed that the information conveyed was successfully utilized and that these types of glyph have great potential. The study also illustrated the predictive power of the earlier findings. These issues were further demonstrated when investigating how 3D perspective displays are unaffected by distortions in data when the patterns displayed were defined by affine properties (paper IV). In addition, a new metric for measuring the efficiency of visualizations is presented (paper III). </p><p>It is concluded that as long as visualizations are specified by qualitative properties, they could most probably be effectively and efficiently used. The need for user studies to determine if, when and how to choose a certain visualization technique for a given task is thereby significantly reduced.</p>

This thesis addresses some important characteristics of human visual perception and their implications for three-dimensional (3D) information visualization. The effort can be divided into two parts. First, findings from vision science are explored and validated. As a starting point, the compilation of perceptually motivated evidence about what constitutes an effective and efficient method for mapping of data is undertaken. Second, the knowledge obtained is used to create candidate visualizations and to demonstrate the predictive power of the findings. Results indicate a general difficulty to convey metric, i.e. quantitative, information in 3D visualizations. Structure as defined by Euclidean geometry is not perceived with accuracy and information encoded by such distinctions are misunderstood or overlooked. On the other hand, qualitative properties as defined by affine geometry are salient are perceived with accuracy (paper I). These findings are also characteristic of two-dimensional (2D) visualizations when these need to be rapidly examined (paper II). A novel method (3D surface glyphs) for abstract multivariate data sets was developed to investigate the possible merit of encoding information by qualitative distinctions, (paper III). Evaluations showed that the information conveyed was successfully utilized and that these types of glyph have great potential. The study also illustrated the predictive power of the earlier findings. These issues were further demonstrated when investigating how 3D perspective displays are unaffected by distortions in data when the patterns displayed were defined by affine properties (paper IV). In addition, a new metric for measuring the efficiency of visualizations is presented (paper III). It is concluded that as long as visualizations are specified by qualitative properties, they could most probably be effectively and efficiently used. The need for user studies to determine if, when and how to choose a certain visualization technique for a given task is thereby significantly reduced.

Relevance assessing is an important part of information retrieval (IR) evaluation in addition to being something that all users of IR systems must do as part of their search for relevant documents. In this thesis, we present a user study conducted to understand the relevance judging behaviour of assessors when the prevalence of relevant documents in a set of documents to be judged is varied. In our user study, we collected judgements of participants on document sets of three different prevalence levels. The prevalence levels that we used were low (0.1), balanced (0.5) and high (0.9). We found that participants who judged documents at the 0.9 level made the most mistakes, and participants who judged documents at the 0.5 level made the least mistakes. We did not find a statistically significant difference in judging quality between 0.1 and 0.5 prevalence levels.