Mobile devices equipped with features (e.g., camera, network connectivity and media player) are increasingly being used for different tasks such as web browsing, document reading and photography. While the portability of mobile devices makes them desirable for pervasive access to information, their small screen real-estate often imposes restrictions on the amount of information that can be displayed and manipulated on them. On the other hand, large displays have become commonplace in many outdoor as well as indoor environments. While they provide an efficient way of presenting and disseminating information, they provide little support for digital interactivity or physical accessibility. Researchers argue that mobile phones provide an efficient and portable way of interacting with large displays, and the latter can overcome the limitations of the small screens of mobile devices by providing a larger presentation and interaction space. However, distributing user interface (UI) elements across a mobile device and a large display can cause switching of visual attention and that may affect task performance. This thesis specifically explores how the switching of visual attention across a handheld mobile device and a vertical large display can affect a single user's task performance during mobile interaction with large displays. It introduces a taxonomy based on the factors associated with the visual arrangement of Multi Display User Interfaces (MDUIs) that can influence visual attention switching during interaction with MDUIs. It presents an empirical analysis of the effects of different distributions of input and output across mobile and large displays on the user's task performance, subjective workload and preference in the multiple-widget selection task, and in visual search tasks with maps, texts and photos. Experimental results show that the selection of multiple widgets replicated on the mobile device as well as on the large display, versus those shown only on the large display, is faster despite the cost of initial attention switching in the former. On the other hand, a hybrid UI configuration where the visual output is distributed across the mobile and large displays is the worst, or equivalent to the worst, configuration in all the visual search tasks. A mobile device-controlled large display configuration performs best in the map search task and equal to best (i.e., tied with a mobile-only configuration) in text- and photo-search tasks.

Previous research has found that programmers spend a significant fraction of their time navigating between different source code locations and that much of that time is spent returning to previously viewed code. Other work has identified the ability to juxtapose arbitrary pieces of code as cognitively important. However, modern IDEs have inherited a user interface design in which, usually, only one source file is displayed at a time, with the result that users must switch back and forth from one file to another. Taking advantage of the increasing availability of large displays, we propose a new interaction paradigm in which an IDE presents parts of multiple source files side by side, using the Mylyn degree-of-interest function to dynamically allocate screen space to them on the basis of degree-of-interest to the current development task. We demonstrate the feasibility of this paradigm with a prototype implementation built on the Eclipse IDE and note that it was used by the author over a period of months in the development of the prototype itself. Additionally, we present two case studies which quantify the potential reduction in navigation and demonstrate the simplicity of the approach and its ability to capture complete concerns on screen. These case studies suggest that the approach has the potential to reduce the time that programmers spend navigating by as much as 50%.

Previous research has found that programmers spend a significant fraction of their time navigating between different source code locations and that much of that time is spent returning to previously viewed code. Other work has identified the ability to juxtapose arbitrary pieces of code as cognitively important. However, modern IDEs have inherited a user interface design in which, usually, only one source file is displayed at a time, with the result that users must switch back and forth from one file to another. Taking advantage of the increasing availability of large displays, we propose a new interaction paradigm in which an IDE presents parts of multiple source files side by side, using the Mylyn degree-of-interest function to dynamically allocate screen space to them on the basis of degree-of-interest to the current development task. We demonstrate the feasibility of this paradigm with a prototype implementation built on the Eclipse IDE and note that it was used by the author over a period of months in the development of the prototype itself. Additionally, we present two case studies which quantify the potential reduction in navigation and demonstrate the simplicity of the approach and its ability to capture complete concerns on screen. These case studies suggest that the approach has the potential to reduce the time that programmers spend navigating by as much as 50%.

Doctor of Philosophy (PhD) / Large displays are everywhere, and have been shown to provide higher productivity gain and user satisfaction compared to traditional desktop monitors. The computer mouse remains the most common input tool for users to interact with these larger displays. Much effort has been made on making this interaction more natural and more intuitive for the user. The use of computer vision for this purpose has been well researched as it provides freedom and mobility to the user and allows them to interact at a distance. Interaction that relies on monocular computer vision, however, has not been well researched, particularly when used for depth information recovery. This thesis aims to investigate the feasibility of using monocular computer vision to allow bare-hand interaction with large display systems from a distance. By taking into account the location of the user and the interaction area available, a dynamic virtual touchscreen can be estimated between the display and the user. In the process, theories and techniques that make interaction with computer display as easy as pointing to real world objects is explored. Studies were conducted to investigate the way human point at objects naturally with their hand and to examine the inadequacy in existing pointing systems. Models that underpin the pointing strategy used in many of the previous interactive systems were formalized. A proof-of-concept prototype is built and evaluated from various user studies. Results from this thesis suggested that it is possible to allow natural user interaction with large displays using low-cost monocular computer vision. Furthermore, models developed and lessons learnt in this research can assist designers to develop more accurate and natural interactive systems that make use of human’s natural pointing behaviours.

Previous research has found that programmers spend a significant fraction of their time navigating between different source code locations and that much of that time is spent returning to previously viewed code. Other work has identified the ability to juxtapose arbitrary pieces of code as cognitively important. However, modern IDEs have inherited a user interface design in which, usually, only one source file is displayed at a time, with the result that users must switch back and forth from one file to another. Taking advantage of the increasing availability of large displays, we propose a new interaction paradigm in which an IDE presents parts of multiple source files side by side, using the Mylyn degree-of-interest function to dynamically allocate screen space to them on the basis of degree-of-interest to the current development task. We demonstrate the feasibility of this paradigm with a prototype implementation built on the Eclipse IDE and note that it was used by the author over a period of months in the development of the prototype itself. Additionally, we present two case studies which quantify the potential reduction in navigation and demonstrate the simplicity of the approach and its ability to capture complete concerns on screen. These case studies suggest that the approach has the potential to reduce the time that programmers spend navigating by as much as 50%. / Science, Faculty of / Computer Science, Department of / Graduate

Les données d'aujourd'hui deviennent de plus en plus complexes à cause de la forte croissance de leurs volumes ainsi que leur multidimensionnalité. Il devient donc nécessaire d'explorer des environnements d'affichage qui aillent au-delà du simple affichage de données offert par les moniteurs traditionnels et ce, afin de fournir une plus grande surface d'affichage ainsi que des techniques d'interaction plus performantes pour l'exploration de données. Les environnements correspondants à cette description sont les suivants : Les écrans large ; les environnements multi-écrans (EME) composés de plusieurs écrans hétérogènes spatialement distribués (moniteurs, smartphones, tablettes, table interactive ...) ; les environnements immersifs. Dans ce contexte, l'objectif de ces travaux de thèse est de concevoir et d'évaluer des solutions d'interaction originales, efficaces et adaptées à chacun des trois environnements cités précédemment. Une première contribution de nos travaux consiste en Split-focus : une interface de visualisation et d'interaction qui exploite les facilités offertes par les environnements multi-écrans dans la visualisation de données multidimensionnelles au travers d'une interface overview + multi-detail multi-écrans. Bien que plusieurs techniques d'interaction offrent plus d'une vue détaillée en simultané, le nombre optimal de vues détaillées n'a pas été étudié. Dans ce type d'interface, le nombre de vues détaillées influe grandement sur l'interaction : avoir une seule vue détaillée offre un grand espace d'affichage mais ne permet qu'une exploration séquentielle de la vue d'ensemble?; avoir plusieurs vues détaillées réduit l'espace d'affichage dans chaque vue mais permet une exploration parallèle de la vue d'ensemble. Ce travail explore le bénéfice de diviser la vue détaillée d'une interface overview + detail pour manipuler de larges graphes à travers une étude expérimentale utilisant la technique Split-focus. Split-focus est une interface overview + multi-détails permettant d'avoir une vue d'ensemble sur un grand écran et plusieurs vues détaillées (1,2 ou 4) sur une tablette. [...] / Today's ever-growing data is becoming increasingly complex due to its large volume and high dimensionality: it thus becomes crucial to explore interactive visualization environments that go beyond the traditional desktop in order to provide a larger display area and offer more efficient interaction techniques to manipulate the data. The main environments fitting the aforementioned description are: large displays, i.e. an assembly of displays amounting to a single space; Multi-display Environments (MDEs), i.e. a combination of heterogeneous displays (monitors, smartphones/tablets/wearables, interactive tabletops...) spatially distributed in the environment; and immersive environments, i.e. systems where everything can be used as a display surface, without imposing any bound between displays and immersing the user within the environment. The objective of our work is to design and experiment original and efficient interaction techniques well suited for each of the previously described environments. First, we focused on the interaction with large datasets on large displays. We specifically studied simultaneous interaction with multiple regions of interest of the displayed visualization. We implemented and evaluated an extension of the traditional overview+detail interface to tackle this problem: it consists of an overview+detail interface where the overview is displayed on a large screen and multiple detailed views are displayed on a tactile tablet. The interface allows the user to have up to four detailed views of the visualization at the same time. We studied its usefulness as well as the optimal number of detailed views that can be used efficiently. Second, we designed a novel touch-enabled device, TDome, to facilitate interactions in Multi- display environments. The device is composed of a dome-like base and provides up to 6 degrees of freedom, a touchscreen and a camera that can sense the environment. [...]

This research focuses on the emergence of 'digital walls' that can project images onto almost all or even the entirety of interior (and perhaps exterior) walls, and what implications this might have for how we arrange our rooms. It demonstrates the arrangement, i.e., the way that domestic products are arranged physically, of rooms changes in the domestic environment in a complex scenario when using large screens. Due to the fast-growing demand for large screens, this product could potentially be available to be used by people in their home environments; however, it does not yet exist in reality at this scale. Constructing large screens can be carried out using different production methods. Here, this concept is called the digital wall, a very thin wall-sized interactive screen. The characteristics of the digital wall will vary to be able to create different scenarios. One such scenario is a space in the home where the surface of the wall is covered with screens, which allows multiple possibilities to experience and interact with digital content.  In this research, the social gathering space of homes, nowadays called the living room, is considered as a highly relevant space for installing the digital wall. In this space, the conceptual framework outlines the basic elements of the research and demonstrates the relationships between people’s interactions with the digital wall and domestic products in the domestic environment. I show two examples from design history to understand how arrangement changes impact the home environment: the transformation of the parlor to the living room, and entry of the television into the living room. These two examples are focused on the place in the home where people gather for socializing. The discussion of these examples led to the elaboration of the relationships between the elements in the conceptual framework. I explored relevant design research methodologies to bring this future scenario into the present to understand the relationships between people and the digital wall. I applied research through design and the constructive design research approaches to frame the design research methodology. In this thesis, I set up seven series of design studies in two cluster groups: Supportive studies and Main studies. All of the design studies were conducted in the Design Research Lab, the actual space for carrying out the design experiments, prototyping the digital wall, and the setting of the experiments for user participation. The Lab was fully equipped with relevant technology and allowed me to use multiple methods to collect data while people were experiencing the design study sessions. The Lab was useful as a platform to understand user experiences, barriers for interactions as well as people's experiences in a simulated space of a domestic environment.  The main contribution of this research is to understand the forms of arrangement changes when people use the digital wall in homes. The research demonstrates two significant implications that are seen in two forms of arrangements: tangible arrangement and imperceptible arrangement. These findings are useful for both designers and users of the elements of domestic contexts and the relations that can be shaped by the presence of a digital wall in home environments. This understanding may provide design guidelines in future scenarios in which the digital wall is used in homes. The findings are also beneficial for designing the domestic environment, improving the arrangement of space, and raising the requirements for designing domestic products.