Analysts often work with large complex non-quantitative datasets in order to better understand concepts, themes, and other forms of insight contained within them. As defined by Pirolli and Card, this act of sensemaking is cognitively difficult, and is performed iteratively and repetitively through various stages of understanding. Immersive analytics has purported to assist with this process through putting users in virtual environments that allows them to sift through and explore data in three-dimensional interactive settings. Most previous research, however, has focused on quantitative data, where users are interacting with mostly numerical representations of data. We designed Immersive Space to Think, an immersive analytics approach to assist users perform the act of sensemaking with non-quantitative datasets, affording analysts the ability to manipulate data artifacts, annotate them, search through them, and present their findings. We performed several studies to understand and refine our approach and how it affects users sensemaking strategies. An exploratory virtual reality study found that users place documents in 2.5-dimensional structures, where we saw semicircular, environmental, and planar layouts. The environmental layout, in particular, used features of the environment as scaffolding for users' sensemaking process. In a study comparing levels of mixed reality as defined by Milgram-Kishino's Reality-Virtuality Continuum, we found that an augmented virtuality solution best fits users' preferences while still supporting external tools. Lastly, we explored how users deal with varying amounts of space and three-dimensional user interaction techniques in a comparative study comparing small virtual monitors, large virtual monitors, and a seated-version implementation of Immersive Space to Think. Our participants found IST best supported the task of sensemaking, with evidence that users leveraged spatial memory and utilized depth to denote additional meaning in the immersive condition. Overall, Immersive Space to Think affords an effective sensemaking three-dimensional space using 3D user interaction techniques that can leverage embodied cognition and spatial memory which aids the users understanding. / Doctor of Philosophy / Humans are constantly trying to make sense of the world around them. Whether they're a detective trying to understand what happened at a crime scene or a shopper trying to find the best office chair, people are consuming vast quantities of data to assist them with their choices. This process can be difficult, and people are often returning to various pieces of data repeatedly to remember why they are making the choice they decided upon. With the advent of cheap virtual reality products, researchers have pursued the technology as a way for people to better understand large sets of data. However, most mixed reality applications looking into this problem focus on numerical data, whereas a lot of the data people process is multimedia or text-based in nature. We designed and developed a mixed reality approach for analyzing this type of data called Immersive Space to Think. Our approach allows users to look at and move various documents around in a virtual environment, take notes or highlight those documents, search those documents, and create reports that summarize what they've learned. We also performed several studies to investigate and evolve our design. First, we ran a study in virtual reality to understand how users interact with documents using Immersive Space to Think. We found users arranging documents around themselves in a semicircular or flat plane pattern, or using various cues in the virtual environment as a way to organize the document set. Furthermore, we performed a study to understand user preferences with augmented and virtual reality. We found a mix of the two, also known as augmented virtuality, would best support user preferences and ability. Lastly, we ran two comparative studies to understand how three dimensional space and interaction affects user strategies. We ran a small user study looking at how a single student uses a desktop computer with a single display as well as immersive space to think to write essays. We found that they wrote essays with a better understanding of the source data with Immersive Space to Think than the desktop setup. We conducted a larger study where we compared a small virtual monitor simulating a traditional desktop screen, a large virtual monitor simulating a monitor 8 times the size of traditional desktop monitors, and immersive space to think. We found participants engaged with documents more in Immersive Space to Think, and used the space to denote importance for documents. Overall, Immersive Space to Think provides a compelling environment that assists users in understanding sets of documents.

Sensemaking is the way in which we understand the world around us. Pirolli and Card developed a sensemaking model related to intelligence analysis, which involves taking raw, unstructured data, analyzing it, and presenting a report of the findings. With lower-cost immersive technologies becoming more popular, new opportunities exist to leverage embodied and distributed cognition to better support sensemaking by providing vast, immersive space for creating meaningful schemas (organizational structures) during an analysis task. This work builds on prior work in immersive analytics on the concept of Immersive Space to Think (IST), which provides analysts with immersive space to physically navigate and use to organize information during a sensemaking task. In this work, we performed several studies that aimed to understand how IST supports sensemaking and how we can develop additional features to better aid analysts while they complete sensemaking in immersive analytics systems, focusing on non-quantitative data analysis. In a series of exploratory user studies, we aimed to understand how users' sensemaking process evolves during multiple session analyses, which identified how the participants refined their use of the immersive space into later stages of the sensemaking process. Another exploratory user study highlighted how professional analysts and novice users share many similarities in immersive analytic tool usage during sensemaking within IST. In addition to looking at multi-session analysis tasks, we also explored how sensemaking strategies change as users become more familiar with the immersive analytics tool usage in an exploratory study that utilized multiple analysis tasks completed over a series of three user study sessions. Lastly, we conducted a comparative user study to evaluate how the addition of new organizational features, clustering, and linking affect sensemaking within IST. Overall, our studies expanded the IST tool set and gathered an enhanced understanding of how immersive space is utilized during analysis tasks within IST. / Doctor of Philosophy / Sensemaking is a process we do in our daily lives. It is how we understand the world around us, make decisions, and complete complex analyses, like journalists writing stories or detectives solving cases. Sensemaking involves gathering information, making sense of it, developing hypotheses, and drawing conclusions, similar to writing a report. This work builds on prior work in Immersive Space to Think (IST), which is a concept of using immersive technologies (Virtual /Augmented Reality) to support sensemaking by providing vast 3D space for organizing the data used in a sensemaking task. Additionally, using these technologies to support sensemaking provides benefits such as increased space for analysis, increased engagement, and natural user interaction, which allow us to interact with information used during sensemaking tasks in new ways. In IST, users are able to move virtual documents around in the space around them to support their analysis process. In this work, we ran a study focused on multi-session analysis within IST, revealing how users refined their document placements over time while completing sensemaking tasks within IST. We also ran a study to understand how professional analysts' and novice users' analysis with IST differed in the IST tool usage. In another user study, we explored how users' strategies for sensemaking and document layouts changed as they became more familiar with the IST tool. Lastly, we conducted a comparative user study to evaluate how new features like clustering and linking affected analysis within IST. Overall, our work contributed to an enhanced understanding of how immersive space is utilized during analysis tasks within IST.

Due to increasing data complexity, researchers are struggling to visually explore data using traditional methods. Research has shown that a variety of analytics-related tasks can be enhanced using new emerging technologies (e.g., virtual reality [VR] and augmented reality [AR]). This paper provides a detailed design approach for developing an immersive virtual engine system for conducting exploratory and descriptive data analytics. This system is called the Immersive Virtual Exploratory Engine (IVEE), is a VR system that allows users to experience full intractability with its graphical elements and examines various datasets within the same immersive environment. Basic plots—such as histograms, line plots, and other exploratory analytics—can be created in both 2D and 3D to visualize datasets. Furthermore, the proposed module provides a data sub-setting system, entitled “Lasso,” as well as a visual element merge system that allows users to subset and merge data using natural interactions only. The system also allows for the simultaneous visualization of multiple representations, thus supporting decisions that require numerous plots. Furthermore, the system supports remote collaboration, allowing users from different locations to come together in a virtual space and work collaboratively as they would in a real-life setting.

Immersive analytics is an emerging field of data exploration and analysis in immersive environments. It is an active research area that explores human-centric approaches to data exploration and analysis based on the spatial arrangement and visualization of data elements in immersive 3D environments. The availability of immersive extended reality systems has increased tremendously recently, but it is still not as widely used as conventional 2D displays. In this dissertation, we described an immersive analysis system for spatiotemporal data and performed several user studies to measure the user performance in the developed system, and laid out design guidelines for an immersive analytics environment. In our first study, we compared the performance of users based on specific visual analytics tasks in an immersive environment and on a conventional 2D display. The approach was realized based on the coordinated multiple-views paradigm. We also designed an embodied interaction for the exploration of spatial time series data. The findings from the first user study showed that the developed system is more efficient in a real immersive environment than using it on a conventional 2D display. One of the important challenges we realized while designing an immersive analytics environment was to find the optimal placement and identification of various visual elements. In our second study, we explored the iterative design of the placement of visual elements and interaction with them based on frames of reference. Our iterative designs explored the impact of the visualization scale for three frames of reference and used the collected user feedback to compare the advantages and limitations of these three frames of reference. In our third study, we described an experiment that quantitatively and qualitatively investigated the use of sonification, i.e., conveying information through nonspeech audio, in an immersive environment that utilized empirical datasets obtained from a multi-dimensional geophysical system. We discovered that using event-based sonification in addition to the visual channel was extremely effective in identifying patterns and relationships in large, complex datasets. Our findings also imply that the inclusion of audio in an immersive analytics system may increase users’ level of confidence when performing analytics tasks like pattern recognition. We outlined the sound design principles for an immersive analytics environment using real-world geospace science datasets and assessed the benefits and drawbacks of using sonification in an immersive analytics setting. / Doctor of Philosophy / When it comes to exploring data, visualization is the norm. We make line charts, scatter plots, bar graphs, or heat maps to look for patterns in data using traditional desktop-based approaches. However, biologically humans are optimized to observe the world in three dimensions. This research is motivated by the idea that representing data in immersive 3D environments can provide a new perspective that may lead to the discovery of previously undetected data patterns. Experiencing the data in three dimensions, engaging multiple senses like sound and sight, and leveraging human embodiment, interaction capabilities, and sense of presence may lead to a unique understanding of the data that is not feasible using traditional visual analytics. In this research, we first compared the data analysis process in a mixed reality system, where real and virtual worlds co-exist, versus doing the same analytical tasks in a desktop-based environment. In our second study, we studied where different charts and data visualizations should be placed based on the scale of the environment, such as table-top versus room-sized. We studied the strengths and limitations of different scales based on the visual and interaction design of the developed system. In our third study, we used a real-world space science dataset to test the liabilities and advantages of using the immersive approach. We also used audio and explored what kinds of audio work for which analytical tasks and laid out design guidelines based on audio. Through this research, we studied how to do data analytics in emerging mixed reality environments and presented results and design guidelines for future developers, designers, and researchers in this field.

We propose the concept of spatial analytic interfaces (SAIs) as a tool for performing in-situ, everyday analytic tasks. Mobile computing is now ubiquitous and provides access to information at nearly any time or place. However, current mobile interfaces do not easily enable the type of sophisticated analytic tasks that are now well-supported by desktop computers. Conversely, desktop computers, with large available screen space to view multiple data visualizations, are not always available at the ideal time and place for a particular task. Spatial user interfaces, leveraging state-of-the-art miniature and wearable technologies, can potentially provide intuitive computer interfaces to deal with the complexity needed to support everyday analytic tasks. These interfaces can be implemented with versatile form factors that provide mobility for doing such taskwork in-situ, that is, at the ideal time and place. We explore the design of spatial analytic interfaces for in-situ analytic tasks, that leverage the benefits of an upcoming generation of light-weight, see-through, head-worn displays. We propose how such a platform can meet the five primary design requirements for personal visual analytics: mobility, integration, interpretation, multiple views and interactivity. We begin with a design framework for spatial analytic interfaces based on a survey of existing designs of spatial user interfaces. We then explore how to best meet these requirements through a series of design concepts, user studies and prototype implementations. Our result is a holistic exploration of the spatial analytic concept on a head-worn display platform. / October 2016

This research was to develop a real-time, modifiable, tangible interactive tabletop system for participatory urban design. The targeting user group was those stakeholders in urban design charrettes. Previous system solutions overlooked the importance of the modifiable tangible medium in the situation of reaching spatial-temporal consensus. These design issues impeded communication between the stakeholders and the professionals. Users of these systems had difficulties expressing ideas to professionals during the collaborative design process. Literature in evolving technology in the smart city context, collaborative urban design, embodied interaction, and depth-sensing was referred to guide the system design. Based on the review, this research identified the pivotal role of a shapeable and tangible medium in the system. The prototype system unified the modifiable, realistic model with its digital equivalent in urban analytics in real-time. By integrating tangible interaction, depth-sensing, and large touch screen tabletop, an intuitive, immersive decision-making interface for non-professional stakeholders could be created. During the system implementation, system elements centering ‘tangible interoperability’ were documented along the system pipeline. A heuristic evaluation, a method of usability inspection, was conducted to assess and to guide the future system design. The result was promising and inspiring. In the end, challenges and directions of system design were discussed. The contribution of this research included: discovering direction, centering tangibility, implementing a prototype, and documenting elements in each stage along the system pipeline of designing a modifiable tangible interactive tabletop system for the urban design charrette.