The success of smartphones and tablets clearly shows that the fusion of input and output within one device can lead to a more direct and natural interaction. While a bigger part of previous research was devoted to the development of techniques for such touch-sensitive displays, this dissertation goes beyond the limitations of an interactive surface and extends the interaction to the physical space above a digital table by means of handheld spatially aware displays. By incorporating their spatial position and orientation, those displays add a further major input channel to the interaction. Even though this idea is not entirely new, the potential of using spatially aware displays (Tangible Displays) above a digital tabletop has rarely been used and requires systematic examination. In pursuit of lessening this gap, this dissertation makes three major contributions: (1) The conceptual framework has been developed as a guide for the design of Tangible Display applications. It offers a systematic description and analysis of the design space under investigation and its basic interaction principles. This includes a detailed overview of the general system components and underlying types of input as well as a categorization of common interaction and usage patterns. Based on that, a classification of four common types of information spaces is provided along with a set of novel techniques for their spatial exploration in midair above a tabletop. On an empirical level, the framework is supported by two comprehensive studies that investigate key aspects of spatial interaction. (2) To facilitate the rapid prototyping of interactive Tangible Display applications, a unifying technological framework has been designed and implemented that integrates the necessary sensor and display hardware and provides simple access to it through an easy-to-use API. Along with a modular architectural design, the API does not only encapsulate the complexity of the underlying input and output technologies, but also allows for their seamless substitution by alternative approaches. (3) On a practical level, the conceptual and technological framework have been validated by four comprehensive interactive systems. Those systems served as a testbed for the iterative development and formative assessment of various novel interaction techniques tailored to address basic tasks in common fields of application. The gathered insights helped refine the conceptual and technological framework and are a valuable starting point for the development of future systems.:Abstract iii Zusammenfassung v Acknowledgements vii Publications ix Supervised Student Theses xiii Acronyms xv Contents xix 1 Introduction 1 1.1 Goals of this Thesis 1.1.1 Research Challenges 1.1.2 Research Objectives 1.2 Scope of this Thesis 1.3 Methodological Approach 1.4 Contributions and Thesis Outline PART I: Conceptual Framework 2 Research Background 2.1 General Research Context 2.1.1 Post-WIMP & Reality-Based Interaction 2.1.2 Ubiquitous Computing 2.1.3 Augmented Reality and Environments 2.2 Interactive Surfaces 2.2.1 Advanced Form Factors 2.2.2 Interaction Beyond the Surface 2.2.3 Multi-display Environments 2.3 Tangible User Interfaces 2.3.1 Basic TUI Genres 2.3.2 Contributions and Qualities of TUI 2.4 Spatially Aware Displays 2.4.1 Potential Benefits 2.4.2 Spatially Aware Displays in Multi-Display Environments 2.4.3 A TUI Genre of its Own:Tangible Displays 2.5 Summary 3 Studying Spatial Input-based Zoom & Pan on Handheld Displays 3.1 Goals and Scope of the Study 3.1.1 Factors of Influence 3.1.2 Hypotheses 3.1.3 Scope of the Study 3.2 Design Rationale 3.2.1 Mapping the Physical to the Virtual World 3.2.2 Clutching and Relative Mode 3.2.3 Zooming and Panning 3.2.4 Responsiveness of the Prototype 3.3 Method 3.3.1 Study Design 3.3.2 Participants 3.3.3 Apparatus 3.3.4 Scenario and Task Design 3.3.5 Procedure 3.4 Results 3.4.1 Statistical Methodology & Collected Performance Data 3.4.2 Analysis of Completion Times 3.4.3 Analysis of Discrete Actions 3.4.4 Utilized Motor Space (Spatial Condition Only) 3.4.5 User Feedback & Fatigue 3.5 Discussion 3.5.1 Verification of Hypotheses 3.5.2 Further Observations 3.5.3 Explaining the Effects 3.5.4 Limitations 3.6 Future Generations of Mobile Displays 3.6.1 Device-Intrinsic Spatial Tracking 3.6.2 A Built-in Tactile Clutch 3.7 Summary 4 Design Space & Interaction Framework 4.1 Design Dimensions 4.1.1 Principle Setup & System Components 4.1.2 Basic Types of Input 4.1.3 Spatial Zones 4.2 Interaction Vocabulary 4.2.1 Vocabulary Based on Spatial Input 4.2.2 Vocabulary Based on Head Input 4.2.3 Vocabulary Based on Surface Input 4.2.4 Vocabulary Inspired by the Representation Aspect 4.3 Topologies for Representing Virtual Spaces 4.3.1 3D Volumes 4.3.2 Zoom Pyramids 4.3.3 Multi-layer Stacks 4.4 Classes of Explorable Information Spaces 4.4.1 Volumetric Information Spaces 4.4.2 Zoomable Information Spaces 4.4.3 Layered Information Spaces 4.4.4 Temporal Information Spaces 4.5 Summary 5 Studying Multi-layer Interaction Above a Tabletop 5.1 Goals and Scope of the Study 5.1.1 Basic Interaction Tasks 5.1.2 Previous Evaluations 5.1.3 Scope of the Study 5.2 Method 5.2.1 Participants 5.2.2 Study Design & Tasks 5.2.3 Procedure 5.2.4 Apparatus 5.3 Results 5.3.1 Collected Performance Data & Statistical Methodology 5.3.2 Basic Analysis Concerning the Three Interaction Tasks 5.3.3 Further Analysis Regarding Interaction Zones 5.3.4 Questionnaires & User Preferences 5.4 Discussion 5.4.1 Layer Thicknesses & Accuracy 5.4.2 Physical Interaction Space & Number of Layers 5.4.3 Design Recommendations & Further Observations 5.5 Summary PART II: Technological Framework 6 Technological Background 101 6.1 Basic Display Approaches 6.1.1 Projective Displays 6.1.2 Active Displays 6.2 Tracking Technologies for Handheld Displays 6.2.1 Infrared Marker Systems 6.2.2 Depth Sensor Systems 6.3 Technologies for Sensing Surface Input 6.3.1 Sensing Touch Input 6.3.2 Digital Pens and Paper 6.4 Summary 7 A Tangible Display Toolkit for Research Labs 7.1 Toolkit Architecture 7.1.1 History and General Overview 7.1.2 Lab and Mobile Setup 7.2 Toolkit Subsystems 7.2.1 Projection Subsystem 7.2.2 Spatial Input Subsystem 7.2.3 Surface Input Subsystem 7.3 Toolkit Interfaces 7.3.1 Inter-process Communication 7.3.2 Servers & UI Tools 7.3.3 Application Programming Interface 7.4 Summary 8 Towards a Tangible Display Ecosystem for Everyone 8.1 Motivation and Vision 8.1.1 Envisioned Hardware Setup 8.1.2 Proactive Cooperation Among Devices 8.2 Revision of the Previous Lab Setup 8.3 Case Study: Tracking via a Low-cost Depth Sensor 8.3.1 Implemented Tracking Algorithm 8.3.2 Evaluation 8.3.3 Areas of Improvement 8.4 Summary PART III: Tangible Display Systems 9 Tangible Lenses for Multimedia Information Spaces 9.1 Case Studies 9.1.1 Volume Slicer 9.1.2 Picture Zoomer 9.1.3 Layer Explorer 9.1.4 Video Browser 9.2 Evaluation 9.2.1 Study Design 9.2.2 Findings 9.3 Improved Navigation Techniques 9.3.1 Navigating through Information Layers 9.3.2 Navigational Aids 9.4 Annotating the Space Above the Tabletop 9.4.1 Creation of Annotations 9.4.2 Guided Exploration of Annotations 9.5 Summary 10 Tangible Views for Information Visualization 10.1 Background and Motivation 10.1.1 Conventional Interactive Visualization 10.1.2 Towards More Direct Interaction in InfoVis 10.1.3 Narrowing the Gap 10.2 The Tangible Views Concept 10.3 Case Studies 10.3.1 Graph Visualization 10.3.2 Scatter Plot 10.3.3 Parallel Coordinates Plot 10.3.4 Matrix Visualization 10.3.5 Space-Time-Cube Visualization 10.4 Initial User Experience & Discussion 10.4.1 Observations 10.4.2 Limitations 10.5 Potential Future Directions 10.5.1 Technology Gap 10.5.2 Integration Gap 10.5.3 Guidelines Gap 10.6 Summary 11 Tangible Palettes for Graphical Applications 11.1 Background and Motivation 11.2 The Tangible Palettes Concept 11.2.1 Spatial Work Zones 11.2.2 Previous Techniques Revisited 11.2.3 Allocating GUI Palettes to Tangible Displays 11.3 Interactive Prototype 11.3.1 Basic Functionality: Drawing and Document Navigation 11.3.2 Inter-Display Transfer of Palettes 11.3.3 Temporary Fade-out of Tool Palettes 11.3.4 Quick Access to Tool Palettes via Spatial Work Zones 11.3.5 Handling of Stacked Graphics Layers 11.4 Initial User Experience & Discussion 11.4.1 General Impression and Limitations 11.4.2 Document Navigation with Handheld Displays 11.4.3 Tool Organization with Spatial Work Zones 11.5 Summary 12 Tangible Windows for 3D Virtual Reality 193 12.1 Background and Motivation 12.1.1 Basic 3D Display Approaches 12.1.2 Seminal 3D User Interfaces 12.2 TheTangibleWindowsConcept 12.2.1 Windows into Virtuality 12.2.2 Head-coupled Perspectives 12.2.3 Tangible Windows Above a Digital Workbench 12.3 Interaction Techniques 12.3.1 Global Viewpoint Control on the Tabletop 12.3.2 Scene Exploration 12.3.3 Object Selection 12.3.4 Object Manipulation 12.3.5 Object Inspection 12.3.6 Global Scene Navigation on the Tabletop 12.4 Application Scenarios & Case Studies 12.4.1 Virtual Sandbox 12.4.2 Interior Designer 12.4.3 Medical Visualization 12.5 Initial User Experience & Discussion 12.5.1 Limitations 12.5.2 Precision and Constraints 12.5.3 More Permanent Representations 12.5.4 Head-coupled Perspectives and Head Input 12.6 Summary 13 Conclusion 13.1 Summary of Contributions 13.1.1 Major Contributions 13.1.2 Minor Contributions 13.2 Critical Reflection 13.2.1 General Limitations due to the Dissertation Scope 13.2.2 Limitations of the Techniques 13.2.3 Limitations of the Studies 13.3 Directions for Future Work 13.3.1 Adaptation to other Settings and Domains 13.3.2 Further Development of the Techniques 13.3.3 Current Developments 13.4 Closing Remarks A Appendix A.1 Materials for Chapter3 (Zoom & Pan Study) A.1.1 List of the 128 Zoom and Pan Tasks A.1.2 Usability Questionnaires A.2 Questionnaires for Chapter 5 (Multi-layer Stack Study) A.3 Materials for Section 9.2 (Evaluation of Tangible Lenses) A.3.1 Scratchpad for Study Leader A.3.2 Usability Questionnaires Bibliography List of Figures List of Tables / Der Erfolg von Smartphones und Tablets hat deutlich gezeigt, dass die Verschmelzung von Ein- und Ausgabe im selben Gerät zu einer direkteren und als natürlicher empfundenen Interaktion führen kann. Während sich ein Großteil bisheriger Forschung der Entwicklung von Touchtechniken auf derartigen berührungsempfindlichen Displays widmet, löst sich diese Dissertation von den Beschränkungen der Interaktion auf Oberflächen und erweitert diese auf den physischen Raum oberhalb eines digitalen Tisches mittels handgehaltener, lagebewusster Displays. Durch die Einbeziehung der räumlichen Position und Orientierung solcher „Tangible Displays” steht ein vielversprechender, zusätzlicher Eingabekanal zur Verfügung. Wenngleich diese Idee nicht vollständig neu ist, wurden die vielfältigen Möglichkeiten, die sich durch ihren Einsatz über einem digitalen Tisch ergeben, bisher wenig genutzt. Im Bestreben diese Lücke zu verringern, leistet diese Dissertation drei wesentliche Beträge: (1) Das Konzeptuelle Rahmenwerk wurde als Leitfaden für den Entwurf von Tangible Display- Anwendungen entwickelt. Es bietet eine systematische Beschreibung und Analyse des zu untersuchenden Entwurfsraums und seiner grundlegenden Interaktionsprinzipien. Neben einer detaillierten Übersicht aller Systemkomponenten und Eingabearten beinhaltet dies vor allem eine Kategorisierung von typischen Interaktions- und Nutzungsmustern. Darauf basierend wird ein neuartiger vereinheitlichender Ansatz zur räumlichen Interaktion mit verschiedenen gängigen Klassen von Informationsräumen über einem Tabletop vorgestellt. Auf empirischer Ebene wird das Konzeptuelle Rahmenwerk durch zwei umfangreiche Studien gestützt, in denen Kernaspekte der räumlichen Interaktion mit handgehaltenen Displays untersucht wurden. (2) Um die Entwicklung von interaktiven Anwendungen zu ermöglichen, wurde ein Technisches Rahmenwerk entworfen und umgesetzt, das die Sensor- und Displayhardware zusammenfasst und einfachen Zugriff darauf mittels eines API bietet. Im Zusammenspiel mit der modularen Software-Architektur kapselt das API nicht nur die Komplexität der verwendeten Ein- und Ausgabetechnologien, sondern ermöglicht auch deren nahtlosen Austausch durch alternative Lösungsansätze. (3) Die Tauglichkeit des Konzeptuellen und des Technischen Rahmenwerkes wird durch vier umfangreiche interaktive Systeme demonstriert. Diese Systeme dienten als Testumgebung für die iterative Entwicklung und formative Bewertung einer Reihe von neuartigen Interaktionstechniken, die gängige Basisaufgaben in verschiedenen Anwendungsbereichen adressieren. Die dabei gewonnenen Erkenntnisse halfen, das Konzeptuelle und das Technische Rahmenwerk zu verfeinern, welche einen wertvollen Ausgangspunkt für die Entwicklung von zukünftigen interaktiven Tangible Display-Systemen bilden.:Abstract iii Zusammenfassung v Acknowledgements vii Publications ix Supervised Student Theses xiii Acronyms xv Contents xix 1 Introduction 1 1.1 Goals of this Thesis 1.1.1 Research Challenges 1.1.2 Research Objectives 1.2 Scope of this Thesis 1.3 Methodological Approach 1.4 Contributions and Thesis Outline PART I: Conceptual Framework 2 Research Background 2.1 General Research Context 2.1.1 Post-WIMP & Reality-Based Interaction 2.1.2 Ubiquitous Computing 2.1.3 Augmented Reality and Environments 2.2 Interactive Surfaces 2.2.1 Advanced Form Factors 2.2.2 Interaction Beyond the Surface 2.2.3 Multi-display Environments 2.3 Tangible User Interfaces 2.3.1 Basic TUI Genres 2.3.2 Contributions and Qualities of TUI 2.4 Spatially Aware Displays 2.4.1 Potential Benefits 2.4.2 Spatially Aware Displays in Multi-Display Environments 2.4.3 A TUI Genre of its Own:Tangible Displays 2.5 Summary 3 Studying Spatial Input-based Zoom & Pan on Handheld Displays 3.1 Goals and Scope of the Study 3.1.1 Factors of Influence 3.1.2 Hypotheses 3.1.3 Scope of the Study 3.2 Design Rationale 3.2.1 Mapping the Physical to the Virtual World 3.2.2 Clutching and Relative Mode 3.2.3 Zooming and Panning 3.2.4 Responsiveness of the Prototype 3.3 Method 3.3.1 Study Design 3.3.2 Participants 3.3.3 Apparatus 3.3.4 Scenario and Task Design 3.3.5 Procedure 3.4 Results 3.4.1 Statistical Methodology & Collected Performance Data 3.4.2 Analysis of Completion Times 3.4.3 Analysis of Discrete Actions 3.4.4 Utilized Motor Space (Spatial Condition Only) 3.4.5 User Feedback & Fatigue 3.5 Discussion 3.5.1 Verification of Hypotheses 3.5.2 Further Observations 3.5.3 Explaining the Effects 3.5.4 Limitations 3.6 Future Generations of Mobile Displays 3.6.1 Device-Intrinsic Spatial Tracking 3.6.2 A Built-in Tactile Clutch 3.7 Summary 4 Design Space & Interaction Framework 4.1 Design Dimensions 4.1.1 Principle Setup & System Components 4.1.2 Basic Types of Input 4.1.3 Spatial Zones 4.2 Interaction Vocabulary 4.2.1 Vocabulary Based on Spatial Input 4.2.2 Vocabulary Based on Head Input 4.2.3 Vocabulary Based on Surface Input 4.2.4 Vocabulary Inspired by the Representation Aspect 4.3 Topologies for Representing Virtual Spaces 4.3.1 3D Volumes 4.3.2 Zoom Pyramids 4.3.3 Multi-layer Stacks 4.4 Classes of Explorable Information Spaces 4.4.1 Volumetric Information Spaces 4.4.2 Zoomable Information Spaces 4.4.3 Layered Information Spaces 4.4.4 Temporal Information Spaces 4.5 Summary 5 Studying Multi-layer Interaction Above a Tabletop 5.1 Goals and Scope of the Study 5.1.1 Basic Interaction Tasks 5.1.2 Previous Evaluations 5.1.3 Scope of the Study 5.2 Method 5.2.1 Participants 5.2.2 Study Design & Tasks 5.2.3 Procedure 5.2.4 Apparatus 5.3 Results 5.3.1 Collected Performance Data & Statistical Methodology 5.3.2 Basic Analysis Concerning the Three Interaction Tasks 5.3.3 Further Analysis Regarding Interaction Zones 5.3.4 Questionnaires & User Preferences 5.4 Discussion 5.4.1 Layer Thicknesses & Accuracy 5.4.2 Physical Interaction Space & Number of Layers 5.4.3 Design Recommendations & Further Observations 5.5 Summary PART II: Technological Framework 6 Technological Background 101 6.1 Basic Display Approaches 6.1.1 Projective Displays 6.1.2 Active Displays 6.2 Tracking Technologies for Handheld Displays 6.2.1 Infrared Marker Systems 6.2.2 Depth Sensor Systems 6.3 Technologies for Sensing Surface Input 6.3.1 Sensing Touch Input 6.3.2 Digital Pens and Paper 6.4 Summary 7 A Tangible Display Toolkit for Research Labs 7.1 Toolkit Architecture 7.1.1 History and General Overview 7.1.2 Lab and Mobile Setup 7.2 Toolkit Subsystems 7.2.1 Projection Subsystem 7.2.2 Spatial Input Subsystem 7.2.3 Surface Input Subsystem 7.3 Toolkit Interfaces 7.3.1 Inter-process Communication 7.3.2 Servers & UI Tools 7.3.3 Application Programming Interface 7.4 Summary 8 Towards a Tangible Display Ecosystem for Everyone 8.1 Motivation and Vision 8.1.1 Envisioned Hardware Setup 8.1.2 Proactive Cooperation Among Devices 8.2 Revision of the Previous Lab Setup 8.3 Case Study: Tracking via a Low-cost Depth Sensor 8.3.1 Implemented Tracking Algorithm 8.3.2 Evaluation 8.3.3 Areas of Improvement 8.4 Summary PART III: Tangible Display Systems 9 Tangible Lenses for Multimedia Information Spaces 9.1 Case Studies 9.1.1 Volume Slicer 9.1.2 Picture Zoomer 9.1.3 Layer Explorer 9.1.4 Video Browser 9.2 Evaluation 9.2.1 Study Design 9.2.2 Findings 9.3 Improved Navigation Techniques 9.3.1 Navigating through Information Layers 9.3.2 Navigational Aids 9.4 Annotating the Space Above the Tabletop 9.4.1 Creation of Annotations 9.4.2 Guided Exploration of Annotations 9.5 Summary 10 Tangible Views for Information Visualization 10.1 Background and Motivation 10.1.1 Conventional Interactive Visualization 10.1.2 Towards More Direct Interaction in InfoVis 10.1.3 Narrowing the Gap 10.2 The Tangible Views Concept 10.3 Case Studies 10.3.1 Graph Visualization 10.3.2 Scatter Plot 10.3.3 Parallel Coordinates Plot 10.3.4 Matrix Visualization 10.3.5 Space-Time-Cube Visualization 10.4 Initial User Experience & Discussion 10.4.1 Observations 10.4.2 Limitations 10.5 Potential Future Directions 10.5.1 Technology Gap 10.5.2 Integration Gap 10.5.3 Guidelines Gap 10.6 Summary 11 Tangible Palettes for Graphical Applications 11.1 Background and Motivation 11.2 The Tangible Palettes Concept 11.2.1 Spatial Work Zones 11.2.2 Previous Techniques Revisited 11.2.3 Allocating GUI Palettes to Tangible Displays 11.3 Interactive Prototype 11.3.1 Basic Functionality: Drawing and Document Navigation 11.3.2 Inter-Display Transfer of Palettes 11.3.3 Temporary Fade-out of Tool Palettes 11.3.4 Quick Access to Tool Palettes via Spatial Work Zones 11.3.5 Handling of Stacked Graphics Layers 11.4 Initial User Experience & Discussion 11.4.1 General Impression and Limitations 11.4.2 Document Navigation with Handheld Displays 11.4.3 Tool Organization with Spatial Work Zones 11.5 Summary 12 Tangible Windows for 3D Virtual Reality 193 12.1 Background and Motivation 12.1.1 Basic 3D Display Approaches 12.1.2 Seminal 3D User Interfaces 12.2 TheTangibleWindowsConcept 12.2.1 Windows into Virtuality 12.2.2 Head-coupled Perspectives 12.2.3 Tangible Windows Above a Digital Workbench 12.3 Interaction Techniques 12.3.1 Global Viewpoint Control on the Tabletop 12.3.2 Scene Exploration 12.3.3 Object Selection 12.3.4 Object Manipulation 12.3.5 Object Inspection 12.3.6 Global Scene Navigation on the Tabletop 12.4 Application Scenarios & Case Studies 12.4.1 Virtual Sandbox 12.4.2 Interior Designer 12.4.3 Medical Visualization 12.5 Initial User Experience & Discussion 12.5.1 Limitations 12.5.2 Precision and Constraints 12.5.3 More Permanent Representations 12.5.4 Head-coupled Perspectives and Head Input 12.6 Summary 13 Conclusion 13.1 Summary of Contributions 13.1.1 Major Contributions 13.1.2 Minor Contributions 13.2 Critical Reflection 13.2.1 General Limitations due to the Dissertation Scope 13.2.2 Limitations of the Techniques 13.2.3 Limitations of the Studies 13.3 Directions for Future Work 13.3.1 Adaptation to other Settings and Domains 13.3.2 Further Development of the Techniques 13.3.3 Current Developments 13.4 Closing Remarks A Appendix A.1 Materials for Chapter3 (Zoom & Pan Study) A.1.1 List of the 128 Zoom and Pan Tasks A.1.2 Usability Questionnaires A.2 Questionnaires for Chapter 5 (Multi-layer Stack Study) A.3 Materials for Section 9.2 (Evaluation of Tangible Lenses) A.3.1 Scratchpad for Study Leader A.3.2 Usability Questionnaires Bibliography List of Figures List of Tables