The smart bookshelf serves as a test-bed to study environments that are intelligently augmented by projector-camera devices. The system utilizes a camera pair and a projector coupled with an RFID reader to monitor and maintain the state of a real world library shelf. Using a simple calibration scheme, the homography induced by the world plane in which book spines approximately lie is estimated. As books are added to the shelf, a foreground detection algorithm which takes into account the projected information yields new pixels in each view that are then verified using a planar parallax constraint across both cameras to yield the precise location of the book spine. The system allows users to query for the presence of a books through a user interface, highlighting the spines of present book using the known locations obtained through foreground detection and transforming image pixels to their corresponding points in the projectors frame via a derived homography. The system also can display the state of the bookshelf at any time in the past. Utilizing RFID tags increases robustness and usefulness of the application. Tags encode information about a book such as the title, author, etc, that can be used to query the system. It is used in conjunction with the visual system to infer the state of the shelf. This work provides a novel foreground detection algorithm that works across views, using loose geometric constraints instead pixel color similarity to robustly isolate foreground pixels. The system also takes into account projected information which if not handled would be detrimental to the system. The intent of this work was to study the feasibility of an augmented reality system and use this application as a testbed to study the issues of building such a system.

Technological advancements in sensor miniaturization, processing power and faster networks has broadened the scope of our contemporary compute-infrastructure to an extent that Context-Aware Intelligent Environment (CAIE)--physical spaces with computing systems embedded in it--are increasingly commonplace. With the widespread adoption of intelligent personal agents proliferating as close to us as our living rooms, there is a need to rethink the human-computer interface to accommodate some of their inherent properties such as multiple focus of interaction with a dynamic set of devices and limitations such as lack of a continuous coherent medium of interaction. A CAIE provides context-aware services to aid in achieving user's goals by inferring their instantaneous context. However, often due to lack of complete understanding of a user's context and goals, these services may be inappropriate or at times even pose hindrance in achieving user's goals. Determining service appropriateness is a critical step in implementing a reliable and robust CAIE. Explicitly querying the user to gather such feedback comes at the cost of user's cognitive resources in addition to defeating the purpose of designing a CAIE to provide automated services. The CAIE may, however, infer this appropriateness implicitly from the user, by observing and sensing various behavioral cues and affective reactions from the user, thereby seamlessly gathering such user-feedback. In this dissertation, we have studied the design space for incorporating user's affective reactions to the intelligent services, as a mode of implicit communication between the user and the CAIE. As a result, we have introduced a framework named CAfFEINE, acronym for Context-aware Affective Feedback in Engineering Intelligent Naturalistic Environments. The CAfFEINE framework encompasses models, methods and algorithms establishing the validity of the idea of using a physiological-signal based affective feedback loop in conveying service appropriateness in a CAIE. In doing so, we have identified methods of learning ground-truth about an individual user's affective reactions as well as introducing a novel algorithm of estimating a physiological signal based quality-metric for our inferences. To evaluate the models and methods presented in the CAfFEINE framework, we have designed a set of experiments in laboratory-mockups and virtual-reality setup, providing context aware services to the users, while collecting their physiological signals from wearable sensors. Our results provide empirical validation for our CAfFEINE framework, as well as point towards certain guidelines for conducting future research extending this novel idea. Overall, this dissertation contributes by highlighting the symbiotic nature of the subfields of Affective Computing and Context-aware Computing and by identifying models, proposing methods and designing algorithms that may help accentuate this relationship making future intelligent environments more human-centric. / Ph. D. / Physical spaces containing intelligent computing agents have become an increasingly commonplace concept. These systems when populating a physical space, provides intelligent services by inferring user’s immediate needs, they are called intelligent environments. With this widespread adoption of intelligent systems, there is a need to design computer interfaces that focuses on the human user’s responses. In order for this service-delivery interaction to feel natural, these interfaces need to sense a user’s disapproval of a wrong service, without the user actively indicating so. It is imperative that implicitly inferring a user’s disapproval of a service by observing and sensing various behavioral cues from the user, will help in making the computing system cognitively disappear into the background. In this dissertation, we have studied the design space for incorporating user’s affective reactions to the intelligent services, as a mode of implicit communication between the user and the intelligent system. As a result, we have introduced an interaction framework named CAfFEINE, acronym for Context-aware Affective Feedback in Engineering Intelligent Naturalistic Environments. The CAfFEINE framework encompasses models, methods and algorithms exploring the validity of the idea of using physiological signal based affective feedback in intelligent environments. To evaluate the models and algorithms, we have designed a set of experimental protocols and conducted user studies in virtual-reality setup. The results from these user studies demonstrate the feasibility of this novel idea, in addition to proposing new methods of evaluating the quality of underlying physiological signals. Overall, this dissertation contributes by highlighting the symbiotic nature of the subfields of Affective Computing and Context-aware Computing and by identifying models, proposing methods and designing algorithms that may help accentuate this relationship making future intelligent environments more human-centric.

The use of machine learning to predict output from data, using a model, is a well studied area. There are, however, a number of real-world applications that require a model to be produced but have little or no data available of the specific environment. These situations are prominent in Intelligent Environments (IEs). The sparsity of the data can be a result of the physical nature of the implementation, such as sensors placed into disaster recovery scenarios, or where the focus of the data acquisition is on very defined user groups, in the case of disabled individuals. Standard machine learning approaches focus on a need for training data to come from the same domain. The restrictions of the physical nature of these environments can severely reduce data acquisition making it extremely costly, or in certain situations, impossible. This impedes the ability of these approaches to model the environments. It is this problem, in the area of IEs, that this thesis is focussed. To address complex and uncertain environments, humans have learnt to use previously acquired information to reason and understand their surroundings. Knowledge from different but related domains can be used to aid the ability to learn. For example, the ability to ride a road bicycle can help when acquiring the more sophisticated skills of mountain biking. This humanistic approach to learning can be used to tackle real-world problems where a-priori labelled training data is either difficult or not possible to gain. The transferral of knowledge from a related, but differing context can allow for the reuse and repurpose of known information. In this thesis, a novel composition of methods are brought together that are broadly based on a humanist approach to learning. Two concepts, Transfer Learning (TL) and Fuzzy Logic (FL) are combined in a framework, Fuzzy Transfer Learning (FuzzyTL), to address the problem of learning tasks that have no prior direct contextual knowledge. Through the use of a FL based learning method, uncertainty that is evident in dynamic environments is represented. By combining labelled data from a contextually related source task, and little or no unlabelled data from a target task, the framework is shown to be able to accomplish predictive tasks using models learned from contextually different data. The framework incorporates an additional novel five stage online adaptation process. By adapting the underlying fuzzy structure through the use of previous labelled knowledge and new unlabelled information, an increase in predictive performance is shown. The framework outlined is applied to two differing real-world IEs to demonstrate its ability to predict in uncertain and dynamic environments. Through a series of experiments, it is shown that the framework is capable of predicting output using differing contextual data.

A speech controlled animation system is both a useful application program as well as a laboratory in which to investigate context aware applications as well as controlling errors. The user need not have prior knowledge or experience in animation and is yet able to create interesting and meaningful animation naturally and fluently. The system can be used in a number of applications ranging from PowerPoint presentations to simulations to children’s storytelling tools. / Singapore-MIT Alliance (SMA)

Thesis (MEng) -- Stellenbosch University, 2014. / ENGLISH ABSTRACT: The ever-increasing existence of electronic systems and devices within the residential environment, along with the human desire to simplify life and daily routine, is generating increased interest in the field of home automation (HA) and intelligent environments. A large variety of home automation solutions have been conceptualised or developed. However, many of these solutions are designed by experts and require professionals to install and/or operate them. Furthermore they lack the potential for seamless integration into an already functioning home environment. To bridge the gap between consumer and expert as well as allowing for integration into any existing home environment without physical alteration to the building, a modular home automation solution with seamless integration potential is proposed. The implemented system uses open source software and hardware allowing for development to continue within the existing, large open source community. It can be installed and configured without professional skills or physical alteration of the environment itself and due to its modular design, the system also allows users to add and remove functional components to and from the system providing them with a seamlessly customisable home automation solution. Conceptual design and practical implementations are covered in this document, along with recommendations for both continued research and potential avenues for expansion.

The elderly population worldwide has an increasing expectation of wellbeing and life expectancy. The monitoring of the majority of elderly people on an individual basis, in a medical sense, will not be a viable proposition in the future due to the projected numbers of individuals requiring such activity. The expectation is that the infrastructure available will not be adequate to meet all the anticipated requirements and subsequently people will have to live at home with inadequate care. A new global objective that aims towards enhancing the quality of life of the elderly is being supported by extensive research. This research has been taking place in the field of ambient intelligence (AmI), considering factors including more comfort, improved health, enhanced security for the elderly, and facilitating the living in their homes longer. Prior research has shown a need for accelerated expansion in the ambient intelligence domain. To that end this work presents a novel learning technique for intelligent agents that can be used in Ambient Intelligent Environments (AIEs). The main objective of this work is to add knowledge to the AmI domain and to explore the practical applications within this research field. The added knowledge is accomplished through the development of an ambient intelligent health care environment that allows a practical assessment of the human well-being to take place. This is achieved by transforming the elderly living environment into an intelligent pseudo robot within which they reside to better understand the human wellbeing. The system developed aims to provide evidence that a level of automated care is both possible and practical. This care is for those with chronic physical or mental disabilities who have difficulty in their interactions with standardised living spaces. The novel integrated hardware and software architecture provides personalised environmental monitoring. It also provides control facilities based on the patient‘s physical and emotional wellness in their home. Entitled Health Adaptive Online Emotion Fuzzy Agent (HAOEFA), the system provides a non-invasive, self-learning, intelligent controlling system that constantly adapts to the requirements of an individual. The system has the ability to model and learn the user behaviour in order to control the environment on their behalf. This is achieved with respect to the changing environmental conditions as well as the user‘s health and emotional states being detected. A change of emotion can have a direct impact on the system‘s control taking place in the environment. Thus HAOEFA combines an emotion recognition system within a fuzzy logic learning and adaptation based controller. The emotion recogniser detects the occupant‘s emotions upon the changes of the physiological data being monitored. In addition to acting as an output to the occupant‘s physiological changes, the detected emotion also acts as input to the whole situation being observed by HAOEFA. This allows HAOEFA to control the Glam i-HomeCare on the user‘s behalf with respect to their emotional status. The system developed incorporates real-time, continuous adaptations to facilitate any changes to the occupant‘s behaviour within the environment. It also allows the rules to be adapted and extended online, assisting a life-long learning technique as the environmental conditions change and the user behaviour adjusts with it. HAOEFA uses the fuzzy c-means clustering methodology for extracting membership functions (MFs) before building its set of fuzzy rules. These MFs together with the rules base constitute a major part of the proposed system. It has the ability to learn and model the individual human behaviour with respect to their emotional status. Following the provided literature review and the presentation of Fuzzy logic MFs (see section 3.3). The thesis presents two chosen unobtrusive self-learning techniques that are used in the development of the intelligent fuzzy system. Each approach combines an emotion recogniser with a fuzzy logic learning and adaptation based technique for systems that can be used in AIEs. A comparison of two different MFs designs is contrasted showing the impact they have on the system learning ability. A number of carefully designed experiments were performed by volunteers in the Glam i-HomeCare test-bed at the University of South Wales to examine the system‘s ability to learn the occupant‘s behaviour with respect to their health and emotional states. The experimental procedures were performed twice by each volunteer, while maintaining the same behavioural actions to compare how much the design of fuzzy membership functions can impact the learning process and the number of rules created by the system. Besides evaluating both systems‘ emotion recognition accuracies and comparing them to one another for each occupant, the empirical outcomes show the potential of the approach in assisting the extension of independent living. The results demonstrate how the type-1 fuzzy system both learnt and adapted to each occupant‘s behaviour with respect to their health and emotional state whilst assessing multiple environmental conditions.

In the future, computers will be virtually everywhere: carried by everyone and integrated into the environment. The increased computation and communication capabilities will enable intelligent environments that react to occupants through automated decision-making. Devices (sensors and actuators) are the key to making intelligent environments a reality. We believe that devices must be made more approachable for average users. Existing approaches to application development for intelligent environments require detailed knowledge about devices and their low-leveling programming interfaces, which greatly limits the number of potential users. Instead of limiting users, we must enable everyone to program the devices around them. Intelligent environments will not be commonplace until average people can set up and manage the hardware and software necessary for their personalized applications. In simplifying the programming of intelligent environments, we first made sensors and actuators accessible to average programmers then extended our work to end-users. We term the former contribution Sensor Enablement for Average Programmers (SEAP); the latter work is Sensor Enablement for End-Users (SEEU). In our experience, devices’ disparate, niche programming languages and communication protocols presented great difficulty in developing intelligent environments. To ease the development effort for average programmers, we abstracted and standardized complex sensor and actuator interactions, allowing users to instead think in terms of well-understood web applications. Users have said that SEAP is easy-to-use and exciting. But what about average people, end-users? We found that end-users are incredibly interested in intelligent environments. By engaging end-users we can create intelligent environments even faster and allow domain experts to tailor their environment. This dissertation’s second contribution, Sensor Enablement for End-Users (SEEU) provides a visual programming interface that allows users to create personalized automated behaviors given available devices and data. We performed several user studies to uncover people’s desires for intelligent environments and determine the best interface for managing an intelligent environment. SEEU combines an intuitive interface with the power and flexibility of SEAP. SEEU is a usable end-user programming framework that allows average people to create useful applications for their intelligent environments. With SEEU and SEAP, we simplified the development of intelligent environments, reducing barriers to adoption of emerging sensing and actuation technologies. We demonstrated the feasability with a series of user studies. / text

Desde dezembro de 2007, o Brasil está implantando o Sistema Brasileiro de TV Digital (SBTVD). Além de esse novo sistema proporcionar imagens em alta definição e permitir a mobilidade da TV, ele oferece a interatividade, através do middleware Ginga, o qual está sendo disponibilizado em receptores de TV digital para permitir que os telespectadores possam interagir com as aplicações que são transmitidas - juntas com o sinal de áudio e de vídeo - pelas emissoras de TV. Esses receptores de TV digital, ou Set-top Boxes (STBs), como são conhecidos, estão se tornando cada vez mais presentes nas residências, o que possibilita a sua integração com os Ambientes Inteligentes (AmIs), que são cenários compostos por sistemas de automação predial/residencial capazes de se adaptar à presença dos usuários, oferecer serviços e permitir uma interação multimodal com o ambiente. Assim, este trabalho propõe um framework para integração entre AmIs e o SBTVD, a fim de permitir que os serviços e os dispositivos de automação presentes nas residências possam ser gerenciados pelos receptores de TV digital compatíveis com o middleware de interatividade Ginga. O framework proposto define uma arquitetura que é implementada para atender às seguintes funcionalidades: (i) mapeamento dos dispositivos físicos de automação presentes no AmI para o mundo computacional; (ii) suporte ao desenvolvimento de aplicações interativas para acesso aos serviços e aos dispositivos de automação do AmI; (iii) criação de cenários de automação independentes da plataforma de hardware em que serão executados; (iv) reutilização de projetos para otimização do tempo de desenvolvimento de novas aplicações interativas; (v) geração automática de código, em que são construídas aplicações baseadas no perfil do hardware da plataforma-alvo e no perfil da linguagem de programação suportada pelo middleware de interatividade do SBTVD. Além disso, este trabalho é validado através de três estudos de casos que utilizam os conceitos e as ferramentas computacionais propostas no âmbito desta tese. / Since December 2007, Brazil has been introducing the Digital TV Brazilian System (SBTVD). This new system not only provides high definition images and TV mobility, but also offers interactivity by means of middleware Ginga, which has become available in Digital TV receivers in order to grant that viewers are able to interact with the transmitted applications – along with the video and audio signal – by broadcasting stations. Such Digital TV receivers, or the well-known Set-top Boxes (STBs), have ranked high in homes, a factor that enables its integration with Intelligent Environments (AmIs), which are sceneries composed by home automation systems, and may adapt to the presence of users, offer services and allow multimodal interaction with the environment. Thus, this thesis presents a framework for integration between AmIs and SBTVD in order to grant that services and automation devices located in homes can be managed by Digital TV receivers with Ginga interactivity middleware. The referred framework defines an architecture that is implemented with the aim of meeting the following functionalities: (i) physical devices mapping of automation in AmIs to the computational world; (ii) support to the development of interactive applications to access the AmI services and automation devices; (iii) creation of automation sceneries which do not depend on the hardware platform they will be run in; (iv) reuse of projects to optimize the development time of new interactive applications; (v) code automatic generation, in which applications will be constructed based on the profile of the target platform hardware and on the programming language profile supported by the SBTVD interactivity middleware. Besides, this work is validated due to three case studies that utilize concepts and computational tools underlying this thesis.

Desde dezembro de 2007, o Brasil está implantando o Sistema Brasileiro de TV Digital (SBTVD). Além de esse novo sistema proporcionar imagens em alta definição e permitir a mobilidade da TV, ele oferece a interatividade, através do middleware Ginga, o qual está sendo disponibilizado em receptores de TV digital para permitir que os telespectadores possam interagir com as aplicações que são transmitidas - juntas com o sinal de áudio e de vídeo - pelas emissoras de TV. Esses receptores de TV digital, ou Set-top Boxes (STBs), como são conhecidos, estão se tornando cada vez mais presentes nas residências, o que possibilita a sua integração com os Ambientes Inteligentes (AmIs), que são cenários compostos por sistemas de automação predial/residencial capazes de se adaptar à presença dos usuários, oferecer serviços e permitir uma interação multimodal com o ambiente. Assim, este trabalho propõe um framework para integração entre AmIs e o SBTVD, a fim de permitir que os serviços e os dispositivos de automação presentes nas residências possam ser gerenciados pelos receptores de TV digital compatíveis com o middleware de interatividade Ginga. O framework proposto define uma arquitetura que é implementada para atender às seguintes funcionalidades: (i) mapeamento dos dispositivos físicos de automação presentes no AmI para o mundo computacional; (ii) suporte ao desenvolvimento de aplicações interativas para acesso aos serviços e aos dispositivos de automação do AmI; (iii) criação de cenários de automação independentes da plataforma de hardware em que serão executados; (iv) reutilização de projetos para otimização do tempo de desenvolvimento de novas aplicações interativas; (v) geração automática de código, em que são construídas aplicações baseadas no perfil do hardware da plataforma-alvo e no perfil da linguagem de programação suportada pelo middleware de interatividade do SBTVD. Além disso, este trabalho é validado através de três estudos de casos que utilizam os conceitos e as ferramentas computacionais propostas no âmbito desta tese. / Since December 2007, Brazil has been introducing the Digital TV Brazilian System (SBTVD). This new system not only provides high definition images and TV mobility, but also offers interactivity by means of middleware Ginga, which has become available in Digital TV receivers in order to grant that viewers are able to interact with the transmitted applications – along with the video and audio signal – by broadcasting stations. Such Digital TV receivers, or the well-known Set-top Boxes (STBs), have ranked high in homes, a factor that enables its integration with Intelligent Environments (AmIs), which are sceneries composed by home automation systems, and may adapt to the presence of users, offer services and allow multimodal interaction with the environment. Thus, this thesis presents a framework for integration between AmIs and SBTVD in order to grant that services and automation devices located in homes can be managed by Digital TV receivers with Ginga interactivity middleware. The referred framework defines an architecture that is implemented with the aim of meeting the following functionalities: (i) physical devices mapping of automation in AmIs to the computational world; (ii) support to the development of interactive applications to access the AmI services and automation devices; (iii) creation of automation sceneries which do not depend on the hardware platform they will be run in; (iv) reuse of projects to optimize the development time of new interactive applications; (v) code automatic generation, in which applications will be constructed based on the profile of the target platform hardware and on the programming language profile supported by the SBTVD interactivity middleware. Besides, this work is validated due to three case studies that utilize concepts and computational tools underlying this thesis.

Desde dezembro de 2007, o Brasil está implantando o Sistema Brasileiro de TV Digital (SBTVD). Além de esse novo sistema proporcionar imagens em alta definição e permitir a mobilidade da TV, ele oferece a interatividade, através do middleware Ginga, o qual está sendo disponibilizado em receptores de TV digital para permitir que os telespectadores possam interagir com as aplicações que são transmitidas - juntas com o sinal de áudio e de vídeo - pelas emissoras de TV. Esses receptores de TV digital, ou Set-top Boxes (STBs), como são conhecidos, estão se tornando cada vez mais presentes nas residências, o que possibilita a sua integração com os Ambientes Inteligentes (AmIs), que são cenários compostos por sistemas de automação predial/residencial capazes de se adaptar à presença dos usuários, oferecer serviços e permitir uma interação multimodal com o ambiente. Assim, este trabalho propõe um framework para integração entre AmIs e o SBTVD, a fim de permitir que os serviços e os dispositivos de automação presentes nas residências possam ser gerenciados pelos receptores de TV digital compatíveis com o middleware de interatividade Ginga. O framework proposto define uma arquitetura que é implementada para atender às seguintes funcionalidades: (i) mapeamento dos dispositivos físicos de automação presentes no AmI para o mundo computacional; (ii) suporte ao desenvolvimento de aplicações interativas para acesso aos serviços e aos dispositivos de automação do AmI; (iii) criação de cenários de automação independentes da plataforma de hardware em que serão executados; (iv) reutilização de projetos para otimização do tempo de desenvolvimento de novas aplicações interativas; (v) geração automática de código, em que são construídas aplicações baseadas no perfil do hardware da plataforma-alvo e no perfil da linguagem de programação suportada pelo middleware de interatividade do SBTVD. Além disso, este trabalho é validado através de três estudos de casos que utilizam os conceitos e as ferramentas computacionais propostas no âmbito desta tese. / Since December 2007, Brazil has been introducing the Digital TV Brazilian System (SBTVD). This new system not only provides high definition images and TV mobility, but also offers interactivity by means of middleware Ginga, which has become available in Digital TV receivers in order to grant that viewers are able to interact with the transmitted applications – along with the video and audio signal – by broadcasting stations. Such Digital TV receivers, or the well-known Set-top Boxes (STBs), have ranked high in homes, a factor that enables its integration with Intelligent Environments (AmIs), which are sceneries composed by home automation systems, and may adapt to the presence of users, offer services and allow multimodal interaction with the environment. Thus, this thesis presents a framework for integration between AmIs and SBTVD in order to grant that services and automation devices located in homes can be managed by Digital TV receivers with Ginga interactivity middleware. The referred framework defines an architecture that is implemented with the aim of meeting the following functionalities: (i) physical devices mapping of automation in AmIs to the computational world; (ii) support to the development of interactive applications to access the AmI services and automation devices; (iii) creation of automation sceneries which do not depend on the hardware platform they will be run in; (iv) reuse of projects to optimize the development time of new interactive applications; (v) code automatic generation, in which applications will be constructed based on the profile of the target platform hardware and on the programming language profile supported by the SBTVD interactivity middleware. Besides, this work is validated due to three case studies that utilize concepts and computational tools underlying this thesis.