Technological advancements in sensor miniaturization, processing power and faster networks has broadened the scope of our contemporary compute-infrastructure to an extent that Context-Aware Intelligent Environment (CAIE)--physical spaces with computing systems embedded in it--are increasingly commonplace. With the widespread adoption of intelligent personal agents proliferating as close to us as our living rooms, there is a need to rethink the human-computer interface to accommodate some of their inherent properties such as multiple focus of interaction with a dynamic set of devices and limitations such as lack of a continuous coherent medium of interaction. A CAIE provides context-aware services to aid in achieving user's goals by inferring their instantaneous context. However, often due to lack of complete understanding of a user's context and goals, these services may be inappropriate or at times even pose hindrance in achieving user's goals. Determining service appropriateness is a critical step in implementing a reliable and robust CAIE. Explicitly querying the user to gather such feedback comes at the cost of user's cognitive resources in addition to defeating the purpose of designing a CAIE to provide automated services. The CAIE may, however, infer this appropriateness implicitly from the user, by observing and sensing various behavioral cues and affective reactions from the user, thereby seamlessly gathering such user-feedback. In this dissertation, we have studied the design space for incorporating user's affective reactions to the intelligent services, as a mode of implicit communication between the user and the CAIE. As a result, we have introduced a framework named CAfFEINE, acronym for Context-aware Affective Feedback in Engineering Intelligent Naturalistic Environments. The CAfFEINE framework encompasses models, methods and algorithms establishing the validity of the idea of using a physiological-signal based affective feedback loop in conveying service appropriateness in a CAIE. In doing so, we have identified methods of learning ground-truth about an individual user's affective reactions as well as introducing a novel algorithm of estimating a physiological signal based quality-metric for our inferences. To evaluate the models and methods presented in the CAfFEINE framework, we have designed a set of experiments in laboratory-mockups and virtual-reality setup, providing context aware services to the users, while collecting their physiological signals from wearable sensors. Our results provide empirical validation for our CAfFEINE framework, as well as point towards certain guidelines for conducting future research extending this novel idea. Overall, this dissertation contributes by highlighting the symbiotic nature of the subfields of Affective Computing and Context-aware Computing and by identifying models, proposing methods and designing algorithms that may help accentuate this relationship making future intelligent environments more human-centric. / Ph. D. / Physical spaces containing intelligent computing agents have become an increasingly commonplace concept. These systems when populating a physical space, provides intelligent services by inferring user’s immediate needs, they are called intelligent environments. With this widespread adoption of intelligent systems, there is a need to design computer interfaces that focuses on the human user’s responses. In order for this service-delivery interaction to feel natural, these interfaces need to sense a user’s disapproval of a wrong service, without the user actively indicating so. It is imperative that implicitly inferring a user’s disapproval of a service by observing and sensing various behavioral cues from the user, will help in making the computing system cognitively disappear into the background. In this dissertation, we have studied the design space for incorporating user’s affective reactions to the intelligent services, as a mode of implicit communication between the user and the intelligent system. As a result, we have introduced an interaction framework named CAfFEINE, acronym for Context-aware Affective Feedback in Engineering Intelligent Naturalistic Environments. The CAfFEINE framework encompasses models, methods and algorithms exploring the validity of the idea of using physiological signal based affective feedback in intelligent environments. To evaluate the models and algorithms, we have designed a set of experimental protocols and conducted user studies in virtual-reality setup. The results from these user studies demonstrate the feasibility of this novel idea, in addition to proposing new methods of evaluating the quality of underlying physiological signals. Overall, this dissertation contributes by highlighting the symbiotic nature of the subfields of Affective Computing and Context-aware Computing and by identifying models, proposing methods and designing algorithms that may help accentuate this relationship making future intelligent environments more human-centric.