The prevalent visions of ambient intelligence leverage natural interactions between users and available services in a smart space. In recent years, we have seen a huge interest from industry and academia in using handheld devices to interact with things, places and people in the real world. To facilitate such interactions, things are usually annotated with RFID tags or visual markers. These tags or markers are read by a handheld device equipped with an integrated RFID reader or a camera, in order to fetch related information and initiate further actions. Interacting with the Internet of Things (IoT) in a real environment has become increasingly desirable and feasible. This thesis contributes to the domain of physical interactions with IoT; however, we use a spatial-geometric approach instead of RFID or marker based solutions. Using this approach, for example, a user can point his/her handheld device to an annotated thing, from a distance, for the purpose of interaction. The pointing direction and location is determined based on the fusion of the mobile position and of the accelerometer data of the handheld device. To annotate things, their geometric coordinates are specified and related information or services are associated to them. In this thesis, we present a comprehensive and extensible framework to integrate various physical interactions with IoT into multimedia applications. The framework supports the implementations of pointMe, touchMe, and context-aware based interactions with geometrically annotated IoT. We define specific methods and practices that can be incorporated in order to build the interactions. We realize smart home, atlas learning, presentation interaction, smart haptic interaction, and learning based video interaction game prototypes in order to perform experiments and demonstrate the applicability and potential of the proposed geometric based annotation approach. In the analysis of the interaction techniques of the prototypes, we present the advantages and disadvantages of the geometric based annotation of IoT as seen by potential users, in comparison to RFID tags or visual markers based approaches.