This thesis presents a recollection of developments and results towards the research of human-like semantic understanding of the environment for robotics systems. Achieving a level of understanding in robots comparable to humans has proven to be a significant challenge in robotics, although modern sensors like stereo cameras and neuromorphic cameras enable robots to perceive the world in a manner akin to human senses, extracting and interpreting semantic information proves to be significantly inefficient by comparison. This thesis explores different aspects of the machine vision field to level computational methods in order to address real-life challenges for the task of semantic scene understanding in both everyday environments as well as challenging unstructured environments.  The works included in this thesis present key contributions towards three main research directions. The first direction establishes novel perception algorithms for object detection and localization, aimed at real-life deployments in onboard mobile devices for %perceptually degraded unstructured environments. Along this direction, the contributions focus on the development of robust detection pipelines as well as fusion strategies for different sensor modalities including stereo cameras, neuromorphic cameras, and LiDARs.  The second research direction establishes a computational method for levering semantic information into meaningful knowledge representations to enable human-inspired behaviors for the task of traversability estimation for reactive navigation. The contribution presents a novel decay function for traversability soft image generation based on exponential decay, by fusing semantic and geometric information to obtain density images that represent the pixel-wise traversability of the scene. Additionally, it presents a novel Encoder-Decoder lightweight network architecture for coarse semantic segmentation of terrain, integrated with a memory module based on a dynamic certainty filter. Finally, the third research direction establishes the novel concept of Belief Scene Graphs, which are utility-driven extensions of partial 3D scene graphs, that enable efficient high-level task planning with partial information.The research thus presents an approach to meaningfully incorporate unobserved objects as nodes into an incomplete 3D scene graph using the proposed method Computation of Expectation based on Correlation Information (CECI), to reasonably approximate the probability distribution of the scene by learning histograms from available training data. Extensive simulations and real-life experimental setups support the results and assumptions presented in this work.

Recently advances in computer vision have been made with the aid of artificial intelligence. This has made tracking sports in real-time a possibility. Specifically, this project aim to track objects in a football exercise in real-time with a single low-angle video camera, and quickly create feedback. Detecting the players during the exercise was mostly successful, but the ball could only be detected in certain areas of the playing area. Keeping track of the different players in the exercise proved to be difficult with the setup. Alternate ways to provide feedback without knowing the identities of the players were possible but limited. To be able to reliably provide insightful feedback the system would likely need to be changed to a high-angle or multi-camera system.

In the field of construction equipment, a future is envisioned in which humans and autonomous machines can collaborate seamlessly. An example of this vision is embodied in the Volvo prototype LX03, an autonomous wheel loader engineered to function as a smart and safe partner with collaborative capabilities. In these situations, it is crucial that humans and machines communicate effectively. One critical aspect for machines to consider is the awareness level of humans, as it significantly influences their decision-making processes. This thesis investigates the feasibility of constructing a deep learning model to classify if a human is aware towards the machine or not using computer vision from the machines Point of View. To test this, a state-of-the-art action recognition model was used, namely RGBPose-Conv3D which is a 3D Convolutional Neural Network. This model uses two modalities, namely RGB and Pose, which could be used together or separately. The model was modified and trained to classify aware and unaware behaviour. The dataset used to train and test the model was collected with actors that mimicked aware or unaware behaviour. When using only RGB the model did not perform well, but when using Pose only or Pose and RGB fused, the model performed well in classifying the awareness state. Furthermore, the model exhibited good generalisability to scenarios on which it had not previously been trained. Such as with a machine movement, multiple people or previously not seen scenarios. The thesis highlights the viability of employing deep learning and computer vision for awareness detection, showcasing a novel method that achieves high accuracy despite minimal comparative research.

The navigation tasks in advanced home robotic applications incorporating reliable revisiting strategies are dependent on very low cost but nevertheless rather accurate localization systems. In this paper a localization system based on the principle of trilateration is described. The proposed system uses only a single small base station, but achieves accuracies comparable to systems using spread beacons and it performs sufficiently for map building. Thus it is a standalone system and needs no odometry or other auxiliary sensors. Furthermore a new approach for the problem of the reliably detection of areas without direct line of sight is presented. The described system is very low cost and it is designed for use in indoor service robotics. The paper gives an overview on the system concept and special design solutions and proves the possible performances with experimental results.

Das Programmieren der Bewegungsbahnen von Robotern erfordert Fachwissen und ist ein zeitintensiver und aufwendiger Prozess. Dieser Beitrag beschäftigt sich mit dem Einsatz von Augmented Reality (AR) in Form eines AR Head Mounted Display (HMD) als intuitives Schnittstelle (engl. Interface) für die Roboterprogrammierung. Zunächst wird ein Überblick über aktuelle und relevante Forschung im Bereich AR Anwendungen in der Robotik gegeben. Aktuelle Forschungsarbeit auf dem Gebiet widmet sich vorrangig der technischen Umsetzung einzelner Funktionalitäten. In diesem Beitrag aus der Praxis sollen die technischen Möglichkeiten den Problematiken potenzieller Anwender:innen angepasst werden. Der Fokus liegt damit auf dem Mehrwert für spezifische Nutzergruppen und der einfachen und intuitiven Bedienung des AR Interfaces selbst. Zunächst wird, einem nutzerzentrierten Entwicklungsprozess folgend, erhoben, welchen Herausforderungen Expert:innen und Laien bei der Roboterprogrammierung begegnen. Auf dieser Basis werden Anforderungen abgeleitet und ein erlebbarer Prototyp entwickelt und gestaltet, der eine weitere Untersuchungen ermöglicht. Ein geplantes Untersuchungskonzept hinsichtlich Aspekten der User Experience (UX) wird im Ausblick beleuchtet.

Vector Symbolic Architectures (VSA) combine a hypervector space and a set of operations on these vectors. Hypervectors provide powerful and noise-robust representations and VSAs are associated with promising theoretical properties for approaching high-level cognitive tasks. However, a major drawback of VSAs is the lack of opportunities to learn them from training data. Their power is merely an effect of good (and elaborate) design rather than learning. We exploit high-level knowledge about the structure of reactive robot problems to learn a VSA based on training data. We demonstrate preliminary results on a simple navigation task. Given a successful demonstration of a navigation run by pairs of sensor input and actuator output, the system learns a single hypervector that encodes this reactive behaviour. When executing (and combining) such VSA-based behaviours, the advantages of hypervectors (i.e. the representational power and robustness to noise) are preserved. Moreover, a particular beauty of this approach is that it can learn encodings for behaviours that have exactly the same form (a hypervector) no matter how complex the sensor input or the behaviours are.

This thesis investigates combining learning action nodes in behavior trees with control barrier functions based on the extended active constraint conditions of the nodes and whether the approach improves the performance, in terms of training time and policy quality, compared to a purely learning-based approach. Behavior trees combine several behaviors, called action nodes, into one behavior by switching between them based on the current state. Those behaviors can be hand-coded or learned in so-called learning action nodes. In these nodes, the behavior is a reinforcement learning agent. Behavior trees can be constructed in a process called backward chaining. In order to ensure the success of a backward-chained behavior tree, each action node must uphold previously achieved subgoals. So-called extended active constraint conditions formalize this notion as conditions that must stay true for the action node to continue execution. In order to incentivize upholding extended active constraint conditions in learning action nodes, a negative reward can be given to the agent upon violating extended active constraint conditions. However, this approach does not guarantee not violating the extended active constraint conditions since it is purely learning-based. Control barrier functions can be used to restrict the actions available to an agent so that it stays within a safe subset of the state space. By defining the safe subset of the state space as the set in which the extended active constraint conditions are satisfied, control barrier functions can be employed to, ideally, guarantee that the extended active constraint conditions will not be violated. The results show that significantly less training is needed to get comparable, or slightly better, results, when compared to not using control barrier functions. Furthermore, extended active constraint conditions are considerably less frequently violated and the overall performance is slightly improved. / Denna avhandling undersöker kombinationen av inlärningsregulatornoder i beteendeträd med styrbarriärfunktioner baserade på utökade aktiva begränsningsvillkor för noderna, samt om detta tillvägagångssätt förbättrar prestandan avseende tränings- och policynkvalitet, jämfört med ett rent inlärningsbaserat tillvägagångssätt. Beteendeträd kombinerar flera regulatorer, kallade regulatornoder, till en enda regulator genom att växla mellan dem baserat på det aktuella tillståndet. Dessa regulatorer kan vara handkodade eller inlärda i så kallade inlärningsnoder. I dessa noder är regulatorn en förstärkningsinlärningsagent. Beteendeträd kan konstrueras genom en process som kallas bakåtkoppling. För att säkerställa framgången för ett bakåtkopplat beteendeträd måste varje regulatornod upprätthålla tidigare uppnådda delmål. Utökade aktiva begränsningsvillkor formaliserar denna uppfattning som villkor som inte får överträdas för att regulatornoden ska fortsätta exekvera. För att uppmuntra till att upprätthålla utökade aktiva begränsningsvillkor i inlärningsnoder kan en negativ belöning ges till agenten vid överträdelse av utökade aktiva begränsningsvillkor. Denna metod garanterar dock inte att utökade aktiva begränsningsvillkor inte kommer att överträdas, eftersom den är helt inlärningsbaserad. Kontrollbarriärfunktioner kan användas för att begränsa de åtgärder som är tillgängliga för en agent så att den förblir inom en säker delmängd av tillståndsrymden. Genom att definiera den säkra delmängden av tillståndsrymden som den uppsättning där de utökade aktiva begränsningsvillkoren uppfylls kan kontrollbarriärfunktioner användas för att, i bästa fall, garantera att de utökade aktiva begränsningsvillkoren inte kommer att överträdas. Resultaten visar att det krävs betydligt mindre träning för att få jämförbara, eller något bättre, resultat jämfört med att inte använda kontrollbarriärfunktioner. Dessutom överträds utökade aktiva begränsningsvillkor betydligt mer sällan och den övergripande prestandan är något förbättrad. I would like to thank Katrina Liang and Petter Ögren for translating the to Swedish. / Diese Arbeit untersucht die Kombination von Lernaktionsknoten in Verhaltensbäumen mit Kontrollbarrierefunktionen, die auf den erweiterten aktiven Einschränkungsbedingungen und Vorbedingungen der Knoten basieren, und ob dieser Ansatz die Leistung hinsichtlich Trainingszeit und Qualität der erlernten Strategie im Vergleich zu einem rein lernbasierten Ansatz verbessert. Verhaltensbäume kombinieren mehrere Regler, die als Aktionsknoten bezeichnet werden, zu einem zusammengesetzten Regler, indem sie abhängig vom aktuellem Zustand zwischen ihnen wechseln. Diese Regler können entweder manuell programmiert oder in sogenannten lernenden Aktionsknoten erlernt werden. In diesen Knoten ist der Regler ein Reinforcement Learning Agent. Verhaltensbäume können in einem Prozess namens Rückwärtsverkettung erstellt werden. Um den Erfolg eines rückwärtsverketteten Verhaltensbaums sicherzustellen, muss jeder Aktionsknoten zuvor erreichte Teilerfolge aufrechterhalten. Sogenannte erweiterte aktive Einschränkungsbedingungen formalisieren diesen Gedanken als Bedingungen, die nicht verletzt werden dürfen, damit der Aktionsknoten die Ausführung fortsetzen kann. Um einen Anreiz für die Aufrechterhaltung erweiterter aktiver Einschränkungsbedingungen in Lernaktionsknoten zu schaffen, kann dem Agenten bei Verstoß gegen erweiterte aktive Einschränkungsbedingungen eine negative Belohnung gewährt werden. Diese Herangehensweise garantiert jedoch nicht die Einhaltung der erweiterten aktiven Einschränkungsbedingungen, da sie rein lernbasiert ist. Kontrollbarrierefunktionen können verwendet werden, um die verfügbaren Aktionen eines Agenten zu beschränken, damit dieser in einer sicheren Teilmenge des Zustandsraums bleibt. Indem die sichere Teilmenge des Zustandsraums als die Menge definiert wird, in der die erweiterten aktiven Einschränkungsbedingungen erfüllt sind, können Kontrollbarrierefunktionen idealerweise verwendet werden, um sicherzustellen, dass die erweiterten aktiven Einschränkungsbedingungen nicht verletzt werden. Die Ergebnisse zeigen, dass im Vergleich zur Nichtverwendung von Kontrollbarrierefunktionen deutlich weniger Training erforderlich ist, um vergleichbare oder etwas bessere Ergebnisse zu erzielen. Darüber hinaus werden erweiterte aktive Einschränkungsbedingungen deutlich seltener verletzt und die Gesamtleistung wird leicht verbessert.

This technical report publishes the proceedings of the 26th Bilateral Student Workshop CTU Prague and HTW Dresden - User Interfaces & Visualization -, which was held on the 1st and 2nd December 2023. The workshop offers a possibility for young scientists to present their current research work in the fields of computer graphics, human-computer-interaction, robotics and usability. The workshop is intended to be a platform to bring together researchers from both the Czech Technical University in Prague (CTU) and the University of Applied Sciences Dresden (HTW). The German Academic Exchange Service offers its financial support to allow student participants the bilateral exchange between Prague and Dresden.:1) Václav Pavlovec: Multi-Boundary Labeling, pp. 2–5 2) Philipp Ballin: Influence of Speed, Direction, and Intensity of Vibrotactile Animations onto Emotional Level, pp. 6–14 3) Niklas Maximilian Kothe, Leon Kolosov: Errors In Pictures, pp. 15–21 4) Jan Trávníček: Automatic Sports Equipment Rental Service, pp. 22–26 5) Jimmy Orawetz, Felix Mühlberg: Capturing and Reproducing Atmospheres, pp. 27–32 6) Vojtěch Leischner: Time Based Audio-movement Graph, pp. 33–37 7) Markéta Machová: Balancing Exercises for Seniors in VR with Interactive Elements, pp. 38–41 8) Radka Olyšarová: MediaPipe Based Leg Tracking Method for Sensorimotor Walking Exercise for Elderly in VR, pp. 42–46 9) Vojtěch Radakulan: Application of Diegetic and Non-Diegetic Navigation in Virtual Reality, pp. 47–51 / Dieser Tagungsband enthält die Beiträge des 26. Bilateralen Studentenworkshops der CTU Prag und der HTW Dresden zu User Interfaces & Visualization, der am 1. und 2. Dezember 2023 stattfand. Der Workshop bietet jungen Wissenschaftlerinnen und Wissenschaftlern die Möglichkeit, ihre aktuellen Forschungsarbeiten in den Bereichen Computergrafik, Mensch-Computer-Interaktion, Robotik und Usability zu präsentieren. Der Workshop soll eine Plattform sein, um Forschende der Tschechischen Technischen Universität Prag (CTU) und der Hochschule für Technik und Wirtschaft Dresden (HTW) zusammenzubringen. Der Deutsche Akademische Austauschdienst stellt die finanzielle Unterstützung bereit, um den studentischen Teilnehmenden den bilateralen Austausch zwischen Prag und Dresden zu ermöglichen.:1) Václav Pavlovec: Multi-Boundary Labeling, pp. 2–5 2) Philipp Ballin: Influence of Speed, Direction, and Intensity of Vibrotactile Animations onto Emotional Level, pp. 6–14 3) Niklas Maximilian Kothe, Leon Kolosov: Errors In Pictures, pp. 15–21 4) Jan Trávníček: Automatic Sports Equipment Rental Service, pp. 22–26 5) Jimmy Orawetz, Felix Mühlberg: Capturing and Reproducing Atmospheres, pp. 27–32 6) Vojtěch Leischner: Time Based Audio-movement Graph, pp. 33–37 7) Markéta Machová: Balancing Exercises for Seniors in VR with Interactive Elements, pp. 38–41 8) Radka Olyšarová: MediaPipe Based Leg Tracking Method for Sensorimotor Walking Exercise for Elderly in VR, pp. 42–46 9) Vojtěch Radakulan: Application of Diegetic and Non-Diegetic Navigation in Virtual Reality, pp. 47–51

Im Rahmen des 180jährigen Jubiläums der technischen Universität Chemnitz fand am 28. April 2016 das zweite Studierendensymposium der Fakultät Informatik statt. Das Studierendensymposium Informatik richtete sich inhaltlich an alle Themen rund um die Informatik und ihre Anwendungen: Ob Hardware oder Software, ob technische Lösungen oder Anwenderstudien, ob Programmierung oder Verwendung, ob Hardcore-Technik oder gesellschaftliche Fragestellungen – alles, was mit informatischen Lösungen zu tun hat, war willkommen. Das Studierendensymposium Informatik war dabei weder auf die Fakultät Informatik noch auf die TU Chemnitz begrenzt. Es wurden explizit Einreichungen aus thematisch angrenzenden Fächern beworben und Hochschulen der Region in die Planung und Organisation eingebunden. Der Tagungsband enthält die 21 Beitrage, die auf dem Symposium vorgestellt wurden. / In the course of the 180 year anniversary of the Technische Universität Chemnitz the Department of Computer Science held the second Students Symposium on April 18, 2016. The symposium addressed topics related to computer science and its applications: Whether hardware or software, whether technical solutions or user studies, whether programming or use, whether hardcore technology or social issues - everything concerned with computational solutions was welcomed. The Students Symposium included explicitly submissions from thematically adjacent departments and involved universities in the region in planning and organization. The proceedings contain the 21 papers (full and short), which were presented at the symposium.

In this thesis we present a detect-tracking algorithm (see figure 3.1) that combines the detection robustness of static region proposal based object detectors, like the faster region convolutional neural network (R-CNN) and the region-based fully convolutional networks (R-FCN) model, with the tracking prediction strength of extended Kalman filters, by using, what we have called, a translating and non-rigid user input region of interest (RoI-) mapping. This so-called RoI-mapping maps a region, which includes the object that one is interested in tracking, to a featureless three-channeled image. The detection part of our proposed algorithm is then performed on the image that includes only the RoI features (see figure 3.2). After the detection step, our model re-maps the RoI features to the original frame, and translates the RoI to the center of the prediction. If no prediction occurs, our proposed model integrates a temporal dependence through a Kalman filter as a predictor; this filter is continuously corrected when detections do occur. To train the region proposal based object detectors that we integrate into our detect-tracking model, we used TensorFlow®’s object detection api, with a random search hyperparameter tuning, where we fine-tuned, all models from TensorFlow® slim base network classification checkpoints. The trained region proposal based object detectors used the inception V2 base network for the faster R-CNN model and the R-FCN model, while the inception V3 base network only was applied to the faster R-CNN model. This was made to compare the two base networks and their corresponding affects on the detection models. In addition to the deep learning part of this thesis, for the implementation part of our detect-tracking model, like for the extended Kalman filter, we used Python and OpenCV® . The results show that, with a stationary camera reference frame, our proposed detect-tracking algorithm, combined with region proposal based object detectors on images of size 414 × 740 × 3, can detect and track a small object in real-time, like a tennis ball, moving along a horizontal trajectory with an average velocity v ≈ 50 km/h at a distance d = 25 m, with a combined detect-tracking frequency of about 13 to 14 Hz. The largest measured state error between the actual state and the predicted state from the Kalman filter, at the aforementioned horizontal velocity, have been measured to be a maximum of 10-15 pixels, see table 5.1, but in certain frames where many detections occur this error has been shown to be much smaller (3-5 pixels). Additionally, our combined detect-tracking model has also been shown to be able to handle obstacles and two learnable features that overlap, thanks to the integrated extended Kalman filter. Lastly, our detect-tracking model also was applied on a set of infra-red images, where the goal was to detect and track a moving truck moving along a semi-horizontal path. Our results show that a faster R-CNN inception V2 model was able to extract features from a sequence of infra-red frames, and that our proposed RoI-mapping method worked relatively well at detecting only one truck in a short test-sequence (see figure 5.22).