Event detection is a critical challenge in many fields like video surveillance, social graph analysis, and multimedia processing. Furthermore, events are “structured” objects involv ing multiple components like the event type, the participants with their roles, and the atomic events in which it decomposes. Therefore, the recognition of an event is not only limited to recognize the type of the event and when it happened, but it involves solving a set of simple tasks. Exploiting background knowledge about events and their relations could then be beneficial for event detection. In the last years, neuro-symbolic integration has been proposed to merge the strengths and overcome the drawbacks of both symbolic and neural worlds. As a consequence, different neuro-symbolic frameworks, which com bine low-level perception of neural networks with a symbolic layer, encoding prior domain knowledge (usually defined in terms of logical rules), have been applied to solve different atemporal tasks. In this thesis, we want to investigate the application of the neuro-symbolic paradigm for event detection. This would also provide a better insight into the strengths and limitations of neuro-symbolic towards tasks involving time.

In the realm of natural language programming, translating free-form sentences in natural language into a functional, machine-executable program remains difficult due to the following 4 challenges. First, the inherent ambiguity of natural languages. Second, the high-level verbose nature in user descriptions. Third, the complexity in the sentences and Fourth, the invalid or semantically unclear sentences. Our first solution is a Large Language Model (LLM) based Artificial Intelligence driven assistant to process free-form sentences and decompose them into sequences of simplified, unambiguous sentences that abide by a set of rules, thereby stripping away the complexities embedded within the original sentences. These resulting sentences are then used to generate the code. We applied the proposed approach to a set of free-form sentences written by middle-school students for describing the logic behind video games. More than 60% of the free-form sentences containing these problems were sufficiently converted to sequences of simple unambiguous object-oriented sentences by our approach. Next, the thesis also presents "IntentGuide," a neuro-symbolic integration framework to enhance the clarity and executability of human intentions expressed in freeform sentences. IntentGuide effectively integrates the rule-based error detection capabilities of symbolic AI with the powerful adaptive learning abilities of Large Language Model to convert ambiguous or complex sentences into clear, machine-understandable instructions. The empirical evaluation of IntentGuide performed on natural language sentences written by middle school students for designing video games, reveals a significant improvement in error correction and code generation abilities compared to previous approach, attaining an accuracy rate of 90%. / Master of Science / Imagine if you could talk to machines in everyday language and they could understand exactly what you meant, turning your words into programs that do exactly what you describe. That's the goal of the thesis. We've developed a system that helps machines make sense of the kind of free-form language that people, especially students, use when they describe what they want a computer to do. Understanding and converting everyday language into computer code is a complex challenge, primarily because the way we naturally speak can be vague, overly detailed, or just complex. This thesis presents a new tool using artificial intelligence that helps break down and simplify these sentences. By transforming them into clearer, rulefollowing instructions, this tool makes it easier for machines to understand and execute the tasks we describe. The technology was tested using descriptions from middle-school students on how video games should work. Over 60% of these complex or unclear descriptions were sufficiently converted into straightforward instructions that a machine could use. Additionally, a new system called "IntentGuide" was introduced, combining traditional AI methods with advanced language models to improve how effectively machines can interpret and act on human instructions. This improved system showed a 90% accuracy in understanding and correcting errors in the students' game descriptions, marking a significant step forward in helping computers better understand us.

In the past decade, reinforcement learning (RL) has achieved breakthroughs across various domains, from surpassing human performance in strategy games to enhancing the training of large language models (LLMs) with human feedback. However, RL has yet to gain widespread adoption in mission-critical fields such as healthcare and autonomous vehicles. This is primarily attributed to the inherent lack of trust, explainability, and generalizability of neural networks in deep reinforcement learning (DRL) agents. While neural DRL agents leverage the power of neural networks to solve specific tasks robustly and efficiently, this often comes at the cost of explainability and generalizability. In contrast, pure symbolic agents maintain explainability and trust but often underperform in high-dimensional data. In this work, we developed a method to distill explainable and trustworthy agents using neuro-symbolic AI. Neuro-symbolic distillation combines the strengths of symbolic reasoning and neural networks, creating a hybrid framework that leverages the structured knowledge representation of symbolic systems alongside the learning capabilities of neural networks. The key steps of neuro-symbolic distillation involve training traditional DRL agents, followed by extracting, selecting, and distilling their learned policies into symbolic forms using symbolic regression and tree-based models. These symbolic representations are then employed instead of the neural agents to make interpretable decisions with comparable accuracy. The approach is validated through experiments on Lunar Lander and Pong, demonstrating that symbolic representations can effectively replace neural agents while enhancing transparency and trustworthiness. Our findings suggest that this approach mitigates the black-box nature of neural networks, providing a pathway toward more transparent and trustworthy AI systems. The implications of this research are significant for fields requiring both high performance and explainability, such as autonomous systems, healthcare, and financial modeling.

Modern autonomous systems should be able to interact with their surroundings in a flexible yet safe manner. To guarantee this behavior, such systems must learn how to approach unseen entities in their environment through the inference of relationships between actions and objects, called affordances. This research project introduces a neuro-symbolic AI system capable of inferring affordances using attribute detection and knowledge representation as its core principles. The attribute detection module employs a visuo-lingual image captioning model to extract the key object attributes of a scene, while the cognitive knowledge module infers the affordances of those attributes using conditional probability. The practical capabilities of the neuro-symbolic AI system are assessed by implementing a simulated robot system that interacts within the problem space of jars and bottles. The neuro-symbolic AI system is evaluated through its caption-inferring capabilities using image captioning and machine translation metrics. The scores registered in the evaluation show a successful attribute captioning rate of more than 71%. The robot simulation is evaluated within a Unity virtual environment by interacting with 50 jars and bottles, equally divided between lifting and twisting affordances. The robot system successfully interacts with all the objects in the scene due to the robustness of the architecture but fails in the inference process 24 out of the 50 iterations. Contrary to previous works approaching the problem as a classification task, this study shows that affordance inference can be successfully implemented using a cognitive visuo-lingual method. The study’s results justify further study into the use of neuro-symbolic AI approaches to affordance inference.

<p>Language “in the wild” is complex and ambiguous and relies on a shared understanding of the world for its interpretation. Most current natural language processing methods represent language by learning word co-occurrence patterns from massive amounts of linguistic data. This representation can be very powerful, but it is insufficient to capture the meaning behind written and spoken communication. </p> <p> </p> <p>In this dissertation, I will motivate neural-symbolic representations for dealing with these challenges. On the one hand, symbols have inherent explanatory power, and they can help us express contextual knowledge and enforce consistency across different decisions. On the other hand, neural networks allow us to learn expressive distributed representations and make sense of large amounts of linguistic data. I will introduce a holistic framework that covers all stages of the neural-symbolic pipeline: modeling, learning, inference, and its application for diverse discourse scenarios, such as analyzing online discussions, mining argumentative structures, and understanding public discourse at scale. I will show the advantages of neural-symbolic representations with respect to end-to-end neural approaches and traditional statistical relational learning methods.</p> <p><br></p> <p>In addition to this, I will demonstrate the advantages of neural-symbolic representations for learning in low-supervision settings, as well as their capabilities to decompose and explain high-level decision. Lastly, I will explore interactive protocols to help human experts in making sense of large repositories of textual data, and leverage neural-symbolic representations as the interface to inject expert human knowledge in the process of partitioning, classifying and organizing large language resources. </p>

While in traditional computer game playing agents were designed solely for the purpose of playing one single game, General Game Playing is concerned with agents capable of playing classes of games. Given the game's rules and a few minutes time, the agent is supposed to play any game of the class and eventually win it. Since the game is unknown beforehand, previously optimized data structures or human-provided features are not applicable. Instead, the agent must derive a strategy on its own. One approach to obtain such a strategy is to analyze the game rules and create a state evaluation function that can be subsequently used to direct the agent to promising states in the match. In this thesis we will discuss existing methods and present a general approach on how to construct such an evaluation function. Each topic is discussed in a modular fashion and evaluated along the lines of quality and efficiency, resulting in a strong agent.

While in traditional computer game playing agents were designed solely for the purpose of playing one single game, General Game Playing is concerned with agents capable of playing classes of games. Given the game's rules and a few minutes time, the agent is supposed to play any game of the class and eventually win it. Since the game is unknown beforehand, previously optimized data structures or human-provided features are not applicable. Instead, the agent must derive a strategy on its own. One approach to obtain such a strategy is to analyze the game rules and create a state evaluation function that can be subsequently used to direct the agent to promising states in the match. In this thesis we will discuss existing methods and present a general approach on how to construct such an evaluation function. Each topic is discussed in a modular fashion and evaluated along the lines of quality and efficiency, resulting in a strong agent.:Introduction Game Playing Evaluation Functions I - Aggregation Evaluation Functions II - Features General Evaluation Related Work Discussion

<p dir="ltr">We now live in a world where we can reach people directly through social media, without relying on traditional media such as television and radio. The landscape of social media is highly distributed, as users generate and consume a variety of content. On the other hand, social media platforms collect vast amounts of data and create very specific profiles of different users through targeted advertising. Various interest groups, politicians, advertisers, and stakeholders utilize these platforms to target potential users to advance their interests by adapting their messaging.  A significant challenge lies in understanding this messaging and how it changes depending on the targeted user groups. Another challenge arises when we do not know who the users are and what their motivations are for engaging with content. The initial phase of our research focuses on comprehensively understanding users and their underlying motivations, whether practitioner-based or promotional. Gaining this understanding is crucial in reshaping our perspective on the content disseminated by these users. Step beyond that, assuming the identification of the involved parties, this study aims to characterize the messaging and explore how it adapts based on various targeted demographic groups. This thesis addresses these challenges by developing computational approaches and frameworks for (1) characterizing user types and their motivations, (2) analyzing the messaging based on topics relevant to the users and their responses to it, and (3) delving into the deeper understanding of the themes and arguments involved in the messaging.</p>