Every day, an overwhelming volume of new information is produced. Information Retrieval systems play a crucial role in managing this deluge and facilitate users to find relevant information. Simultaneously, the volume of scientific literature is also rapidly increasing, requiring powerful retrieval tools in this domain. Current methods of Information Retrieval employ language models based on the Transformer architecture, called Transformer-Encoder models, in this work. These models are generally trained in two phases: initially, a pre-training on general natural language data is performed, then fine-tuning follows, which adapts the model to a specific task such as classification or retrieval. Since Transformer-Encoder models are pre-trained on general natural language corpora, they perform well on these documents. However, scientific documents exhibit different features. The language in these documents is characterized by mathematical notation, such as formulae. Applying Transformer-Encoder models to these documents results in a low retrieval performance (effectiveness). A possible solution is to adapt the model to the new domain by further pre-training on a data set originating in the new domain. This process is called Domain-Adaptive Pre-Training and has been successfully applied to other domains. Mathematical Answer Retrieval involves finding relevant answers from a large corpus for mathematical questions. Both the question and the answers can contain mathematical notation and natural language. To retrieve relevant answers, the model must 'understand' the problem specified in the question and the solution of the answers. This property makes the task of Mathematical Answer Retrieval well suited to evaluate whether Transformer-Encoder models can model mathematical and natural language in conjunction. Transformer-Encoder models showed a low performance on this task compared to traditional retrieval approaches, which is surprising given the success of Transformer-Encoder models in other domains. This thesis, therefore, deals with the domain-adaption of Transformer-Encoder models for the domain of mathematical documents and the development of a retrieval approach using these models for Mathematical Answer Retrieval. We start by presenting a retrieval pipeline using the Cross-Encoder setup, a specific architecture of applying Transformer-Encoder models for retrieval. Then, we enhance the retrieval pipeline by adapting the pre-training schema of the Transformer-Encoder models to capture mathematical language better. Our evaluation demonstrates the strengths of the Cross-Encoder setup using our domain-adapted Transformer-Encoder models. In addition to these contributions, we also present an analysis framework to evaluate what knowledge of mathematics the models have learned. This analysis framework is used to study Transformer-Encoder models before and after fine-tuning for mathematical retrieval. We show that Transformer-Encoder models learn structural features of mathematical formulae during pre-training but rely more on other superficial information for Mathematical Answer Retrieval. These analyses also enable us to improve our fine-tuning setup further. In conclusion, our findings suggest that Transformer-Encoder models can be applied as a suitable and powerful approach for Mathematical Answer Retrieval.

Different road users follow different behaviors and intentions in the trajectories that they traverse. Predicting the intent of these road users at intersections would not only help increase the comfort of drive in autonomous vehicles, but also help detect potential accidents. In this thesis, the research objective is to build models that predicts future positions of road users (pedestrians,cyclists and autonomous shuttles) by capturing behaviors endemic to different road users.  Firstly, a constant velocity state space model is used as a benchmark for intent prediction, with a fresh approach to estimate parameters from the data through the EM algorithm. Then, a neural network based LSTM sequence modeling architecture is used to better capture the dynamics of road user movement and their dependence on the spatial area. Inspired by the recent success of transformers and attention in text mining, we then propose a mechanism to capture the road users' social behavior amongst their neighbors. To achieve this, past trajectories of different road users are forward propagated through the LSTM network to obtain representative feature vectors for each road users' behaviour. These feature vectors are then passed through an attention-layer to obtain representations that incorporate information from other road users' feature vectors, which are in-turn used to predict future positions for every road user in the frame. It is seen that the attention based LSTM model slightly outperforms the plain LSTM models, while both substantially outperform the constant velocity model. A comparative qualitative analysis is performed to assess the behaviors that are captured/missed by the different models. The thesis concludes with a dissection of the behaviors captured by the attention module.