Driving Behavior Analysis (DBA) plays a pivotal role in designing intelligent transportation systems, enhancing road safety, and advancing Autonomous Vehicles (AVs). Driver identification, as a key aspect of DBA, has the potential to provide unprecedented opportunities for enhanced security and driver profiling. However, the current solutions for driver identification suffer from demanding extensive data collection, limited scalability, and inadequate generalization. Furthermore, DBA is also essential for training AVs, addressing the main challenges they face: accurately perceiving their surroundings to make informed decisions and to navigate safely, and effectively handling unforeseen scenarios. In the first part of this thesis, we concentrate on behavior analysis for driver identification and verification and design two novel schemes aiming to reduce data dependency and enhance the generalization ability of existing approaches. First, we propose a novel driver identification model, called DriverRep, which reduces data dependency by presenting a fully unsupervised triplet loss training. DriverRep is the first model that extracts the latent representations associated with each driver, called driver embeddings, in an unsupervised manner. In addition, we develop a novel model to tackle driver verification and impostor detection tasks based on DBA and extracted driver embeddings. In the second part, we focus on behavior prediction for AVs and their surrounding agents. First, we tackle behavior prediction in dynamic and complex scenarios by introducing three novel prediction models for forecasting drivers intentions and behaviors at unsignalized intersections. We then address social reasoning by proposing a novel prediction model that analyzes agent interactions using graph neural networks, making the scene understanding process more informative for AVs. Our proposed prediction model, called STAG, explicitly activates social modeling with a directed graph representation while considering spatial and temporal inter-agent correlations. We further design a novel prediction system, namely CAPHA, which conditions the future behavior of agents on grid-based plans modeled as a Markov decision process and solves the prediction task via inverse reinforcement learning to produce scene compliant behaviors. Moreover, we introduce a novel goal-based prediction model, called GMP, which encodes interactions between agents and dynamic and static context information to estimate the distribution of target goals, efficiently considering the inherent uncertainty in agents behavior. Extensive quantitative and qualitative comparisons have been conducted between the developed solutions and related benchmark schemes using various scenarios and environments. The obtained results demonstrate the potential of these solutions for the understudy tasks of DBA and real-world applications.