Autonomous driving could bring profound benefits for our society. The benefits range from economic and safety benefits due to the reduction of the number of traffic accidents, to environmental gains due to reduced traffic congestion. However, the utopian future of self-driving vehicles is yet to come. To this end, we propose machine learning methods to address three pivotal aspects of autonomous driving: visual privacy, 3D depth perception, and trajectory prediction modeling. We begin by exploring the crucial issue of visual privacy within person-aware visual systems. We propose the use of depth-information to protect privacy in person-aware visual systems while preserving important foreground subjects and scene structures. We aim to preserve the identity of foreground subjects while hiding superfluous details in the background that may contain sensitive information. In particular, for an input color and depth image pair, we first create a sensitivity map which favors background regions (where privacy should be preserved) and low depth-gradient pixels (which often relates a lot to scene structure but little to identity). We then combine this per-pixel sensitivity map with an inhomogeneous image obscuration process for privacy protection. We tested the proposed method using data involving different scenarios including various illumination conditions, various number of subjects, different context, etc. The experiments demonstrate the quality of preserving the identity of humans and edges obtained from the depth information while obscuring privacy intrusive information in the background. Next, we focus on the label layout problem: AR technologies can overlay virtual annotations directly onto the real-world view of a self-driving vehicle (SDV). Autonomous vehicles operate in dynamic environments, due to the complexity of the traffic scene and the interactions between the participating agents. Overlaying virtual annotations directly onto the real-world view of a SDV, can provide additional context, such as highlighting important information or projecting the future trajectories of other participants. Designing a layout of labels that does not violate domain-specific design requirements, while at the same time satisfying aesthetic and functional principles of good design, can be a daunting task even for skilled visual designers. Presenting the annotations in 3D object space instead of projection space, allows for the preservation of spatial and depth cues. This results in stable layouts in dynamic environments, since the annotations are anchored in 3D space. In this domain, we make two major contributions. First, we propose a technique for managing the layout and rendering of annotations in Virtual/Augmented Reality scenarios by manipulating the annotations directly in 3D space. For this, we make use of Artificial Potential Fields and use 3D geometric constraints to adapt them in 3D space. Second, we introduce PartLabeling: an open source platform in the form of a web application that acts as a much-needed generic framework allowing to easily add labeling algorithms and 3D models. This serves as a catalyst for researchers in this field to make their algorithms and implementations publicly available, as well as ensure research reproducibility. The PartLabeling framework relies on a dataset that we generate as a subset of the original PartNet dataset consisting of models suitable for the label management task. The dataset consists of 1,000 3D models with part annotations. Finally, we focus on the trajectory prediction task in the context of autonomous driving. Predicting the trajectories of multiple participating agents in the context of autonomous driving is a challenging problem due to the complexity of the traffic scene and the interactions between the agents. Autonomous vehicles need to effectively anticipate the behavior of other movingparticipants in the traffic scene (human pedestrians, cyclists, animals, other moving vehicles). The task of modeling human driver behavior, as well as the interactions between the traffic participants must be addressed to enable a safe and optimized autonomous vehicle systems. There are many factors that traffic participants take into consideration in order to safely interact with other traffic participants. Human drivers have sophisticated interaction strategies that come naturally to them. Given the highly interactive nature of traffic scenarios, representing the interactions between multiple participating agents in a traffic scene in the form of a graph structure is a natural conclusion. In order to leverage the influences between multiple agents in a traffic scene, we structure the scene as a graph whose nodes represent the traffic participants. The node features are each agent’s surrounding context encoded as a raster image. For this purpose, we leveragel R-GCN (Relational Graph-Convolutional Netowrks). Then, we propose a novel Cross-Modal Attention Network (CMAN) to encourage interactions between two modalities: 1) the latent features of an ego-agent’s raster image and 2) the latent features of the surrounding agents’ influences on the ego-agent, in a manner that allows these two modalities to complement each other. / Computer and Information Science