3D face modeling is currently a popular area in Computer Graphics and Computer Vision. Many techniques have been introduced for this purpose, such as using one or more cameras, 3D scanners, and many other systems of sophisticated hardware with related software. But the main goal is to find a good balance between visual reality and the cost of the system. In this thesis, reconstruction of a 3D human face from a pair of stereo cameras is studied. Unlike many other systems, facial feature points are obtained automatically from two photographs with the help of a dot pattern projected on the object&amp / #8217 / s face. It is seen that using projection pattern also provided enough feature points to derive 3D face roughly. These points are then used to fit a generic face mesh for a more realistic model. To cover this 3D model, a single texture image is generated from the initial stereo photographs.

The use of realistic 3D avatars in Virtual Reality (VR) has gained significant traction in applications such as telecommunication and gaming, offering immersive experiences and face-to-face interactions. However, standalone VR devices often face limitations in computational resources and real-time rendering requirements, necessitating the optimization of 3D models through mesh simplification to enhance performance and ensure a smooth user experience. This thesis presents a pipeline that utilizes a Convolutional Neural Network to reconstruct realistic 3D human facial models in a static form from single RGB head images. The reconstructed models are then subjected to the Quadric Error Metrics simplification algorithm, enabling different levels of simplification to be achieved. An evaluation was conducted, utilizing 30 photos from the NoW dataset, to examine the trade-offs associated with employing mesh simplification on the generated facial models within the VR environment. The evaluation results demonstrate that reducing the polygon count improves frame rates and reduces GPU usage in VR, thereby enhancing overall performance. However, this improvement comes at the cost of increased simplification execution time and geometric errors, and decreased perceptual quality. This research contributes to the understanding of mesh simplification’s impact on human facial models within the VR context, providing insights into balancing model complexity and real-time rendering performance, particularly in resource-constrained environments such as mobile devices or cloud-based applications, as well as for models located farther away from the cameras. / Användningen av realistiska 3D-avatarer inom Virtuell Verklighet (VR) har fått betydande uppmärksamhet inom tillämpningar som telekommunikation och spel, vilket erbjuder en uppslukande upplevelse och ansikte mot ansikte-interaktioner. Dock möter fristående VR-enheter ofta begränsningar i beräkningsresurser och krav på realtidsrendering, vilket gör det nödvändigt att optimera 3D-modeller genom nätverksförenkling för att förbättra prestanda och säkerställa en smidig användarupplevelse. Denna avhandling presenterar en pipeline som använder sig av ett konvolutionellt neuralt nätverk för att rekonstruera realistiska 3D-modeller av mänskliga ansikten i en statisk form från enstaka RGB-bilder av huvudet. De rekonstruerade modellerna genomgår sedan nätverksförenkling med Quadric Error Metrics-algoritmen, vilket möjliggör olika nivåer av förenkling. En utvärdering genomfördes, med hjälp av 30 foton från NoW-datasetet, för att undersöka avvägningarna i samband med att använda nätverksförenkling på de genererade ansiktsmodellerna inom VR-miljön. Utvärderingsresultaten visar att minskning av polygonantal förbättrar bildhastigheten och minskar GPU-användningen inom VR, vilket förbättrar den övergripande prestandan. Dock sker denna förbättring på bekostnad av ökad tid för förenklingsexekvering och geometriska fel, samt minskad perceptuell kvalitet. Denna forskning bidrar till förståelsen av nätverksförenklingens påverkan på mänskliga ansiktsmodeller inom VR-sammanhanget och ger insikter om att balansera modellkomplexitet och realtidsrenderingsprestanda, särskilt i resursbegränsade miljöer som mobilenheter eller molnbaserade applikationer, samt för modeller som är längre bort från kamerorna.

3D human face reconstruction has been an extensive research for decades due to its wide applications, such as animation, recognition and 3D-driven appearance synthesis. Although commodity depth sensors are widely available in recent years, image based face reconstruction are significantly valuable as images are much easier to access and store. In this dissertation, we first propose three image-based face reconstruction approaches according to different assumption of inputs. In the first approach, face geometry is extracted from multiple key frames of a video sequence with different head poses. The camera should be calibrated under this assumption. As the first approach is limited to videos, we propose the second approach then focus on single image. This approach also improves the geometry by adding fine grains using shading cue. We proposed a novel albedo estimation and linear optimization algorithm in this approach. In the third approach, we further loose the constraint of the input image to arbitrary in the wild images. Our proposed approach can robustly reconstruct high quality model even with extreme expressions and large poses. We then explore the applicability of our face reconstructions on four interesting applications: video face beautification, generating personalized facial blendshape from image sequences, face video stylizing and video face replacement. We demonstrate great potentials of our reconstruction approaches on these real-world applications. In particular, with the recent surge of interests in VR/AR, it is increasingly common to see people wearing head-mounted displays. However, the large occlusion on face is a big obstacle for people to communicate in a face-to-face manner. Our another application is that we explore hardware/software solutions for synthesizing the face image with presence of HMDs. We design two setups (experimental and mobile) which integrate two near IR cameras and one color camera to solve this problem. With our algorithm and prototype, we can achieve photo-realistic results. We further propose a deep neutral network to solve the HMD removal problem considering it as a face inpainting problem. This approach doesn't need special hardware and run in real-time with satisfying results.

The prevalence of wireless networks and the convenience of mobile cameras enable many new video applications other than security and entertainment. From behavioral diagnosis to wellness monitoring, cameras are increasing used for observations in various educational and medical settings. Videos collected for such applications are considered protected health information under privacy laws in many countries. Visual privacy protection techniques, such as blurring or object removal, can be used to mitigate privacy concern, but they also obliterate important visual cues of affect and social behaviors that are crucial for the target applications. In this dissertation, we propose to balance the privacy protection and the utility of the data by preserving the privacy-insensitive information, such as pose and expression, which is useful in many applications involving visual understanding. The Intellectual Merits of the dissertation include a novel framework for visual privacy protection by manipulating facial image and body shape of individuals, which: (1) is able to conceal the identity of individuals; (2) provide a way to preserve the utility of the data, such as expression and pose information; (3) balance the utility of the data and capacity of the privacy protection. The Broader Impacts of the dissertation focus on the significance of privacy protection on visual data, and the inadequacy of current privacy enhancing technologies in preserving affect and behavioral attributes of the visual content, which are highly useful for behavior observation in educational and medical settings. This work in this dissertation represents one of the first attempts in achieving both goals simultaneously.

[Non communiqué] / In the past decade, 3D statistical face model (3D Morphable Model) has received much attention by both the commercial and public sectors. It can be used for face modeling for photo-realistic personalized 3D avatars and for the application 2D face recognition technique in biometrics. This thesis describes how to achieve an automatic 3D face reconstruction system that could be helpful for building photo-realistic personalized 3D avatars and for 2D face recognition with pose variability. The first systems we propose Combined Active Shape Model for 2D frontal facial landmark location and its application in 2D frontal face recognition in degraded condition. The second proposal is 3D Active Shape Model (3D-ASM) algorithm which is presented to automatically locate facial landmarks from different views. The third contribution is to use biometric data (2D images and 3D scan ground truth) for quantitatively evaluating the 3D face reconstruction. Finally, we address the issue of automatic 2D face recognition across pose using 3D Morphable Model

In the field of face recognition one challenge is to recognize individuals in photographs taken of the person in profile. In this project it was investigated whether 3D face reconstructions could improve the face recognition results in these cases. Images used in the experiments were historical images of persons who lived in Stockholm during the late nineteenth to early twentieth centuries. Additionally, face frontalization algorithms were applied to images taken in profile-view of the person to investigate if this idea could improve the results of 3D face reconstruction of images in profile-view. The results of the experiments showed that using 3D face reconstructions did not improve the results of face recognition. However, in most cases the individuals in the 3D face reconstructions could be recognized. Furthermore, experiments also showed that face frontalization did not improve the results of the 3D face reconstructions of images taken in profile angle. Aside from these results, the thesis demonstrates a, to the best of our knowledge, new approach to evaluate 3D face reconstruction results by using face recognition.