De-identification, also known as anonymization, is a broad term that refers to the process of redacting or obscuring personal data, or data that identifies an individual. In the context of video and image data de-identification, the most tangible personal information is the face. Faces are considered biometric data, thus change little compared to other aspects of an individual, such as clothing and hairstyle. Humans possess a strong innate ability to recognize faces. Computers are also adept at recognizing faces, and face recognition models are exceptionally powerful at identifying and comparing faces. Consequently, it is widely recognized as crucial to obscure the faces in video and images to ensure the integrity of de-identified data. Conventionally, this has been achieved through blurring or masking techniques. However, these methods are destructive of data characteristics and thus compromise critical attribute information such as eye gaze, pose, expression and the fact that it is a face. This is a particular problem because our society is data-driven in many ways. This information is useful for a plethora of functions such as traffic safety. One obvious such aspect is autonomous driving and driver monitoring, where necessary algorithms such as object detectors rely on deep learning to function. Due to the data hunger of deep learning, combined with society's demand for privacy and integrity through regulations such as the General Data Protection Regulation (GDPR), face de-identification, which preserves useful information, becomes significantly important.

This Thesis investigates the potential and possible limitations of de-identifying faces, while preserving the aforementioned useful attribute information. The Thesis is especially focused on the sustainability perspective of de-identification, where the perseverance of both integrity and utility of data is important. The baseline method to achieve this is through methods introduced from the face swapping and face manipulation literature, where the current research focuses on changing the face (or identity) with generative models while keeping the original attribute information as intact as possible. All while being integrated and consistent in an image and/or video. Specifically, this Thesis will demonstrate how generative target-oriented and subject-agnostic face manipulation models, which aim to anonymize facial identities by transforming original faces to resemble specific targets, can be used for realistic de-identification that preserves attributes.

While this Thesis will demonstrate and introduce novel de-identification capabilities, it also addresses and highlight potential vulnerabilities and security issues that arise from naively applying generative target-oriented de-identification models. First, since state-of-the-art face representation models are typically restricting the face representation embeddings to a hyper-sphere, maximizing the privacy may lead to trivial identity retrieval matching. Second, transferable adversarial attacks, where adversarial perturbations generated by surrogate identity encoders cause identity leakage in the victim de-identification system. Third, reconstruction attacks, where bad actor models are able to learn and extract enough information from subtle cues left by the de-identification model to consistently reconstruct the original identity.

Through this, this Thesis points out several approaches that are: 1) Controllable, meaning that the proposed models do not naively change the identity. This means that the type and magnitude of identity change is adjustable, and thus tunable to ensure anonymization. 2) Subject agnostic, meaning that the models can handle any identity or face. 3) Fast, meaning that the models are able to run efficiently. Thus having the potential of running in real-time. 4) Non-reversible, this Thesis introduces a novel diffusion-based method to make generative target-oriented models robust against reconstruction attacks. The end product consists of a hybrid generative target-oriented and diffusion de-identification pipeline that achieves state-of-the-art performance on privacy protection as measured by identity retrieval, pose retention, expression retention, gaze retention, and visual fidelity while being robust against reconstruction attacks.