Unlearn and Protect: Selective Identity Removal in Diffusion Models for Privacy Preservation

Diffusion models are capable of generating high-quality synthesis images with intricate identity features, but this raises privacy concerns, as personal identities may be used without consent. What if we need to remove a specific identity from an already trained model without retraining it from scratch? Inspired by the success of concept removal from generative models, we propose an approach to address the under-explored challenge of identity removal in pretrained diffusion models. Our method achieves this by aligning the image distribution of the identity to be removed with that of a target identity, ensuring the model avoids generating the specified identity. Extensive experiments, including quantitative and qualitative analyses, demonstrate that our approach eliminates the specified identity while preserving the integrity of other identities within the model, achieving a low Acc_U = 1.50% and FID_R = 15.8. Additionally, we introduce a new Selective Removal and Keep (SRK) metric based on facial recognition (FR) models, incorporating the accuracy on unlearned and retained identities, for evaluating identity unlearning in generative models, providing a comprehensive assessment of the unlearning process and its impact on model performance.