On the Exploitation of Deepfake Model Recognition
WORKSHOP ON MEDIA FORENSICS - CVPR 2022
Luca Guarnera1, Oliver Giudice2, Matthias Niessner3, Sebastiano Battiato1
1 Department of Mathematics and Computer Science, University of Catania, Italy
2 Bank of Italy, Applied Research Team, Rome, Italy
3 Technical University of Munich, Germany
luca.guarnera@unict.it, giudice@dmi.unict.it, niessner@tum.de, battiato@dmi.unict.it
Schematic representation of the robust processing pipeline proposed for Deepfake Model Recognition.
At first, a dataset has to be created (a) by fine-tuning different models and thus generating images.
In general, an efficient Deepfake detector/classifier could be employed at first from those available in literature.
Then, an encoder (b) is trained for the recognition task on the a-priori known models.
The obtained feature vectors on a two-dimensional space shows the discriminative power of features encoded by the baseline
encoder. Finally, the first layer of the trained encoder is exploited in a metric learning approach.
ABSTRACT
Despite recent advances in Generative Adversarial Networks (GANs), with special focus to the Deepfake phenomenon there is no a clear understanding neither in terms of explainability nor of recognition of the involved models. In particular, the recognition of a specific GAN model that generated the deepfake image compared to many other possible models created by the same generative architecture (e.g. StyleGAN) is a task not yet completely addressed in the state-of-the-art. In this work, a robust processing pipeline to evaluate the possibility to point-out analytic fingerprints for Deepfake model recognition is presented. After exploiting the latent space of 50 slightly different models through an in-depth analysis on the generated images, a proper encoder was trained to discriminate among these models obtaining a classification accuracy of over 96\%. Once demonstrated the possibility to discriminate extremely similar images, a dedicated metric exploiting the insights discovered in the latent space was introduced. By achieving a final accuracy of more than 94\% for the Model Recognition task on images generated by models not employed in the training phase, this study takes an important step in countering the Deepfake phenomenon introducing a sort of signature in some sense similar to those employed in the multimedia forensics field (e.g. for camera source identification task, image ballistics task, etc).
|
Download Paper |
