Blog

Biometry Research Group (BioRG) @ IPLAB



When Can We Trust AI-Based Biometric and Multimedia Evidence?

Adversarial Machine Learning (AML) is usually framed as a problem of model robustness. In forensic applications, the more relevant question is whether an entire evidence-processing pipeline – from acquisition to the conclusion presented to an investigator or a court – can remain trustworthy when someone is actively trying to subvert it.

From model robustness to evidence integrity

AI is increasingly embedded in forensic workflows: biometric identification, presentation attack detection (PAD), deepfake detection, multimedia authentication and other forms of digital evidence analysis. None of these systems operates in isolation. Evidence passes through a chain of stages – acquisition and preprocessing, feature extraction, inference and decision, and finally interpretation and presentation – and each stage consumes the output of the previous one.

Adversarial Machine Learning has a well-established vocabulary for attacks across the lifecycle of a model, distinguishing training-time from deployment-time threats and characterising attackers by their goals, knowledge and capabilities [1,2]. What is less developed is how these threats compose when models are chained inside a forensic process. An attacker does not need to “break” every component: it may be enough to exploit one stage and let the error propagate downstream, where it is inherited, and sometimes reinforced, by later components. In our recent position paper we describe this as a systemic threat: AML can compromise the inferential machinery used to analyse evidence, not merely an individual prediction.

An illustrative scenario

Consider a hypothetical but technically plausible case. A manipulated face video is submitted as evidence. The manipulation is post-processed to suppress the frequency-domain traces a deepfake detector relies on, so the detector labels it as authentic. An explanation method then produces a heatmap highlighting natural facial regions, which appears to corroborate the verdict. A face recognition system subsequently links the face to an identity, and the analyst's report combines these outputs into a single, apparently consistent conclusion.

Each component may have performed well on its benchmark. Yet the conclusion is wrong, and the components that were supposed to provide independent assurance – the detector and the explanation – are precisely the ones that were circumvented. This is the core of the problem: component-level validation does not imply chain-level integrity.

A broader attack surface

The threats relevant to forensic AI are distributed across the whole lifecycle of the evidence and of the models that process it:

  • Acquisition. Physical presentation attacks target biometric sensors directly [4], while face morphing can produce a single image that is successfully matched against more than one individual, with direct implications for identity documents [3].
  • Inference. Evasion attacks manipulate inputs to cross a decision boundary. Face recognition has been shown to be vulnerable both to physically realisable attacks, such as adversarial eyeglass frames [5], and to query-based black-box attacks [6]. PAD algorithms, which are meant to protect recognition systems, have their own adversarial vulnerabilities [7]. Deepfake detectors can be evaded through white- and black-box perturbations [8] and through targeted manipulation of the frequency-domain traces they exploit [9,10].
  • Generation. The shift from GAN-based to diffusion-based generators exposes generalisation gaps in detectors trained on earlier generation pipelines [11], and the field is still consolidating how to evaluate detectors against rapidly evolving generators [12].
  • Training. Poisoning and backdoor attacks can compromise a model before it ever processes operational evidence: a small number of poisoned training samples can implant a physical trigger in a face recognition model [13], and gray-box poisoning of intermediate training data has been shown to degrade biometric authentication models [14].

The weakest link may not be the model

Forensic systems are typically evaluated component by component: accuracy, robustness, detection rates, explainability. When components are coupled, strong performance at each stage does not guarantee the integrity of the final conclusion.

Multimodal biometrics illustrates this clearly. Combining traits generally improves accuracy, and is often assumed to improve security as well. However, it has been shown that spoofing a single modality can be sufficient to evade common fusion rules [4], and adversarial attacks can be designed against multi-biometric authentication systems as a whole [15]. Fusion, in other words, is itself an attack surface.

The same applies to explainability. Explanations are often presented as a safeguard that allows a human expert to verify a model's reasoning. But post-hoc methods such as LIME and SHAP can be deliberately fooled into hiding a model's actual behaviour [16], and gradient-based explanation maps can be manipulated almost arbitrarily by imperceptible input perturbations that leave the model's output essentially unchanged [17]. An explanation is therefore not an independent check unless its own robustness has been tested.

What should “robust” mean in forensic AI?

Is a forensic AI system robust when each of its models resists adversarial attacks in isolation, or only when the entire evidence-processing chain preserves the validity of its conclusions under adversarial pressure?

We argue for the second definition. It shifts the object of evaluation from the model to the evidential claim the system ultimately supports, and makes adversarial robustness a property of the process rather than of a single classifier.

Towards evaluation that reflects forensic reality

Adopting this view has concrete methodological consequences. Building on established guidance for adversarial evaluation [1,2,18], we see at least five requirements:

  1. Explicit threat models. Every robustness claim should state the attacker's goal, knowledge (white-, gray- or black-box), capabilities and point of access in the pipeline. A claim without a threat model cannot be interpreted.
  2. Adaptive attacks. Defences should be evaluated against attackers who know the defence and adapt to it, not only against fixed, off-the-shelf perturbations [18].
  3. End-to-end metrics. Alongside per-component scores, systems should be assessed on the error rate of the final conclusion when an attack is injected at each stage, so that error propagation becomes measurable.
  4. Explanations as tested components. Where explanations are used to support expert judgement, their stability under manipulation should be evaluated like any other component.
  5. Traceability. Model versions, training data provenance and intermediate outputs should be logged so that a conclusion can be audited and, if necessary, challenged.

Why this matters now

This is not only a research concern. Under the EU Artificial Intelligence Act, AI systems used for biometric identification and categorisation, in law enforcement and in the administration of justice fall among the high-risk use cases, and providers of high-risk systems must ensure an appropriate level of accuracy, robustness and cybersecurity, including measures against data poisoning, model poisoning and adversarial examples [19]. The regulation sets the requirement; the scientific community still has to provide the evaluation methods that make it verifiable for forensic pipelines.

A research agenda for BioRG

The aim is not to suggest that AI-based forensic systems are inherently unreliable. It is to make adversarial robustness an explicit, measurable property whenever AI contributes to the authentication or interpretation of evidence. For BioRG, this defines a research agenda at the intersection of Adversarial Machine Learning, Biometrics, Multimedia Forensics, Deepfake Detection and Trustworthy AI, centred on one question: how can we establish, and quantify, trust in biometric and multimedia evidence when the computational processes used to analyse it may themselves be targeted?

References

  1. Vassilev, A., Oprea, A., Fordyce, A., Anderson, H., Davies, X., & Hamin, M. (2025). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. NIST AI 100-2e2025. National Institute of Standards and Technology.
  2. Biggio, B., & Roli, F. (2018). Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84, 317–331.
  3. Scherhag, U., Rathgeb, C., Merkle, J., Breithaupt, R., & Busch, C. (2019). Face Recognition Systems Under Morphing Attacks: A Survey. IEEE Access, 7, 23012–23026.
  4. Biggio, B., Akhtar, Z., Fumera, G., Marcialis, G. L., & Roli, F. (2012). Security evaluation of biometric authentication systems under real spoofing attacks. IET Biometrics, 1(1), 11–24.
  5. Sharif, M., Bhagavatula, S., Bauer, L., & Reiter, M. K. (2016). Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition. ACM CCS, 1528–1540.
  6. Dong, Y., Su, H., Wu, B., Li, Z., Liu, W., Zhang, T., & Zhu, J. (2019). Efficient Decision-based Black-box Adversarial Attacks on Face Recognition. IEEE/CVF CVPR.
  7. Agarwal, A., Vatsa, M., & Singh, R. (2025). On Adversarial Robustness of Face Presentation Attack Detection Algorithms. IEEE/CVF ICCV Workshops (CV4BIOM).
  8. Carlini, N., & Farid, H. (2020). Evading Deepfake-Image Detectors with White- and Black-Box Attacks. IEEE/CVF CVPR Workshops.
  9. Guarnera, L., Guarnera, F., Ortis, A., Battiato, S., & Puglisi, G. (2024). Evasion Attack on Deepfake Detection via DCT Trace Manipulation. ICPR, 157–169.
  10. Guarnera, F., Guarnera, L., Ortis, A., Battiato, S., & Puglisi, G. (2025). A Novel Adversarial Gray-Box Attack on DCT-Based Face Deepfake Detectors. IEEE Access.
  11. Corvi, R., Cozzolino, D., Zingarini, G., Poggi, G., Nagano, K., & Verdoliva, L. (2023). On the Detection of Synthetic Images Generated by Diffusion Models. ICASSP, 1–5.
  12. Amerini, I., et al. (2025). Deepfake Media Forensics: Status and Future Challenges. Journal of Imaging, 11(3), 73.
  13. Chen, X., Liu, C., Li, B., Lu, K., & Song, D. (2017). Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. arXiv:1712.05526.
  14. Al-Qudah, R., Aloqaily, M., Ouni, B., Guizani, M., & Lestable, T. (2023). An Incremental Gray-Box Physical Adversarial Attack on Neural Network Training. IEEE ICC, 45–50.
  15. Lee, M., Yoon, J., & Choi, C. (2024). Adversarial attack vulnerability for multi-biometric authentication system. Expert Systems, 41(10), e13655.
  16. Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020). Fooling LIME and SHAP: Adversarial Attacks on Post Hoc Explanation Methods. AAAI/ACM Conference on AI, Ethics, and Society, 180–186.
  17. Dombrowski, A.-K., Alber, M., Anders, C. J., Ackermann, M., Müller, K.-R., & Kessel, P. (2019). Explanations can be manipulated and geometry is to blame. NeurIPS.
  18. Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow, I., Madry, A., & Kurakin, A. (2019). On Evaluating Adversarial Robustness. arXiv:1902.06705.
  19. Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, 2024 (Art. 15 and Annex III).

This article is based on the BioRG research perspective presented in “Adversarial Machine Learning: A Systemic Threat to Biometric Forensics and Multimedia Authentication”.