Control ControlNet: Multidimensional Backdoor Attack Based on ControlNet

EasyChair Preprint 13980, version 1

Abstract

Stable Diffusion (SD) has demonstrated remarkable performance in the realm of text2image generation. Furthermore, by appending additional conditions, such as the canny edge image, depth map and pose skeleton, can impose supplementary constraints on the generated images. Nevertheless, these conditions could render the model susceptible to subtle backdoor attacks. In this paper, we propose a backdoor attack method involving a hybrid injection strategy, which includes the first use of adversarial adjustments to text encoders and the first use of multi-dimensional composite triggers. Attackers can backdoor the ControlNet to generate various images they expected by injecting backdoors into the additional conditions and text prompts. In comparison to existing methods, the experimental results shows our approach has greater levels of secrecy and semantic robustness. In the ablation study, we investigated the impact of using different dimension triggers and non-Adversarial text encoder on the evaluation metrics. Our code is available at https://github.com/paoche11/ControlNetBackdoor.

Keyphrases: Backdoor Attacks, ControlNet, Stable Diffusion models, backdoor attack method, backdoor attacks on sd models, backdoor attacks on stable diffusion models, diffusion model, diffusion models

@booklet{EasyChair:13980,
  author    = {Yu Pan and Jiahao Chen and Lin Wang and Bingrong Dai},
  title     = {Control ControlNet: Multidimensional Backdoor Attack Based on ControlNet},
  howpublished = {EasyChair Preprint 13980},
  year      = {EasyChair, 2024}}