Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference

Researchers introduced Visual Concept Fusion, a test-time method for guiding text-to-image diffusion models with both a prompt and a reference image.

The method aligns visual features with text embeddings and fuses them during generation, enabling dual conditioning without concept-specific retraining.

Built on models like Stable Diffusion and CLIP, it uses feature alignment and optimization techniques to keep prompt adherence while incorporating image cues.

The approach offers a practical inference-time alternative to costly fine-tuning for image-conditioned generation.