Few-Shot Visual Concept Extraction for Steering Diffusion Transformers

We steer image generation by extracting visual concept directions directly from the image-side activations of a Diffusion Transformer, with only a handful of positive and negative prompt pairs.

Recent steering methods for diffusion models mostly work in the text-conditioning space or rely on costly auxiliary training. We observe that scene-level concepts such as lighting, weather and style are distributed across image tokens, while local concepts such as object colors, textures, materials and facial attributes are concentrated in specific regions. We therefore use a global difference-of-means for distributed concepts and a masked difference-of-means for local ones, which prevents local signals from being diluted by global averaging.

The resulting directions are reusable and are injected at inference for continuous steering, without any training. They give substantially stronger edits than text-encoder steering and training-free activation-editing baselines, approach trained reference-image editing models, and can be applied at patch level to steer several concepts in the same image.

A direction is extracted once and is independent of where it is applied. Different concepts can therefore be sent to different regions of the same image, in a single steering pass. Drag across each image to compare.

originalsteered

Left car blue, right car green.

originalsteered

Left teapot wooden, right teapot gold.

originalsteered

Left person beard, right person smile.

originalsteered

Two global concepts at once: anime + snowy.

We evaluate 23 concepts (global, object and face) on held-out prompts, measuring edit success with ΔVQA and drift from the original image with LPIPS, at the strongest evaluated strength of each method.

Pairs = positive/negative prompt pairs used for extraction.

Our directions give much stronger edits than the training-free baselines, and approach the trained editing models conditioned on the reference image. Five pairs are enough: 10 or 20 pairs barely change the results.

Masking the extraction improves both edit success and preservation for local concepts, most clearly for faces.

The same recipe transfers to FLUX.2-Klein 4B and to steering an existing reference image. Directions are extracted only from synthetic prompt pairs. The conditioning prompt is empty and only the steering strength changes. Unlike a single prompt-based edit, the strength of the concept is controlled continuously.

Reference

α low

α mid

α high

FLUX.2 edit

@inproceedings{quignon2026fewshot,
  title     = {Few-Shot Visual Concept Extraction for Steering Diffusion Transformers},
  author    = {Quignon, Nabyl and Dantcheva, Antitza},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}