Headless SDXL: ControlNet and IP-Adapter Without a WebUI

At Samsung R&D Institute India I inherited an image generation workflow that only existed inside a browser tab. Every asset came out of the AUTOMATIC1111 WebUI: someone opened the page, typed a prompt, dragged three sliders, picked a ControlNet preprocessor from a dropdown, clicked Generate, and saved the PNG by hand. Nothing about that survives a handoff. The settings that produced yesterday's batch lived in widget state, not in a file, and there was no way to run a thousand of them overnight. My job as Team Lead was to get SDXL, ControlNet and IP-Adapter out of that tool and into a standalone headless pipeline that runs identically on a workstation, a shared GPU box, or a scheduled job.

#Why UI-coupled tooling blocks automation

The WebUI README describes it plainly as a web interface for Stable Diffusion implemented using the Gradio library, and it is the de facto reference implementation for the ecosystem at roughly 164.5k GitHub stars. That popularity is exactly the problem. Because it is the reference, extensions target it, model authors document their settings against it, and the generation logic accretes inside callbacks that expect a live Gradio session. Reading the code, the ordering of operations that actually matters (prompt parsing, scheduler selection, ControlNet hint preprocessing, high-resolution fix, VAE decode) is spread across UI event handlers and a script callback system.

Three concrete failures followed from that coupling. First, reproducibility: a request was a set of widget values, so two people producing the same asset a week apart could not prove they used the same configuration. Second, scheduling: there was no clean process boundary to call, so batch work meant a human clicking, or brittle browser automation. Third, review: nothing about a run was reviewable in a diff, because there was no artifact to diff. What I wanted was a binary-shaped interface. Give it a JSON manifest, get back images plus a record of the seeds and weights that produced them, and a non-zero exit code when a job fails.

#What SDXL actually is under the UI

Before pulling anything apart it was worth being precise about the model, because a lot of WebUI defaults are inherited from SD 1.5 and quietly wrong for SDXL. Podell et al. report an SDXL UNet backbone of 2.6B parameters, roughly three times the 860M UNet in SD 1.4 and 1.5 and the 865M UNet in SD 2.0 and 2.1. Stability AI describes the shipped SDXL 1.0 as a 3.5B-parameter base model plus a 6.6B-parameter model ensemble pipeline when the refiner is included, generating at native 1024x1024.

The detail that shapes a headless reimplementation most is the text conditioning. SDXL uses two fixed pretrained text encoders, OpenCLIP ViT-bigG and CLIP ViT-L, totalling 817M parameters, with a cross-attention context dimension of 2048 against 768 in SD 1.4 and 1.5. Any code path that builds embeddings by hand has to concatenate two prompt embeddings and carry a pooled text embedding alongside them. Get that wrong and you do not get an error, you get quietly mediocre images, which is the worst failure mode to debug.

Resolution is the other inherited assumption worth killing. SDXL's base model was pretrained at 256x256 for 600,000 steps, continued at 512x512 for 200,000 steps, and then fine-tuned with multi-aspect training on bucketed aspect ratios of approximately 1024x1024 pixel area. It is not a fixed-square model. A pipeline that hardcodes 512x512, or that only permits exact squares, is fighting the training distribution.

#Isolating the modules from the WebUI

The method was unglamorous. I traced a single Generate click through the WebUI call graph and wrote down every tensor operation that touched the latents, in order, along with every configuration value read from UI state. Then I rebuilt that ordering on top of the underlying model classes directly, with no Gradio import anywhere in the tree. Anything that turned out to be pure UI convenience (prompt token counters, preview streaming, the gallery) was dropped. Anything that changed pixels (scheduler choice and its sigma spacing, ControlNet conditioning scale and its start and end fractions, prompt weighting, VAE precision) was kept and promoted to an explicit field in a config dataclass.

The result is a build step that takes a validated config object and returns a ready pipeline. Loading is the expensive part, so it happens once per process and the pipeline is reused across every job in a run.

import torch
from diffusers import (
    StableDiffusionXLControlNetPipeline,
    ControlNetModel,
    AutoencoderKL,
    DPMSolverMultistepScheduler,
)

DTYPE = torch.float16


def build_pipeline(cfg):
    controlnet = ControlNetModel.from_pretrained(
        cfg.controlnet_id,              # diffusers/controlnet-canny-sdxl-1.0
        torch_dtype=DTYPE,
        use_safetensors=True,
    )
    # fp16-safe VAE: the stock SDXL VAE overflows in half precision
    vae = AutoencoderKL.from_pretrained(
        "madebyollin/sdxl-vae-fp16-fix", torch_dtype=DTYPE
    )
    pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
        cfg.base_id,                    # stabilityai/stable-diffusion-xl-base-1.0
        controlnet=controlnet,
        vae=vae,
        torch_dtype=DTYPE,
        variant="fp16",
        use_safetensors=True,
        add_watermarker=False,
    )
    pipe.scheduler = DPMSolverMultistepScheduler.from_config(
        pipe.scheduler.config,
        use_karras_sigmas=True,
        algorithm_type="sde-dpmsolver++",
    )

    if cfg.ip_adapter:
        pipe.load_ip_adapter(
            "h94/IP-Adapter",
            subfolder="sdxl_models",
            weight_name="ip-adapter-plus_sdxl_vit-h.safetensors",
        )
        pipe.set_ip_adapter_scale(cfg.ip_adapter_scale)   # 0.0 to 1.0

    for lora in cfg.loras:
        pipe.load_lora_weights(lora.path, adapter_name=lora.name)
    if cfg.loras:
        pipe.set_adapters(
            [l.name for l in cfg.loras],
            adapter_weights=[l.weight for l in cfg.loras],
        )

    pipe.to(cfg.device)
    pipe.set_progress_bar_config(disable=True)
    return pipe

Every value in cfg comes from a file on disk that lives in the repository next to the code. That single change is what made the work reviewable: a prompt engineer's tweak to controlnet_conditioning_scale arrives as a pull request, not as a Slack message describing where to drag a slider.

#Stacking ControlNet and IP-Adapter

Both adapters were straightforward to lift out precisely because neither one modifies the base model. ControlNet adds spatial conditioning to a frozen pretrained diffusion model by keeping a locked copy of the weights alongside a trainable copy, joined by zero-initialized convolutions, and Zhang, Rao and Agrawala report that its training is robust with both small datasets under 50k images and large ones over 1m. IP-Adapter takes the other axis: image prompting through a decoupled cross-attention mechanism that gives text features and image features their own cross-attention layers, at only 22M trainable parameters, and the paper notes it composes with existing controllable generation tools.

Practically, that means the two do not fight over the same weights, but they do fight over the same output. ControlNet pins geometry, IP-Adapter pins appearance, and if the ControlNet conditioning scale is high the reference image stops mattering. The settings that held up across our asset categories were a ControlNet scale around 0.6 to 0.7 with a conditioning end fraction near 0.8, so structure guidance releases before the final denoising steps and the model can resolve texture on its own, paired with an IP-Adapter scale around 0.5. Those two numbers became named presets in the config rather than folklore, which is the whole point of moving off the UI.

Training cost mattered too, because we wanted the option to fit our own conditioning. The ControlNet paper reports that training on a single NVIDIA A100 PCIe 40GB needs only about 23% more GPU memory and 34% more time per training iteration than optimizing Stable Diffusion itself, and that competitive depth-to-image results were reached on one RTX 3090Ti in five days with 200,000 samples. That is affordable enough that custom conditioning stays on the table for a team without a cluster.

#LoRA and DreamBooth in a batch workflow

Product work needs specific subjects, not generic ones, so the pipeline had to carry fine-tuning. I used the two approaches for different jobs. LoRA freezes the pretrained weights and injects trainable rank-decomposition matrices into each attention layer, which Hu et al. report cuts trainable parameters by 10,000x and GPU memory by 3x against full fine-tuning of GPT-3 175B with Adam, and adds no inference latency, unlike adapter-based methods. The full text gives the storage argument even more bluntly: at r=4 on the query and value projections, the checkpoint drops from 350GB to 35MB, training VRAM from 1.2TB to 350GB, with a 25% training speedup.

Those ratios are why the pipeline ships adapters, not checkpoints. A style is a small file, several can be loaded and weighted at once as in the build step above, and the base model stays a single cached artifact on every machine. Swapping a style became a line in a manifest instead of a multi-gigabyte download.

DreamBooth covered the other case: a specific object that has to stay recognisable across contexts. The project page states that typically 3 to 5 images suffice, binding the subject to a unique identifier and using a class-specific prior preservation loss to keep the class prior diverse. The prior preservation term is the part people skip and then regret, because without it the class token collapses onto the subject and the model loses the ability to draw anything else in that class. In the headless setup a fine-tune is just another CLI verb writing a versioned artifact.

# 4 subject images, prior preservation on, LoRA rank 8
sdxl-cli train dreambooth \
  --base stabilityai/stable-diffusion-xl-base-1.0 \
  --instance-data ./data/subjects/sks_speaker \
  --instance-prompt "a photo of sks speaker" \
  --class-data ./data/class/speaker \
  --class-prompt "a photo of a speaker" \
  --with-prior-preservation --prior-loss-weight 1.0 \
  --lora-rank 8 --lora-alpha 16 \
  --resolution 1024 --train-batch-size 1 --gradient-accumulation-steps 4 \
  --learning-rate 1e-4 --max-train-steps 800 \
  --mixed-precision fp16 --gradient-checkpointing \
  --output ./adapters/sks_speaker/v3

# register it so job manifests can reference it by name
sdxl-cli adapters register sks_speaker v3 ./adapters/sks_speaker/v3

#Batch inference and determinism

With the pipeline built once and adapters registered, the runner is small. The two things it must guarantee are that a job is reproducible from its manifest alone, and that a single bad job does not lose the rest of the batch. Seeds are derived deterministically from the job id rather than sampled, so re-running a manifest reproduces the same images without anyone recording a random number, and the seed is still written to the output manifest so a human can pin or perturb it later.

import json
import hashlib
import pathlib
import torch


def seed_for(job_id: str, index: int) -> int:
    digest = hashlib.sha256(f"{job_id}:{index}".encode()).digest()
    return int.from_bytes(digest[:4], "big")


def run_batch(pipe, jobs, out_dir, micro_batch=4):
    out = pathlib.Path(out_dir)
    out.mkdir(parents=True, exist_ok=True)
    manifest, failures = [], []

    for start in range(0, len(jobs), micro_batch):
        chunk = jobs[start:start + micro_batch]
        gens = [
            torch.Generator(pipe.device).manual_seed(seed_for(j["id"], i))
            for i, j in enumerate(chunk, start)
        ]
        try:
            images = pipe(
                prompt=[j["prompt"] for j in chunk],
                negative_prompt=[j.get("negative", "") for j in chunk],
                image=[load_control(j["control"]) for j in chunk],
                ip_adapter_image=[load_reference(j.get("reference")) for j in chunk],
                controlnet_conditioning_scale=[j.get("cn_scale", 0.65) for j in chunk],
                control_guidance_end=[j.get("cn_end", 0.8) for j in chunk],
                num_inference_steps=30,
                guidance_scale=6.0,
                height=j_height(chunk), width=j_width(chunk),
                generator=gens,
            ).images
        except torch.cuda.OutOfMemoryError:
            torch.cuda.empty_cache()
            failures.extend(j["id"] for j in chunk)
            continue

        for job, image, gen in zip(chunk, images, gens):
            path = out / f"{job['id']}.png"
            image.save(path)
            manifest.append({
                "id": job["id"],
                "seed": gen.initial_seed(),
                "adapters": job.get("adapters", []),
                "path": str(path),
            })
        torch.cuda.empty_cache()

    (out / "manifest.json").write_text(json.dumps(manifest, indent=2))
    return manifest, failures

Micro-batching within a single loaded pipeline is where the speed came from. The WebUI reloads and re-resolves a great deal per request because it assumes a human is waiting between clicks; a long-lived process amortises model loading, keeps the ControlNet and adapters resident, and lets the GPU stay saturated. Measured on our own hardware against the AUTOMATIC1111 baseline for the same job set, the headless pipeline ran batch inference 40% faster.

#Measuring quality with SSIM and FID

Once generation is automated, nobody is looking at every image, so quality needs a number. I used two metrics because they answer different questions, and confusing them is a common mistake.

• SSIM for regression against a known target. Wang et al. define it over luminance, contrast and structure, bounded above by 1, equal to 1 if and only if the two images are identical, computed locally with an 11x11 circular-symmetric Gaussian window of standard deviation 1.5 samples. Their worked example is the reason to prefer it over MSE: images with an identical MSE of 210 score anywhere from 0.6949 for JPEG artifacts to 0.9900 for a mean shift, because pixel error does not track what a person sees.
• FID for distribution drift after a fine-tune. Heusel et al. model Inception-v3 coding-layer activations of real and generated images as multidimensional Gaussians and report the Frechet distance between them; lower is better, and their Figure 3 shows it rising monotonically as noise, blur, occlusion, swirl and dataset contamination are added. Their protocol uses 50,000 generated images to estimate the model distribution, which is worth knowing before quoting an FID computed on a few hundred.

import torch
from torchmetrics.image.fid import FrechetInceptionDistance
from torchmetrics.functional import structural_similarity_index_measure as ssim

# feature=2048 selects the Inception-v3 final pooling layer used in the paper
fid = FrechetInceptionDistance(feature=2048, normalize=True)


def score_run(generated, references, device="cuda"):
    """generated / references: float tensors in [0, 1], shape (N, 3, H, W)."""
    fid.to(device)
    fid.update(references.to(device), real=True)
    fid.update(generated.to(device), real=False)

    per_image = [
        ssim(
            g.unsqueeze(0).to(device),
            r.unsqueeze(0).to(device),
            data_range=1.0,
            gaussian_kernel=True,
            kernel_size=11,
            sigma=1.5,
        ).item()
        for g, r in zip(generated, references)
    ]
    return {
        "fid": float(fid.compute()),
        "ssim_mean": sum(per_image) / len(per_image),
        "ssim_min": min(per_image),
        "regressions": [i for i, s in enumerate(per_image) if s < 0.80],
    }

Wiring this into the run loop turned a subjective argument into a gate. A fixed reference set goes through the pipeline on every meaningful change to a scheduler, an adapter weight or a conditioning default, and the scores are compared against the previous run. Against those SSIM and FID benchmarks the pipeline held 80% image quality accuracy on our internal target set, measured by our own scoring suite on our own reference images. The ssim_min and regressions fields matter more than the mean in day-to-day use, because a mean stays comfortable while a handful of assets quietly fall apart.

#What I would keep

The work won the Outstanding Award and the Excellence Award internally, but the parts worth carrying to any similar project are mundane:

• Trace one full request through the UI code and write down only the operations that change pixels. Everything else is presentation, and deleting it early keeps the port small.
• Promote every UI control that affects output into a typed config field committed to the repository. If a setting cannot appear in a diff, it will be lost.
• Derive seeds from job identity instead of sampling them, and write the seed back into an output manifest. Reproducibility should not depend on anyone remembering to copy a number.
• Ship adapters, not checkpoints. LoRA's storage ratios make styles cheap to distribute and cheap to combine, and DreamBooth with prior preservation covers the subject-specific cases LoRA styles do not.
• Pick metrics before you need them. SSIM guards known targets, FID guards the distribution, and neither is a substitute for the other.

The pipeline outlived the browser workflow because it could be scheduled, reviewed and re-run. That is the whole return on pulling a model out of a UI: the model was never the bottleneck, the interface was.