LoRA Stacking Without Breaking Your Image

Why LoRA stacks break so easily in Stable Diffusion

LoRAs are the everyday “light plug-ins” of the Stable Diffusion ecosystem: inject a character face, art style, or clothing detail into SDXL text-to-image without swapping the whole checkpoint. Stack several at once and identity, style strokes, and base priors start fighting—faces melt, styles muddy, or a missing trigger word means the LoRA never fired.

This guide gives a reusable stacking system you can validate against the models page and the Stable Diffusion generator.

Rule of thumb: cap total weight first

Keep total LoRA weight under about 1.4–1.6 unless you already know the combo is stable.

Examples:

• Safe: 0.75 + 0.55 + 0.25 = 1.55
• Risky: 0.9 + 0.8 + 0.6 = 2.3 (styles often turn to mud)

If a character LoRA must sit high (e.g. 0.9), push style/detail LoRAs down to 0.3–0.45 so identity still has room.

Recommended load order

1. Character LoRA (identity) — lock “who”
2. Style LoRA (aesthetic) — then “what look”
3. Detail LoRA (optional, low) — materials / eyes / wardrobe last

Identity is the easiest thing for a style LoRA to erase in Stable Diffusion stacks. Anchor the character first so rollback is simple.

Trigger words and prompting

Many LoRA cards require a trigger word. Miss it and the adapter may as well be off. Do this:

1. Copy the official trigger from the LoRA card.
2. Place it early in the positive prompt.
3. Avoid conflicting style language (photoreal face LoRA + wall-to-wall anime tags).
4. Keep negatives maintained (blurry, bad hands, watermark…). See the negative prompts guide.

For SDXL portrait skeletons, use the photoreal portrait guide.

Example stack

character LoRA @ 0.75 + style LoRA @ 0.55 + detail LoRA @ 0.25

Text-to-image test loop:

1. Character LoRA only @ 0.75 — confirm identity.
2. Add style from 0.4; nudge toward 0.55 if needed.
3. Detail LoRA from 0.2; keep only if you see clear gain.
4. Lock seed, SDXL base, and sampler; change only LoRA weights.

Pairing with CFG, steps, and base model

Variable	Suggestion	Notes
Base	Must match	SDXL LoRA ≠ SD 1.5; don’t casually force onto SD 3.5
CFG	Slightly lower than no-LoRA habit	Face collapse → lower CFG before raising LoRA
Steps	25–35	Too few steps and LoRA detail never settles
Multi-LoRA	Total ≤ 1.6 to start	Cut style first, then detail

Choosing between SDXL and SD 3.5? Read SD 3.5 vs SDXL. Need pose/composition lock? Add ControlNet via the ControlNet workflow.

Debug checklist (in order)

1. Enable one LoRA at a time to isolate identity vs style failure.
2. Face collapse → lower CFG and slightly cut style weight.
3. Confirm base match (SDXL LoRA on SDXL).
4. Confirm trigger words are in the positive prompt.
5. Check positive/negative conflicts (want realism but never blocked anime, etc.).
6. Still unstable → fall back to character + one style; drop the third LoRA.
7. Read licenses before commercial use—character LoRA, style LoRA, and base rights can differ.

Failure pattern table

Symptom	Common cause	Try first
Not the character	Missing trigger / low weight / wrong base	Add trigger, raise to 0.7+, verify base
Right face, dirty style	Total weight too high	Cut style/detail weights
Plastic / hard face	High CFG + multi-LoRA	Lower CFG; drop detail LoRA
Wild variance	Unlocked seed / too many changes	Lock seed; single-variable iterates
Broken hands	Hands too prominent in framing	Reframe or inpaint later

Next steps

Follow Models guidance, then test on the Stable Diffusion generator; start prompts from the prompt library if needed. Once “weight cap + load order + single-variable debug” is muscle memory, LoRA stacking stops being superstition and becomes a controllable Stable Diffusion advanced skill.