ControlNet Depth & Canny Workflow for Consistent Composition

When to use ControlNet in Stable Diffusion

ControlNet is the composition lock in Stable Diffusion workflows: prompts handle style and materials, while the control map handles structure. For SDXL product tables, room perspective, architecture edges, and character pose, ControlNet is usually more reliable than prompt stacking alone.

Quick picks:

• Need stable scene layout (rooms, product stands, landscape depth) → start with Depth
• Need hard edges (architecture, packaging outlines, silhouettes) → use Canny
• Need pose lock → use OpenPose
• Commercial product shots often blend Depth + Canny at moderate weights

For facade mood studies that stay text-first—and when to escalate to Depth/Canny—see SDXL architecture exterior prompts.

Understanding these maps also helps when you use this site’s Stable Diffusion generator and tools.

Depth vs Canny vs OpenPose

Preprocessor	Best at	Weaker at
Depth	Spatial layers, product placement	Tiny text edges
Canny	Clean outlines, buildings, packaging	Over-locking soft organic materials
OpenPose	Standing/sitting pose skeletons	Complex scene perspective (pair with Depth)

Practical order: Depth for big structure, Canny for hard edges if needed, OpenPose for people. Don’t max all three at once.

Starting weights (SDXL)

Preprocessor	Suggested weight	Notes
Depth	0.6–0.9	Lower if the image feels plastic
Canny	0.4–0.7	Too high locks style
OpenPose	0.8–1.0	Best for character pose lock
Depth+Canny stack	~0.55 / 0.45	Common product starting point

Higher is not always better. In Stable Diffusion, oversized ControlNet weight flattens style tokens and makes results look like a recolored depth map.

Repeatable workflow (txt2img / img2img)

1. Prepare a base: generate a usable layout with Stable Diffusion text-to-image, or upload a sketch/product photo.
2. Extract maps at the target resolution (match final output size).
3. Write the prompt for style, materials, and light—without fighting the pose/layout.
4. Tune separately: never change CFG and ControlNet weight in the same step; lock seed first.
5. Save a preset: model + weights + prompt skeleton, then only rewrite materials next time.

Example: e-commerce headphones—Depth 0.7 locks placement while you swap matte plastic vs brushed titanium. That is the commercial value of Stable Diffusion + ControlNet.

Parameter pairing

Variable	Suggestion	Why
CFG	5–8 on SDXL	Don’t crank CFG on top of strong ControlNet
Steps	25–35	Structure is already guided
Aspect ratio	Match the reference photo	Prevents stretched maps
Negatives	Keep watermark/text/blurry	Structure lock ≠ clean image

For portraits, pair with the SDXL photoreal portrait guide. If you also stack LoRAs, keep total weight sane—see the LoRA stacking guide.

Common mistakes

• Weight too high → over-constrained, lifeless images
• Prompt fights the pose (“sitting” while OpenPose stands)
• Resolution mismatch between map and latent size
• Stacking too many ControlNets without lowering individual weights
• Empty prompts while relying only on ControlNet → materials and lighting still fail
• Base mismatch → use SDXL-compatible ControlNet pipelines for SDXL work

Debug order

1. Disable all ControlNets; confirm the pure text-to-image prompt works.
2. Enable Depth only; raise from 0.5 until structure is enough.
3. Add Canny from 0.35 if hard edges are missing.
4. If edges still slip, check crop/resolution before adding more weight.
5. If style is weak, lower control weight or enrich material/lighting terms—don’t only raise CFG.

Practice

Take one product photo, run depth-only at 0.7, then depth+canny at 0.55/0.45. Compare edge fidelity vs style freedom, then save your favorite preset for the Stable Diffusion generator. Pull style lines from the prompt library if needed.

Once Depth and Canny click, Stable Diffusion stops feeling like luck and starts feeling like a reproducible pipeline: structure locked, style swappable.