There are control nets and LoRAs from community. These includes different modes described below:
Depth- Generates stylized output similar to 3D space.
Canny Edge- For strong edge detailing.
Soft Edge- Similar to canny but for smooth edges.
Pose- For replicating the pose structure for consistent generation.
Inpaint- To modify the specific part of your image.
 |
| Canny illustration |
 |
| Depth illustration |
 |
| Pose illustration |
 |
| Soft Edge illustration |
1. First of all, make sure your ComfyUI is up to date. If its not, just do it from the Manager by selecting Update All.
2. Get the Qwen basic workflow setup in ComfyUI.
A. ControlNets By InstantX
It is the unified ControlNet (released by InstantX) supporting canny, depth, pose and soft edge for Qwen Image. This ControlNet is built with 5 double blocks taken from the pre-trained transformer layers.
Its been trained from scratch on a 10M high-quality dataset (mix of general and human images) with training setup: Steps: 50K , Resolution: 1328×1328, Precision: BFloat16 , Batch size: 64 Learning rate: 4e-5 and Text drop ratio: 0.10 values.
1. Download Qwen Image ControlNet Union (Qwen-Image-InstantX-ControlNet-Union.safetensors) . Then, save this to your ComfyUI/models/controlnet folder.
2. Download Lotus Depth V1 (lotus-depth-d-v1-1.safetensors) . Then, save this to your ComfyUI/models/diffusion_models folder.
3. Download VAE (vae-ft-mse-840000-ema-pruned.safetensors). You can also use any SD1.5's VAE for this. Now, save this to your ComfyUI/models/vae folder.
4. Next, you need comfyui_controlnet_aux custom nodes (by author- Fannovel16) for preprocessing as the native nodes not fully supported. This custom node can be installed from Manager by selecting Custom Nodes Manager.
5. Restart ComfyUI and refresh it.
B. Qwen Image DiffSynth ControlNet Model Patches
This is just the patch models that mimics like the Control nets(depth, canny and inpaint) but not the actual controlnet model.
1. Download these models:
(a) Qwen image canny (qwen_image_canny_diffsynth_controlnet.safetensors)
(b) Qwen image depth (qwen_image_depth_diffsynth_controlnet.safetensors)
(c) Qwen image inpaint (qwen_image_inpaint_diffsynth_controlnet.safetensors)
2. Save these into your ComfyUI/models/model_patches folder. If not available then just create new folder as model_patches.
C. Qwen Image Union ControlNet LoRA
It supports canny, depth, pose, lineart, softedge, normal, openpose features.
1. Download Qwen Image Union LoRA (qwen_image_union_diffsynth_lora.safetensors) from Hugging face repository. Then, put this into ComfyUI/models/loras folder.
2. You also need comfyui_controlnet_aux nodes (by author- Fannovel16) for preprocessing as the native nodes not fully supported. This custom node can be installed from Manager by selecting Custom Nodes Manager.
3. Restart ComfyUI and refresh it.
Workflow
1. All the workflows can be found from our Hugging Face repository.
(a) Qwen Image Controlnet Patch (Qwen_Image_Controlnet_Patch.json) for depth, canny, inpaint features.
(b) Qwen Image Controlnet by InstantX (Qwen_Image_InstantX_Controlnet.json) for canny, depth, pose and soft edge features.
(c) Qwen Image Union Control Lora (Qwen_Image_Union_Control_Lora.json)
2. Drag and drop into ComfyUI.
All the modes (canny, depth, pose and soft edge) already embedded into single workflow. Just enable/disable using shortcut keys Ctrl+B.
If you want to use the comfyui_controlnet_aux custom nodes just search it AIO Aux Preprocessor(by double clicking on clear canvas) and connect it to Load Image node, PreviewImage node and the Apply ControlNet node.
You need to use the same resolution value as uploaded into your Load Image node. This will make your output not to get mismatched.
 ){kind=link}