Remove Objects & Relight Image with Latent Bridge Matching

Transforming images haven't been so easier. Relighting model uses Bridge Matching techniques in latent space to convert images released by JasperAI. Unlike previous methods that require multiple processing steps, Latent Bridge Matching (LBM) accomplishes impressive transformations in just one inference step. 

You have been familiar of lighting effect that we have already explained in our IC Light tutorial. Yes, this one gives you similar kind of effect to you image.

You can use LBM for so many different tasks: 

-remove unwanted objects from your photos

-generate realistic normal and depth maps

-relight objects in your images that looks natural.

Reference- Jasper AI official page

These provides you to manage and control on Image relighting techniques, advanced image restoration, seamless object removal, lighting and shadow generation with Normals and Depth Maps estimation

They have also developed a conditional framework that gives you even more control. This means you can precisely adjust lighting conditions and shadow generation to get exactly the effect you are looking for. 

The code is released under the Creative Commons BY-NC 4.0 license, so researchers and developers can experiment with it for non-commercial projects. You can find more in-depth information by accessing their research paper.

Table of contents:

Installation

1. First, you need to install and setup ComfyUI and get the basics of ComfyUI if you do not know what is it.

2. Move inside ComfyUI/custom_nodes folder. Clone the repository using the command prompt by typing the following command:

git clone https://github.com/kijai/ComfyUI-LBMWrapper.git

3. Now, download the LMB relighting model from JasperAI's Hugging Face repository and save it into your "ComfyUI/models/diffusion_models" folder.

4. Restart ComfyUI.

Workflow

1. After installation you will get the workflow inside your "ComfyUI/custom_nodes/ComfyUI-LBMWrapper/example_workflows"folder.

2. Drag and drop into ComfyUI.

(a) Load background image

(b) Load Subject image. 

For instance- Lets say want to design a promotional image of soft drink. We uploaded the diet coke as the subject image.

(c) Load the downloaded LBM relight model.

(d) Set the background image resolution.

This is the background image we uploaded. You also need to choose your background image wisely that suit your subject image.

(e) Set the Sampler steps from the LBM Sampler node. 

(f) Set the subject image resolution. This is important as the result will directly depend on your subject and mismatch of your subject image with background will create somewhat inaccurate results. 

(g) Click Run button to initiate execution

This is our first attempt that turned out to be wrong. You will get these type of result if you set wrong image resolution of your subject and background image. 

Then, we shirked the subject image without losing its quality using Photoshop, retried and this is what we got. Not, so perfect, but  its good to be used for object removal, background removal etc.

We realized that its not that perfect as IC light that works well as compared with this but its wonderful that JasperAI has tried some thing new that is really interesting. Well, we can see future potential if they try to improve this project.