Controlnet inpainting change too much of masking area about diffusers HOT 5 OPEN

I don't quite understand your whole problem because you're mixing a lot of things in the same issue.

Compel or textual inversion are unrelated to the mask so they don't have anything to do with how the unmasked area is affected.

Also if you're changing the whole background, why are you using a controlnet? which controlnet are you using and why?

I think it all comes to how good or bad your mask is, for example if I use this mask I get this result:

mask	result

Doesn't look too bad, it was a just a quick test without controlnet or anything else, just inpainting with a simple mask. The clothes or the color doesn't change at all because of the mask.

Also just a technicality, but in my opinion when you're changing the whole background is outpainting and not inpainting and to get good results you'll need to do a little more than with inpainting.

from diffusers.

I don't quite understand your whole problem because you're mixing a lot of things in the same issue.

Compel or textual inversion are unrelated to the mask so they don't have anything to do with how the unmasked area is affected.

Also if you're changing the whole background, why are you using a controlnet? which controlnet are you using and why?

I think it all comes to how good or bad your mask is, for example if I use this mask I get this result:

mask result

Doesn't look too bad, it was a just a quick test without controlnet or anything else, just inpainting with a simple mask. The clothes or the color doesn't change at all because of the mask.

Also just a technicality, but in my opinion when you're changing the whole background is outpainting and not inpainting and to get good results you'll need to do a little more than with inpainting.

this is text 2 image controlnet pipeline
controlnet is inpainting

before i use inpainting model without controlnet inpainting, but still got bad result
and finetune inpainting is hard with tool(onetrainer)

how possible to get result like yours

i will try your mask

do you use

output_image = pipe.image_processor.apply_overlay(mask_image1, input_image, output_image)

i use your mask still

# process mask

def make_inpaint_condition (image, image_mask):
    image = np.array(image.convert("RGB")).astype(np.float32) / 255.0
    image_mask = np.array(image_mask.convert("L")).astype(np.float32) / 255.0

    assert image.shape[0:1] == image_mask.shape[0:1], "image and image_mask must have the same image size"
    image[image_mask > 0.5] = -1.0  # set as masked pixel
    image = np.expand_dims(image, 0).transpose(0, 3, 1, 2)
    image = torch.from_numpy(image)
    return image

# pipe

pipe = StableDiffusionControlNetPipeline.from_single_file(
  '/home/zznet/workspace/stable-diffusion-webui/models/Stable-diffusion/realisticVisionV60B1_v51VAE.safetensors',
  torch_dtype = torch.bfloat16,
  controlnet = [
      depth_control,
      inpaint_control,
  ],
)

pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas = True, algorithm_type = 'sde-dpmsolver++', euler_at_final = True)

pipe.enable_freeu(s1 = 0.9, s2 = 0.2, b1 = 1.2, b2 = 1.4)

pipe.to('cuda')

pipe.load_textual_inversion([
  'easynegative.safetensors',
  './FastNegativeV2.pt'
], token = [
  'wtf0',
  'wtf1',
])

# sample

output_image = pipe(
    image = [
        depth_control,
        inpaint_control,
    ],
    prompt = prompt,
    negative_prompt = 'wtf0, wtf1, dof, dummy, mannequin',
    width = renderWidth,
    height = renderHeight,
    num_inference_steps = num_inference_steps,
    guidance_scale = guidance_scale,
    guidance_rescale = 0.5,
    # generator = _generator,
    clip_skip = 0,
    controlnet_conditioning_scale = [
        0.0, # disable for test
        0.999,
    ],
).images[0]

# output_image = pipe.image_processor.apply_overlay(mask_image1, input_image, output_image)

from diffusers.

I'm literally using the the basics:

base = load_image("original.jpeg")
mask = load_image("mask.png")

inpaint = pipe(
    prompt,
    width=1024,
    height=1280,
    image=base,
    mask_image=mask,
    guidance_scale=10,
    strength=0.99,
    num_inference_steps=50,
).images[0]

Also as I don't see the need of controlnet, I'm using just StableDiffusionXLInpaintPipeline. Are you using an inpainting model?

Also I'm using SDXL that's why you see that the quality it's better too, even though I reduced a bit the image size so it doesn't take too long to generate.

Edit: I now see with your edit that you're not using an inpainting model, probably that's your solution.

from diffusers.

re: Edit: I now see with your edit that you're not using an inpainting model, probably that's your solution.

i did use inpainting model, but the mask area still changed, i will try again
after i test with sdwebui(text2image + inpainting controlnet), so i switched
i did use sd 1.5 inpainting and sdxl inpainting
the result is degraded, anyway i will try again

sd 1.5 inpainting with controlnet depth map

sd 1.5 inpainting without controlnet depth map

from diffusers.

This is my test switching just the model to SD 1.5

So I think that's the difference between SD 1.5 and SDXL, the model is less capable of blending the background, also this should be expected, I had to lower the resolution to 512x640 which is half of what I used for SDXL.

from diffusers.