Configuration for models other than SDXL Turbo

[yairshp]

Great work!

When I use SDXL Turbo with the configurations you provided I get good results. for example:

But when I try using other models the results seems bad, including SDXL (not turbo). Here are the relevant code I used:

model_type = Model_Type.SDXL
scheduler_type = Scheduler_Type.DDIM
pipe_inversion, pipe_inference = get_pipes(model_type, scheduler_type, device=device)

input_image = Image.open("example_images/lion.jpeg").convert("RGB").resize((512, 512))
prompt = "a lion"

config = RunConfig(
    model_type=model_type,
    scheduler_type=scheduler_type,
    num_inversion_steps=50,
    num_inference_steps=50,
    num_renoise_steps=1,
    perform_noise_correction=False,
)

edit_img, inv_latent, noise, all_latents = invert(
    input_image,
    prompt,
    config,
    pipe_inversion=pipe_inversion,
    pipe_inference=pipe_inference,
    do_reconstruction=True,
    edit_prompt="a tiger in the field",
)

I believe it is a matter of specific configurations each model has. I tried to modify the configurations according to the appendix as you can see, but it still did not work. Here is the result:

Thanks in advance!

[danielgaribi]

Hi Yair,

We focused on fast diffusion models in the paper, so I didn't try to edit images in that naive way using SDXL.

I wasn't able to reproduce the same result you sent but I also got a bad result:

You can solve this by increasing the CFG, in the example is 1.0. For example with CFG=1.5:

And with CFG=2.0:

Let me know if that doesn't work for you, the only change I made in the inversion_example_sdxl.py is to change the prompt and CFG value when calling pipe_inference.

Good luck,
Daniel