ControlNet funny result

[j2l]

Describe the bug

Using a ControlNet Rig in Blender + AI render with ControlNet more recent control_v11p_sd15_openpose.pth (https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main) I get this funny result:

(A future statue erected to AI maybe?)

My settings:

Any idea what I missed?

To reproduce

Install openpose from https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main
add a openpose rig + mesh
setup as above
F12

Error log

none, but ControlNet openpose should work

Environment

• Blender version (upper right corner of splash screen): 3.3.7
• AI Render version (find in Preferences > Add-ons): 0.7.9
• Operating system (Windows/Mac/Linux): Ubuntu 22.04

Screenshots/video

No response

Additional information

No response

[j2l]

Please @steve-salmond could you please explain how you get some ControlNet result?
I'm pretty sure I'm doing something wrong here 😄

[benrugg]

Hmm, that's no good. Thanks for posting this. I'm not sure what's going wrong. While developing this feature, I tested with the depth model and one of the architectural ones, but I didn't actually test with openpose.

If I had to guess, I think there's a possibility that the preprocessor is the problem. I think the point of the preprocessor is to take a regular image (like one of a person) and then turn it into an image for the model to use (like a pose).

So maybe try without any preprocessor.

(I am traveling right now, without computer, so I won't be able to help much until later in the week. But feel free to send me more info on whatever you try!)

[j2l]

Any news?

[benrugg]

Has anyone tried this while setting the preprocessor to none?

[j2l]

Any news?
AFAIK, prepocessor to none wouldn't treat the skeleton as a pose.

[benrugg]

AFAIK, prepocessor to none wouldn't treat the skeleton as a pose.

I actually think that's exactly the way ControlNet works. With a preprocessor, it converts a regular image into a skeleton image. Without it, it uses the image input as the skeleton image.

I could be wrong - I haven't tested this with the openpose model. But I think this would be the best next step for you to try.

[j2l]

Ok, I see what you mean.
Nope, it keeps on using the pose as image, not as pose:

I guess my process is wrong.
Are you successful using a pose?

[j2l]

You're right about the preprocessor, in webui @benrugg , it is not needed.

t works only in webui, if I select Control Type "All" or "openpose":

But if I select Control Type as "canny":

So I guess, we're missing a selection of priority_order in your code?

[benrugg]

Ok, yeah, there must be a different way that Automatic1111's API is expecting the input. I will check it out and try to fix it when I have a chance. I am in the middle of a launch on a different (unrelated) product right now, so it might be a few weeks before I can dedicate time to this.

[j2l]

I understand @benrugg ... and wait 😄

[j2l]

Would you please let me know about an ETA @benrugg ?

[benrugg]

Hey, sorry, I can't give an ETA. Like I mentioned, I'm head down on a different project right now, so it will probably be a while. If you can find anyone else who can experiment with this, I'd be happy to review a PR.

[j2l]

Sure, but I need some little guidance first 😄
Do you also think we miss priority_order ?
Quickly, do you have hints how to add it in the addon UI?
Thanks

[benrugg]

I think it will likely take some experimentation to figure out what data and settings need to be passed to Automatic1111's API. My intuition is that it's not priority_order, but it's possible. I think someone will need to dive into their API and my python code.

[j2l]

I can try to dive in, I'd just need to know how to add a setting like priority_order in the UI.

[benrugg]

I think for better or worse, the UI would probably be the last step in the process. Most of the work would be in python. Blender has a lot of guides, but it takes a lot to dive into it unless you already know how to code.

[j2l]

OK. Let me know when you tackle this one.