Control Cameras and Consistency

[Aleon3]

Hi again Tin2Tin,
When I initially installed Palladium I was looking forward to being able to make movies that take advantage of Blender's camera controls as well as SD creative gen and all the audio gen goodness that you have integrated. You moved very fast and added many features, however it seemed as if the video SD behavior wasn't really directed toward the Blender donut demo type of animation pipeline that I expected.

I was able build a stable workflow that delivered pretty good smoothness and consistency (albeit slow) without running into out-of-memory errors. After being away from Blender and Palladium for a while, I tried to update again but it broke the workflow (creating very nice slide shows and SVD animations, but without the control of the previous config.)

I think Palladium is great anyway. I still haven't seen any solutions like this. However, I do think many would find it even more useful if you would be interested in collaborating on tweaking the extension as demonstrated in the attached videos. Perhaps with the assistance of LCM and / or TURBOXL, Palladium could become the animation solution with the most speed, flexibility and control available today.

(P.S. If I have missed something and it can already do what I suggest, I would love your help in accomplishing that goal... :-)

2024-02-22.17-57-33.mp4

palladiumDemo.mp4

[tin2tin]

Thanks for your kind words. Very few reaches out, so I often get the feeling, that I'm the only one using Pallaidium. When doing what you want to do, what part is missing now? Did you do image to image of a video, or did you do video to video?

I disabled img2img of video, since it was too flickering to be any good, imo. Video2video should still be working, but I haven't checked for a long time. If something is broken, then do a report on that specific feature, and include the steps to reproduce it.
(The SVD video stuff is too heavy to run on my poor old computer, so I do not use it much).

Btw. using Scene strips in later versions of Blender this bug makes it difficult: https://projects.blender.org/blender/blender/issues/118276#issuecomment-1124290 Though it may sound like they fixed it, the copied strip is still broken. I need to find a workaround for this.

The Speed checkbox is LCM, and yesterday I implemented the 24-hours old Lightning versions of SDXL, Proteus and Dreamshaper. The Turbo versions are also implemented, but gave poor results, maybe they have improved with the latest update of Diffusers(but Lightning may be better and maybe just as fast).

[Aleon3]

When the update broke my workflow I uninstalled it and restored the last working version, (Ver1.4 )
My flow is to stage animation in blender, render out an AVI from the timeline (72frames), import and run vid2vid. For me, the flickering isn't bothersome in many styles... I think it can be tweaked to look even better depending on framerate and theme of the movie. I am running Blender 3.6.2.

I need to do a fresh install and start from scratch with the same clips to get the best data and error to figure this out. What was the version of your last solid Vid2Vid test? I don't really need to have the latest version.

I just noticed something about my current install... Even though it is configured for Video2Video, the video model (Potatv1: the only one that works right) was originally meant to be a text2video model.

Also, Dude your work is great! I think some people may just be a little confused on how to best use it. Because it does so much... Many people become overwhelmed by the speed of AI's development and have to invest the time to catchup.

[tin2tin]

Can you make a screenshot of your settings in the sidebar?

[Aleon3]

Sorry for the delay... I have some time today. Also, I did a fresh install of Blender and v2.0 on a Win11 machine last night.

I haven't been able to make much work yet though... [I got a "deep.. (something can't remember)" error on dependency install.] but it seems to be running , just not rendering.

Here is the screenshot of my last working ver1.4 running on Win10.

[tin2tin]

Try to start Blender as admin. And install dependencies. And then send me the log of the error.

[Aleon3]

The fresh install isn't on my machine... but I took a screenshot. If you need more just let me know.

[tin2tin]

Just updated it. Try now.

[Aleon3]

Would you please re-add potat v1 to the model list ? I notice that is no longer an option...

[tin2tin]

Okay. Click uninstall dependencies, reboot your computer, try to install dependencies again, and see if mediapipe installs okay now.

[Aleon3]

Is there another way to send the full log files other than on this board? (Just trying to be careful)

[tin2tin]

Typically, the error is in the last line?

[Aleon3]

Thank you! I should be able to test later this afternoon...

[Aleon3]

Sorry for the delay... So, I finally got a clean install without errors. I had de-activate, remove , reinstall, uninstall dependencies, re-start Blender, re-install dependencies.. then success! Now I am just trying to get it to render a strip without an out-of-memory error. No luck yet.. I will keep you posted.

[Aleon3]

I was able to AnimateDiff to render, but not the others. Not really any camera control... Also, it appears that without potat v1 the out-of-memory errors are consistent. I will keep trying a variety of adjustments...