Hybrid Video Input in 2D/3D Mode #185
Replies: 6 comments 10 replies
-
Thank you very much for this feature. I tried it, I can't get results as smooth as videos on your youtube channel but we can clearly see the possibilities it offers. |
Beta Was this translation helpful? Give feedback.
-
@reallybigname I keep testing your feature which i like more and more :) 1-2x-RIFE-RIFE4.0-50fps.mp4I think i have found some good settings, there are still problems with consistency with background, but i think it is irremediable when using videos with black backgrounds. |
Beta Was this translation helpful? Give feedback.
-
@reallybigname Is this new option a game changer in term of tracking or result is just different ? Can not wait to see it ! Here are 3 different renderings. Only the "hybrid_comp_mask_type" setting has been changed for each one. hybrid_comp_mask_type : Depth HybridDepth.mp4hybrid_comp_mask_type : Video Depth HybridVidDepth.mp4hybrid_comp_mask_type : Difference HybridVidDif.mp4hybrid_comp_mask_type : Blend HybridVidBlend.mp4hybrid_comp_mask_type : None HybridNone.mp4For now and for me the best "consistent results" are obtained with "None" & "Difference" settings |
Beta Was this translation helpful? Give feedback.
-
THIS IS BADASS! |
Beta Was this translation helpful? Give feedback.
-
@reallybigname looks awesome! For some reason I can't get it to work with video input though. It seems to me that it only works in 2D & 3D animation mode (at least that what it says in automatic1111 gui), did I get this right? |
Beta Was this translation helpful? Give feedback.
-
Hi everybody! @Funofabot encouraged me to introduce myself.
I created a mod for the Deforum notebook that composites video into normal 2D/3D animation mode. It has different modes for compositing:
After I made the compositing, I wanted more. When you mix video stuff and there's motion, things tend to stick to the screen rather than the environment. So, I developed some code for copying motion from video and applying the motion to the prev_img to prime the next frame:
I have a pull request in for the main Deforum notebook. But, I am now actively working on getting my same stuff into this webui version of Deforum, and I should have a PR soon! I have the UI stuff all done, my code copied into functions, and I'm just fixing a few minor incompatibilities.
DISCO Diffusion has a RAFT implementation of optical flow, and this can bring us a lot closer to that functionality. It's still way better for doing video input than Deforum, simply because of optical flow. And, even with my mods, RAFT is still a little bit better. But, RAFT takes some significant memory, as it is AI-based. All of my motion functions are part of cv2, which is already included code, and relatively light on impact. I may try to do a RAFT implementation in the future, but I would want to preprocess the flows, then clear that model from memory for the render. The way my stuff works is that video gets broken up into jpgs at the start, but the mixing and depth and motion stuff is all done at runtime, so it resumes easy, and doesn't have to save lots of files beforehand, or build big data structures for all frames. Although, I do have an option to save all of the extra images it makes during the process, composite masks, prev_img, and even a visualization of the optical flow with colors and arrows.
I'm happy to contribute to this awesome project!
I've worked hard to try to make my stuff compact and have no impact on other code. So, hopefully ya'll will adopt it. I've been a maverick, on my own, so far. But, I'll try to lend a hand where I can as part of a team.
If you want to see some of the various experiments I've been doing, check out my 🎬 YouTube channel, especially some of the more recent videos (although I have a few recent experiments with img2img and depth model on there - ignore those)
Here's one video to get you started on the type of things my code can do:
📺 Deforum Hybrid Video Experiments - EPIC FPV Drone One Shot in the Bar
Beta Was this translation helpful? Give feedback.
All reactions