Accept 2D motion data from webcam

[mholt]

Thanks for replying on HN. I think it'd be super fun to do this with the family: scan in the kids' drawings, turn on our webcam, and start moving around so that the drawn characters mimic our motions. Then all we have to do is record that, overlay some audio / voice, and we have ourselves a little makeshift family-friendly movie studio :)

Understandably this would be a ton of work but I just wanted to put it out there if there's interest!

[justinduynguyen]

i'm interested into this too, hope it will be implemented soon :D

[Aditya-Naresh]

Sounds like a nice feature to work on. I think it's possible to reuse some of the existing feature designs from other meta projects and slightly tweak it to fit it here for the above purpose. Easier said than done :)

[justinduynguyen]

it sounds like realtime motion capture from video webcam Vào Th 2, 17 thg 4, 2023 vào lúc 14:19 Aditya Naresh < ***@***.***> đã viết:

…

Sounds like a nice feature to work on. I think it's possible to reuse some of the existing feature designs from other meta projects and slightly tweak it to fit it here for the above purpose. Easier said than done :) — Reply to this email directly, view it on GitHub <#98 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEKC2O2ZZOKZ2R6QRGDVHPTXBTVIDANCNFSM6AAAAAAW66JYWQ> . You are receiving this because you commented.Message ID: ***@***.***>

[hjessmith]

This seems like a good extension, but it'll be a bit of work and require some refactoring. In essence, the BVH class needs to be refactored into a 'Motion Source' class, which can be subclassed to support both 2D and 3D data. In addition, 'Motion Source' class can be subclassed to serve both full motion clips (as the BVH class current does) as well as streaming sources of motion data.

[yujinqiu]

@hjessmith this feature will make AnimatedDrawings much more fun and solve the most difficult part of motion data capture,
plz consider support it.

[huangxin168]

The job is amazing! but the rokoko hardware/software seams to be the bigest barrier.
looking forward to a raw wabcam capture support...

[LordMoMA]

I like this idea, but the webcam should be positioned well to achieve this. A rectangular shape can be placed in the center, indicating people should move within the rectangle in order to generate good quality animations.

This limit might reduce some development difficulties.

[thhung]

interested on this one as well.

[nashuii]

I'm also looking forward to this feature.

[aiXia121]

great work,

[Brian417-cup]

Hello everyone, I have successfully exported a great demo that you can only provide drawings and video as input. The code can automatically generate 3D pose and corresponding BVH file from 2D data. All processes are offline and end-to-end in a virtual conda environment. The result is like this:

I have also successfully removed redundant OpenMMLab dependencies that may cause some problems in docker. That means you don't generate intermediate results from torchserver.

[mholt]

That's amazing! Looking forward to trying it :)

[hjessmith]

@Brian417-cup that's exciting to hear! I'll be on the lookout for a PR from you. I'm sure folks will really enjoy it.

[Brian417-cup]

@hjessmith . Sure, I'm very glad to share the complete scripts. I also believe it will inspire others to find better solutions, especially for the projection method proposed in the thesis (Currently, it's static and pre-defined in the config file).

[hjessmith]

Thanks for submitting the PR. This is a big one- it's going to take me a while to sort though, and I foresee some potential licensing issues with Animated Drawings' MIT licensing if I merge the PR as is. I'll have to investigate the proper path forward here.

In the meantime, I'll drop a link to the PR here in case anyone wants to download it directly and give it a try:
#277