Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to retrieve multiple predictions from maDLC trained network #57

Open
FabianPlum opened this issue Nov 25, 2021 · 9 comments · May be fixed by #72
Open

Unable to retrieve multiple predictions from maDLC trained network #57

FabianPlum opened this issue Nov 25, 2021 · 9 comments · May be fixed by #72
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@FabianPlum
Copy link

FabianPlum commented Nov 25, 2021 •
edited

Hi there,

I'm working on a little Blender addon to make use of DLC-Live for multi-animal motion capture in foraging experiments where individuals freely enter and leave the scene. I'm passing cropped detections from a separate network (Yolo v4) to DLC-Live via OpenCV as the number of animals and joints are otherwise prohibitively large and not constant. The addon is taking shape and your DLC-Live implementation works great for the most part!

At the moment, however, I fail to retrieve multiple predictions at the same time.

Loading the graph and passing an image to initialise the network shows that there is only a single set of output predictions from sess with shape (num_keypoints , 3)

I have attached the pose_cfg.yaml file below, in case you can spot some issues in there that are not immediately obvious to me.
Thanks a lot for your help!

Fabi

pose_cfg.yaml.txt
@AlexEMG
Copy link
Member

AlexEMG commented Nov 26, 2021

Indeed, currently DLC-Live only supports single animals.

@AlexEMG AlexEMG added enhancement New feature or request help wanted Extra attention is needed labels Nov 26, 2021
@FabianPlum
Copy link
Author

Ah! Thanks for clarifying!
I'll try to find a workaround with more elaborate cropping for now and report back if I find a nicer solution.

All the best
Fabi

@nizheyi
Copy link

nizheyi commented Feb 18, 2022

Indeed, currently DLC-Live only supports single animals.

May I ask for the progress of the mobilenet that support the multi-animal project? It would be of great help to my studies!

@thomasilmer
Copy link

thomasilmer commented Apr 8, 2022 •
edited

Indeed, currently DLC-Live only supports single animals.

Are there any plans for multi animal support in DLC-Live?

@sneakers-the-rat
Copy link
Collaborator

I don't know the DLC internals well enough to answer this, but wonder if any of the other maintainers could comment on what this might take? we could give a sketch for a PR maybe?

@MMathisLab
Copy link
Member

Hey there, it is currently a WIP in the lab (sorry, long list these days), but we just released 3D maDLC, which goes along this path too, see 2.2.1

@thomasilmer
Copy link

Hey there, it is currently a WIP in the lab (sorry, long list these days), but we just released 3D maDLC, which goes along this path too, see 2.2.1

Thanks for the answer.

Is there any rough estimate for when this will be done? And is there any possibility to offer financial compensation to speedup the implementation of this in DLC-live?

@niels-ilmer
Copy link

Would be great! really need this feature. @FabianPlum did you manage to make a good working workaround? How do you train the model to perform best?

@jeylau jeylau linked a pull request Jun 9, 2022 that will close this issue
Draft
2 tasks
@jeylau
Copy link
Member

jeylau commented Jun 9, 2022

Guys, work is in progress here #72.
It is functional (in my hands 😄), but will definitely require some polishing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add multi-animal support to DLCLive jeylau/DeepLabCut-live
8 participants
@sneakers-the-rat @thomasilmer @AlexEMG @nizheyi @FabianPlum @MMathisLab @jeylau @niels-ilmer