Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Properly support positional constraints in skeletal model #142

Open
kitlith opened this issue Mar 27, 2022 · 5 comments · May be fixed by #920
Open

Properly support positional constraints in skeletal model #142

kitlith opened this issue Mar 27, 2022 · 5 comments · May be fixed by #920
Labels
Area: Skeletal Model Deals with the model of the skeleton and its pose Difficulty: Large refactor Requires modifying large sections of the codebase Difficulty: Math Heavy For something that requires some non-trivial math Priority: Normal The default priority

Comments

@kitlith
Copy link
Member

kitlith commented Mar 27, 2022 •
edited

(aka: "Let me mix slimevr knees with vive tracker hips/feet damnit")

@Louka3000 already got elbows working (:heart:) by using the controllers as a new "root", so they're in a working state. Some more enhanced calibration might be nice for it, but as long as you don't need shoulder tracking (looking at VRChat) it seems to work nicely as-is without touching calibration at all (from my limited experience using joycons)

The main thing currently is that vive trackers don't integrate well into the main skeleton. They apparently don't use their position, only maintain their rotation, and somehow affect the rotation of child trackers? (is there some code specifically for vive trackers already that needs changing?)

Short term, if we can fix whatever weird thing is going on with rotation, override the position of the hip/waist using the vive tracker's actual position, then add parameters to offset the tracker into the right place (especially for people using a tracker on their side, instead of in front) then we'll be gold. I don't know much about server internals right now, but this should require minimal touching of the internal skeleton.

An interesting option is to have the option of performing a repeat of elbow tracking -- use foot trackers as a new root, then do the knees based off of them. BVH is a big concern here, though.

I have some thoughts about root bones in general that I've mentioned in discord, but a vauge idea that might require redoing the way the skeleton works is not worth mentioning in this particular issue. tbh, I'm mostly transcribing a discussion from discord into somewhere perhaps slightly more permenant/visible/referencable.
@Erimelowo Erimelowo mentioned this issue Sep 28, 2022
Closed
@TheButlah TheButlah added the Status: Unlabeled A maintainer has not yet labeled this label Oct 5, 2022
@Erimelowo Erimelowo added Difficulty: Math Heavy For something that requires some non-trivial math Area: Skeletal Model Deals with the model of the skeleton and its pose Priority: Normal The default priority and removed Status: Unlabeled A maintainer has not yet labeled this Difficulty: Math Heavy For something that requires some non-trivial math labels Oct 5, 2022
@TheButlah
Copy link
Contributor

one thing that would be helpful is a brief explanation of how it works currently. @Louka3000 could you elaborate if you have some time?

@TheButlah
Copy link
Contributor

TheButlah commented Oct 6, 2022 •
edited

As an aside, positional constraints were one of the motivating factors with the implementation of the rust skeletal model. Feel free to reference that preliminary work if its helpful. However, the approach taken there is more advanced than probably what would be used in the java model.

If you are curious, I've taken extensive effort to document the algorithm in plain English here

@TheButlah TheButlah changed the title Trackers from the feeder app do not interact well with the main skeleton Properly support positional constraints in skeletal model Oct 6, 2022
@Erimelowo
Copy link
Member

Erimelowo commented Oct 6, 2022 •
edited

one thing that would be helpful is a brief explanation of how it works currently. @Louka3000 could you elaborate if you have some time?

Only the rotation of VRTrackers (e.g: Vive trackers) is used. Not the position.
They are thus treated the same way as normal SlimeVR trackers (IMUTrackers) minus the reset/mounting.

  • The first step to make this better is to make reset work, which should be very easy to do. (I would however wait for Add mounting reset (two-pose reset) #251 to be merged!)

  • The second step is to have some form of positional constraint work, but that will require a partial refactor of the HumanSkeleton class at the very least.

As a side note: this whole issue is something I would like to work on (and I'm probably the best person since I got Vive + SlimeVR, and I'm very familiar with how the skeleton works). If you already wrote some code for a positional constraint system in Rust, I could try my hand at porting it in Java. If not, I'll try making it work myself from scratch.

@TheButlah
Copy link
Contributor

TheButlah commented Oct 6, 2022 •
edited

There already is some code written yes, and the algorithm is in english and not rust, if that helps.

The TLDR is:

  • The skeleton uses a graph data structure with edges and nodes as distinct objects, with different data stored in each.
  • There are input&output positions&rotations
  • the solver takes the inputs and computes any missing outputs using FK. Positional and rotational constraints are just copied from the input to the output, no need to solve for those.
  • When doing the FK, it doesn't just start at the head and work its way down. It instead starts at any nodes with a known position and works "outwards" (i.e. breadth first traversal). This ensures that position is based on the closest node.

@Erimelowo
Copy link
Member

Yes, I had already read your whole documentation when you first sent it to me :p
Doing this will require refactoring, but I think the hardest part would be dealing with the tracker imputing and averaging (or just need a lot of ifs...)

@TheButlah TheButlah added the Difficulty: Large refactor Requires modifying large sections of the codebase label Nov 18, 2022
@Erimelowo Erimelowo added the Difficulty: Math Heavy For something that requires some non-trivial math label Jul 20, 2023
@ImUrX ImUrX linked a pull request Feb 12, 2024 that will close this issue
Positional tracker support #920
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Area: Skeletal Model Deals with the model of the skeleton and its pose Difficulty: Large refactor Requires modifying large sections of the codebase Difficulty: Math Heavy For something that requires some non-trivial math Priority: Normal The default priority
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Positional tracker support Stermere/SlimeVR-Server-legTweaks
3 participants
@TheButlah @kitlith @Erimelowo