Action-based Input System

[bjornbytes]

Rough thoughts on action system, interested in thoughts.

Background:

• Headset module provides input from a wide variety of input sources
• It's hard to resolve all of these inputs into high level controls for a cross-device app
• Example: A/B/X/Y buttons on index/touch controllers.
  • Index has A/B on each hand.
  • Touch has A/B on one hand and X/Y on the other.
  • You have to write Lua to detect controller type and query correct buttons.
  • Other cases exist, they all add up and get annoying.
• Solutions?
  • (a) Don't even try
    • Adopt the attitude that this belongs in Lua
      • LÖVR would strictly expose raw inputs
      • Writing Lua is more expressive than what C can provide, can change faster
      • Multiple input libraries can exist, giving people options
    • WebXR does this. The builtin input sources only provide an array of booleans for buttons and an array of floats for axes. Then there's a JavaScript Library with JSON schemas providing metadata for different controllers.
  • (b) Add more high level concepts to the input system
    • e.g. A/B/X/Y could be treated as abstract "face buttons"
    • Touchpads and thumbsticks could just be "the primary axis"
    • This is opinionated -- LÖVR is smooshing stuff into a "universal device".
    • Information would be lost along the way and there are multiple ways to do it.
    • Still, if there is one clear universal device that makes sense, it would be convenient.
  • (c) Actions
    • A lot of companies with smart people working at them were already burned by this
    • They agreed on a solution called "actions" -- implemented in OpenVR/OpenXR
    • Define abstract actions, use config syntax to map them onto devices/buttons.
    • The runtime does all of the low level input logic, you just ask about action state.
    • Less code for Lua/LÖVR -- everything would be handled by the VR runtime.
    • VR runtime developers want us to use actions
    • Aligning with OpenXR likely leads to less work in the future
    • Gotta make sure defining actions isn't worse than current system

Implementation:

• Actions are defined in conf.lua (actions can only be defined at init time in OpenXR).
• Here are some rough sketches of what it would look like:

function lovr.conf(t)
  t.actions = {
    select = {
      type = 'button',
      name = 'Select',
      bindings = {
        index = { 'hand/left/trigger/click', 'hand/right/trigger/click' },
        quest = { 'hand/left/trigger/click', 'hand/right/trigger/click' },
        mouse = { 'leftButton/click' }
      }
    },
    turn = {
      type = 'axis',
      name = 'Turn',
      bindings = {
        index = { 'hand/left/thumbstick/x', 'hand/right/thumbstick/x' },
        quest = { 'hand/left/thumbstick/x', 'hand/right/thumbstick/x' },
        gamepad = { 'joystick/left/x' },
        mouse = { 'cursor/delta/x' }
      }
    },
    handPose = {
      type = 'pose',
      name = 'Hand Pose',
      bindings = {
        index = { 'hand/left/grip/pose', 'hand/right/grip/pose' },
        quest = { 'hand/left/grip/pose', 'hand/right/grip/pose' }
      }
    }
  }
end

• Yuck! Some shorthand could make it a little better:

function lovr.conf(t)
  t.actions = {
    -- Infer name by de-camel-casing the name
    -- Maybe the type can be inferred based on bindings
    -- Use globs to target multiple devices
    -- Allow specification of default bindings that apply to all devices
    select = {
      default = 'hand/*/trigger/click',
      mouse = 'leftButton/click'
    },
    turn = {
      default = 'hand/*/thumbstick/x',
      gamepad = 'joystick/left/x',
      mouse = 'cursor/delta/x'
    },
    handPose = 'hand/*/grip/pose'
  }
end

Maybe that's manageable? I don't know. There would be a small set of builtin actions for things like head/hand poses, trigger buttons, and maybe a couple of other things.

It would potentially be useful if there was a way to have a set of actions shared across multiple LÖVR projects on the same system, maybe sourced by an environment variable or a file somewhere. This would be for development and probably shouldn't apply in fused mode.

Querying actions would be pretty similar to the way things work now, but instead of querying devices and buttons, you'd query actions. One interesting part is that you can specify an optional device to filter by, for the actions that are bound to multiple devices (like the ones that are bound to hand/left and hand/right):

lovr.headset.getPose('handPose', 'hand/left') -- left hand pose
lovr.headset.getPose('handPose') -- returns either left/right pose, unknown which
lovr.headset.isDown('select') -- are any of the select bindings active
lovr.headset.isDown('select', 'hand/right') -- just the right hand trigger
lovr.headset.getAxis('turn') -- return max value of all the turn axes

There would also be functions to help introspect how the actions are bound:

lovr.headset.isActive('handPose') -- does the handPose action have data (are hands tracked)
lovr.headset.getSource('turn') -- returns "index", "quest", "gamepad", "mouse", etc.

[karai17]

https://github.com/excessive/ludum-dare-38/blob/master/src/input.lua
https://github.com/excessive/ludum-dare-44/blob/master/src/GameInput.hx

We tend to already abstract our input into actions so if we just let lovr/openxr handle it, that seems like a win.

[jmiskovic]

My 2 cents. Proposed action framework is nice, but introduces new syntax to keep in head, and hidden camel casing rules will bite beginners. Additionally, if you fetch one axes you'll most likely also need the second one because they will be handled by same piece of code, so they could be part of same action.

Lua is great language for mapping and introducing levels of indirection, so I would actually prefer for action mapping to be left to users. A good third-party library will surface with time and become preferred solution.

For now, maybe RawInput example could be made clearer so that it is easier to extract useful code from it. It does a lot of string concatenation and hides some axes, which makes it hard to understand what function calls are relevant to specific device.

[karai17]

Why even bother with inferring? Lua tables can be keyed with any kind of string, so just make it 1:1.

function lovr.conf(t)
  t.actions = {
    select = {
      type = 'button',
      bindings = {
        index = { 'hand/left/trigger/click', 'hand/right/trigger/click' },
        quest = { 'hand/left/trigger/click', 'hand/right/trigger/click' },
        mouse = { 'leftButton/click' }
      }
    },
    Turn = {
      type = 'axis',
      bindings = {
        index = { 'hand/left/thumbstick/x', 'hand/right/thumbstick/x' },
        quest = { 'hand/left/thumbstick/x', 'hand/right/thumbstick/x' },
        gamepad = { 'joystick/left/x' },
        mouse = { 'cursor/delta/x' }
      }
    },
    ["Hand Pose"] = {
      type = 'pose',
      bindings = {
        index = { 'hand/left/grip/pose', 'hand/right/grip/pose' },
        quest = { 'hand/left/grip/pose', 'hand/right/grip/pose' }
      }
    }
  }
end

[karai17]

Also notable, it is important to allow actions to be defined without any keymappings or bindings so that games are able to allow users to customize their controls, at least to some extent. I don't mind an action having a static type (button, axis, etc) but maybe allow for bindings to be read from either a default list or remapped after the game initializes.

return {
   select = {
      index = { 'hand/left/trigger/click', 'hand/right/trigger/click' },
      quest = { 'hand/left/trigger/click', 'hand/right/trigger/click' },
      mouse = { 'leftButton/click' }
   },
   Turn = {
      index = { 'hand/left/thumbstick/x', 'hand/right/thumbstick/x' },
      quest = { 'hand/left/thumbstick/x', 'hand/right/thumbstick/x' },
      gamepad = { 'joystick/left/x' },
      mouse = { 'cursor/delta/x' }
   },
   ["Hand Pose"] = {
      index = { 'hand/left/grip/pose', 'hand/right/grip/pose' },
      quest = { 'hand/left/grip/pose', 'hand/right/grip/pose' }
   }
}

function lovr.conf(t)
  t.actions = {
    select = { type = 'button' },
    Turn = { type = 'axis' },
    ["Hand Pose"] = { type = 'pose' }
  }
end

function lovr.load()
  local mappings = require "default-mappings"
  lovr.actions:bind("Hand Pose", "quest", mappings["Hand Pose"].quest)
  lovr.actions:unbind("Hand Pose", "index")
end

[bjornbytes]

Re: remapping bindings dynamically: In OpenXR you can't change actions without tearing down and reinitializing the VR session. Could still be done but would need to probably A) rebind everything at once so you aren't reinitializing VR a bunch (like calling :bind in a loop) and B) be careful to avoid stutters? But this is why I'm biasing towards making them as static as possible.

I think the reason to separate action key from action name is mainly for localization -- the name shows up in things Oculus/SteamVR so you could load different names depending locale, but use the same action ID in code.

[karai17]

not being able to bind mappings to actions after init is a complete failure of open xr, then. the localization makes sense but i don't think inference is the right way to go about it. you'd want a table inside the action named locale that your can set names for various languages, either defaulting to "en" or key 1 or simply the action key name.

…

On Thu., Apr. 23, 2020, 18:48 Bjorn, ***@***.***> wrote: Re: remapping bindings dynamically: In OpenXR you can't change actions without tearing down and reinitializing the VR session. Could still be done but would need to probably A) rebind everything at once so you aren't reinitializing VR a bunch and B) be careful to avoid stutters? I think the reason to separate action key from action name is mainly for localization -- the name shows up in things Oculus/SteamVR so you could load different names depending locale, but use the same action ID in code. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#245 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEQD7D2N5M3MIQVNH2CHQ3ROCZTPANCNFSM4MN42G2A> .

[mcclure]

Some high-level comments:

I think there are two cases I care most about. One is "I am doing something super fancy and smart and I want all the raw information I can get", IE, I'm writing something like the rawInput example. The other case is "I am writing a tiny one-page script, either because I am a new user or I am writing a sample for new users". (I think it is a strength of lovr that you can write interesting standalone apps in a single page of code; if every single app is required to load a lua rebinding library to do even basic things, we lose this.) Right now we sort of try to serve both goals and wind up serving neither. I can't write my raw input handler because I can't query which buttons are present on the device, and I can't easily write my simple one-pager because I have to check both "thumbstick" and "touchpad" and I have to know to do that.

I think as long as I still had a path to get raw input if I needed it, this system sounds very good. I like "getSource" and "isActive" very much :) I think the devil is going to be in the implementation details tho.

Some little comments:

I think rebinding makes a lot of sense (if we have this new system plus rebinding, it would open the possibility of simple "controls options" screens) but I do not think that necessarily is required for version 1.0 of the feature.

A small but crucial thing I don't see addressed here is thumbstick deadzones. This is something that I have some ugly code for in my projects now and I know @shakesoda hit the same thing. It seems natural the deadzone of a thumbstick should be taken into account if this is a hardware abstraction library. But also there's maybe a risk that we are trying to put too much into lovr (there is SOME line where the correct answer is "put that in a 3rd party lua lib").

At some point I want to write a lua library for "complex" actions like: Window-averaged curl of at least 3 fingers goes below 0.1 and is held there for 0.4 seconds, then goes above 0.8. This feature doesn't really matter to that plan as long as the feature doesn't get in the way (but I don't think the current design does?)

I agree it would be nice if we could somehow avoid camelcasing.

[bjornbytes]

This love library is good inspiration for actions https://github.com/tesselode/baton