onClick and user input issues with rendered content

[hzgand]

There appears to be a bug in handling user input when interacting with meshes/models. I have provided the code I am working with below, although the example code https://github.com/artcom/react-three-arjs/tree/main/example also produces the same result.

When trying to test the onclick handler, clicking directly on the mesh does not trigger the onclick handler.

However, I noticed that when resizing the window (or rotating the screen on a mobile device), the mesh gets rendered at a different position away from the marker it is supposed to be rendered on top of. That position is where all the user input handlers work.

When refreshing the page to correctly display the mesh on the marker, tapping or clicking at the spot where the mesh was incorrectly displayed (often to the side of the marker) triggers all the user input, even though there is nothing rendered there!

TLDR: It is almost as if the actual mesh that handles user input is invisible at an offset/incorrect position and a fake mesh that does not handle user input is rendered in the expected position.

I am wondering if this is perhaps due to an issue on my end, because I am finding this to be a problem with all rendered content regarding user input, and I would have expected more conversation about them.

import { ARCanvas, ARMarker } from "@artcom/react-three-arjs"
import React, { useState, useRef } from "react"
import { createRoot } from "react-dom/client"

function Box(props) {
  // This reference gives us direct access to the THREE.Mesh object
  const ref = useRef()
  // Hold state for hovered and clicked events
  const [hovered, hover] = useState(false)
  const [clicked, click] = useState(false)
  // Return the view, these are regular Threejs elements expressed in JSX
  return (
    <mesh
      {...props}
      ref={ref}
      scale={clicked ? 1.5 : 1}
      onClick={(event) => click(!clicked)}
      onPointerOver={(event) => hover(true)}
      onPointerOut={(event) => hover(false)}>
      <boxGeometry args={[1, 1, 1]} />
      <meshStandardMaterial color={hovered ? 'hotpink' : 'orange'} />
    </mesh>
  )
}

createRoot(document.getElementById("root")).render(
  <ARCanvas
    gl={{ alpha: true, antialias: false, powerPreference: "default", physicallyCorrectLights: true, precision: "highp", logarithmicDepthBuffer: true }}
    onCameraStreamReady={() => console.log("Camera stream ready")}
    onCameraStreamError={() => console.error("Camera stream error")}
    onCreated={({ gl }) => {
      gl.setSize(window.innerWidth, window.innerHeight)
    }}>
    <ambientLight />
    <pointLight position={[10, 10, 0]} intensity={10.0} />
    <ARMarker
      params={{ smooth: true }}
      type={"pattern"}
      patternUrl={"data/patt.hiro"}
      onMarkerFound={() => {
        console.log("Marker Found")
      }}>
      <Box />
    </ARMarker>
  </ARCanvas>,
)

IMG_2209.mp4

[j-era]

Hi hzgand, thanks for the description. The error is known for a while, but only now had time to deal with it. Could you please try with 0.5.6 (https://codesandbox.io/s/jolly-hodgkin-ssu33)

[j-era]

#22